If a nonconforming structure has been torn down, damaged or has physically deteriorated to the extent of more than 65% of its fair market value, as determined by the Board of Adjustment at the time of such damage, no permit shall be issued for its restoration except in conformity to this chapter.