If the Building Official refuses to issue a permit for moving a building on the grounds that the move would be detrimental to public safety and welfare or damaging to public or private property, an appeal may be made from his decision to the Planning, Zoning and Building Appeals Board which may review the Building Official's decision. The decision made by the Board will be final.