To avoid undue hardship, nothing in this Zoning Code deemed to require a change in plans, construction, or designated use of any building on which actual construction was lawfully begun prior to the effective date of adoption or amendment of this Zoning Code and upon which actual building construction has been carried on. Actual construction is hereby defined to include the placing of construction materials in permanent position and fastened in a permanent manner. Where demolition or removal of an existing building has begun preparatory to rebuilding, the demolition or removal shall be deemed to be actual construction, provided that the work shall be carried out diligently.