To avoid undue hardship, nothing in this Zoning Ordinance shall be deemed to require a change in the plans, construction or designated use of any building on which actual construction was lawfully begun prior to the effective date of adoption or amendment of this Zoning Ordinance and upon which actual building construction has been diligently carried on. Actual construction is hereby defined to include the placing of construction materials in permanent position and fastened in a permanent manner.