On any lot which was designated on a recorded plat at the time the Zoning Ordinance took effect, a building may be constructed or enlarged regardless of lot size, but new construction or enlargement must comply with the setback, side line and rear line restrictions. No existing lot shall be diminished in lot area or frontage to less than the requirements now in effect, except for the purpose of enlarging adjoining lots.