In the case of the issuance of a building permit for an outbuilding simultaneously with a permit for a principal building, the permit shall be held to be invalid unless the principal building to which the outbuilding is, or intended to be, appurtenant is erected and constructed. Failure to erect or construct the principal building shall be sufficient cause to revoke the permit for construction of the outbuilding. The outbuilding shall thereafter be removed upon receipt of an order from the building inspector calling for such removal. (1962 Code § 10-726)