"Secondary or appurtenant structure" means a structure, the use of which is incidental or accessory to that of the main building and which is attached to the main building or located on the same premises with it, including retaining walls, fences and exterior walks and steps.