The Board of Zoning Appeals may allow the nonconforming use of a building or structure to be changed to another nonconforming use of the same or of a more restricted classification, provided it finds that the change will be no more detrimental to the neighborhood than the existing nonconforming use. Whenever a nonconforming use has been changed to a more restrictive use or to a conforming use, such use shall not thereafter be changed to a less restricted use.