Any part of a building, structure, or land occupied by a nonconforming use may be changed to a use of the same or a more restricted classification (subject to such permits as may be required for proposed use); but where the use of a nonconforming building, structure, or land is hereafter changed to a more restricted classification, it shall not thereafter be changed to a use of less restricted classification.