A building or structure or portion thereof, which is nonconforming as to use, shall be subject to the regulations of the district in which that building or structure would normally be located, unless the district in which the building or structure is located is more restrictive than would be the normal district, then the more restrictive regulations would apply.