Temporary buildings or trailers used in conjunction with construction work only may be permitted in any district during the period construction work is in progress, but such temporary buildings shall be removed upon completion of the construction work.