No person in any manner shall interfere with, take down, alter or deface any number on any house or building, or put up or retain an improper number, or substitute any other number on his house or building other than the numbers specified and defined by virtue of the provisions of this chapter. (1937 Code, Sec. 97.)