The towers themselves were built by Italian immigrant construction worker Sabato ("Sam" or "Simon") Rodia in his spare time over a period of 33 years, from 1921 to 1954.

The sculptures' armatures are constructed from steel pipes and rods, wrapped with wire mesh and coated with mortar. The main supports are embedded with pieces of porcelain, tile, and glass. They are decorated with found objects, including bed frames, bottles, ceramic tiles, scrap metal and sea shells. Rodia called the towers Nuestro Pueblo ("our town" in Spanish).

Rodia bent much of the Towers' framework from scrap rebar, using nearby railroad tracks as a makeshift vise. Other items came from alongside the Pacific Electric Railway right of way between Watts and Wilmington.

Rodia reportedly did not get along with his neighbors, some of whom allowed their children to vandalize his work. Rumors that the towers were antennae for communicating with enemy Japanese forces or contained buried treasure caused suspicion and further vandalism.