Throughout its history, Latvia’s architecture has reflected more than artistic styles or construction techniques—it has carried the weight of political regimes, cultural ideologies, and an evolving national identity. Natural stone, in particular, has served as a powerful medium through which authority, resistance, and memory have been inscribed into the built environment. This article traces how different political powers—from German crusaders and Swedish rulers to the Russian Empire, Soviet regime, and modern Latvian state—have used architecture and stone materials to assert dominance, foster identity, or rewrite the past. By examining this architectural heritage through the lens of symbolic meaning and conservation politics, the study reveals how decisions about what to build, restore, or demolish are never neutral—they are acts of cultural expression. Bridging history and heritage studies, this article calls for a more ethically conscious approach to conservation—one that recognizes architecture not only as a product of its time but as an ongoing narrative of collective memory.