This study explores the complex historical context of the former private library founded by Masatake Terauchi (1852–1919), the first Governor-General of Korea and later the 18th Prime Minister of Japan. Over the past few decades, the building has largely been forgotten and now stands as an abandoned structure. To achieve this, the author employed a framework by organizing key concepts as follows: “A = initial meaning at the time of construction”, “B = cultural catalyst”, “A’ = transformed meaning”, and “C = semantic shifts occurring throughout the transformation process”. A document survey and site investigation of the historical building, constructed in 1921, were conducted within this framework. The investigation revealed that various elements of “B” were presented in the library’s history. Notably, the Korean Pavilion, which stood as a political symbol, was located nearby, Terauchi’s identity as a military figure overshadowed his cultural legacy; and the prevailing political relationship between Japan and Korea contributed to multiple interpretations of “A’”, These factors played a significant role in the building’s decline and eventual ruin. By embracing this pluralistic approach to memory, new possibilities for cultural exchange may emerge, offering a deeper understanding that transcends the simple movement between past and present.