Urban development is widely considered one of the major challenges World Heritage is currently facing. Issues often emerge from development proposals beyond the official boundaries, within the so-called wider setting. A concept that is referred to in those cases is visual integrity—which, we argue, lacks conceptual clarity and methodological operationalization, leaving room for subjective interpretation and influence by vested interests. A major challenge is that heritage is often not yet fully understood, despite current approaches for a systemic understanding: Urbanistic features of monuments and sites as well as characteristics of and relationships with their surroundings are frequently unaddressed or not formally recognized. This chapter argues that, to reconcile heritage preservation and (urban) development, it is necessary to grasp and define the relationship between site and spatial context in all its complexity and to integrate it in further nominations and other management tasks. We demonstrate how heritage doctrines and discourses have historically contributed to anchoring and perpetuating the gap of understanding between site and setting, before we illustrate the evolution of approaches to overcome it. Current concepts such as the Historic Urban Landscape and Heritage-based Urban Development, which recognize the transformative and systemic character of heritage, are a large step forward and should form the basis of future debates. However, practical experience reveals structural issues that complicate processes such as impact assessments, which need to be based on formally defined attributes and values. This chapter, as well as the doctoral thesis in the context of which it is written, is inspired by such practical experience, shortcomings, and challenges, and aims at facilitating future practice. We therefore propose a framework to describe (World) heritage sites—explicitly including their urbanistic features and aspects of the setting—to be applied in future processes of nomination and management. Based on a qualitative, systematic analysis of reference documents for 18 World Heritage sites using Grounded Theory, we developed a framework of visual elements—a concept we propose to systematically integrate broader spatial contexts, beyond the current sets of attributes which are usually limited to the heritage perimeter itself. This framework, while recognizing the individuality of each site, is abstract and broad enough to cover a wide range of elements and to consider the dynamic nature and complexity of values. It consists of three dimensions (material, perceptive, and interpretative) and three spatial levels (micro, meso, and macro), allowing semantic and spatial relationships to be drawn between the elements. Following the idea of understanding theories as tools, the framework was applied to analyze and compare World Heritage sites with regard to the visual elements that are mentioned in related documents. The analysis revealed a high relevance of urbanistic features and characteristics of the setting, such as silhouettes, skylines, the interplay between objects, and visual relations—despite their lack of formal recognition. Consequently, the framework can help to systematically analyze and document the characteristics of sites and their surroundings in a comprehensive way. It thus promotes the operationalization of a systemic understanding of heritage as another step to bridging the gap between heritage preservation and development.