Cultural heritage is located in attractive sites that lead to tourism development; nevertheless, tourism and other economic activities can influence the degradation of cultural heritage sites and pollute the environmental surroundings. Numerous official data from different institutions can provide insight into the relationship between the tangible cultural heritage sites and the environmental settings in which it is located. In addition, research involving the local population can significantly help assess the quality of the environment and tangible cultural heritage sites at the local level. To examine the environment state and its relationship with natural resources pollution and tangible cultural heritage at the local level, a survey of respondents from 52 municipalities in Serbia was conducted. The results indicate that a third of respondents believe that environment and natural resources (air, surface water, groundwater, and soil) near their municipality’s most significant cultural heritage sites are moderately polluted. Most respondents singled out air as a heavily polluted natural resource. Additionally, more than half of the respondents believed intensive construction endangers their municipality’s tangible cultural heritage sites. A more detailed analysis in this study included the correlation of residents’ responses and socio-economic variables. The results obtained can serve to improve the management of tangible cultural heritage sites and environmental protection.