Recent advances in ontology matching have increasingly relied on language models to capture the lexical information of entities. However, this focus on surface terminology often overlooks the formal semantics encoded in the ontological structure and limits the interpretability of alignment decisions. We introduce EXACT-OM, a context-aware model that integrates ontology-derived semantics directly into the alignment scoring process. It integrates lexical similarity, contextual similarity based on informative relation-specific subgraphs, and language model-based signals within an adaptive scoring framework. The system is intrinsically interpretable: it decomposes every decision into component contributions, provides per-triple importance via perturbation analysis, and renders graph-based visual explanations with concise natural-language summaries. On the OAEI Bio-ML benchmark, EXACT-OM achieves competitive performance while delivering fine-grained, auditable explanations. In a user study (n=12), the majority of users preferred EXACT-OM explanations to more traditional tools, and they were found to substantially improve validation efficiency for high performing users, enabling faster decision-making without compromising accuracy. Importantly, both performance and preferences differed across users, suggesting that explanation effectiveness depends more on individual strategies than on domain knowledge or technical skill. Overall, this shows that the principled integration of semantic structure with language models can enhance explainability and expert validation workflows without sacrificing performance.