Large language models (LLMs) have demonstrated impressive capabilities in capturing semantic relationships from unstructured text, yet it remains unclear how effectively they can reason over heterophilic graph structures, where connected nodes exhibit dissimilar labels or attributes. This paper reveals that LLM-derived semantic signals, although valuable, can introduce noisy or misleading connections when naively applied to heterophilic graphs. To address this challenge, we propose SemFuse, a structure-constrained semantic enhancement framework that leverages graph topology to filter, validate, and supplement LLM-derived semantic edges. SemFuse distills task-relevant semantic discrimination into a lightweight local LLM, constructs a high-confidence semantic graph by validating and augmenting edges under structural constraints, and fuses structural and semantic views via cross-graph attention. Extensive experiments on multiple heterophilic benchmarks demonstrate that our approach substantially improves node classification accuracy, outperforming previous state-of-the-art methods by an average margin of 2.02%. These results underscore the crucial role of structural guidance in leveraging LLM semantics for graph learning, providing a principled approach to integrating textual knowledge with topological dependencies in heterophilic scenarios.