Detector-based image matching methods have been widely applied in various computer vision tasks. However, they remain fundamentally limited by two critical issues: (1) susceptibility to extracting redundant or misleading background features in low-texture and complex scenes, and (2) inadequate semantic understanding and integration capabilities, both of which compromise matching accuracy and robustness. To address these limitations, we propose SeViMatch, a novel detector-based image matching framework with semantic-visual fusion. It combines three key innovations: the Adaptive Feature Optimization Module (AFOM) that incorporates multiple attention mechanisms to selectively emphasize foreground regions and suppress background noise; the Semantic-Aware Module (SAM) that models visual-semantic distributions through an encoder-decoder architecture to achieve cross-image semantic alignment and awareness; and the Dynamic Semantic Fusion Module (DSFM) that adaptively adjusts the fusion ratio between semantic and visual features based on content, enabling a deep semantic-visual collaborative representation. Extensive experiments on multiple challenging benchmarks demonstrate that SeViMatch significantly enhances matching precision and robustness across diverse challenging scenarios.