<p>Ensuring accurate crack detection and segmentation in disaster environments is essential for maintaining structural safety. However, complex noise, irregular lighting, and diverse facility conditions often lead to false positives and reduced segmentation accuracy. To overcome these challenges, this study proposes a structural segmentation-based damage recognition framework. The proposed approach integrates structural context into both CNN-based and Transformer-based models and was validated under various experimental settings. When comparing the results before and after the application of structural segmentation, precision improved by 3.84–16.79%, F1-score by 1.45–7.07%, and mIoU by 3.00–11.86%, while unnecessary crack candidates were effectively suppressed, leading to a noticeable reduction in false positives. These improvements were particularly evident in lightweight models, thereby demonstrating both the effectiveness and generalizability of the proposed approach. Moreover, structural segmentation enabled precise localization of cracks, while segmentation analysis captured their morphological characteristics. This finding is particularly significant, as the structural risk level of the same crack may vary depending on its location within a facility (e.g., wall, column, beam, ceiling). Overall, the proposed framework goes beyond simple crack detection by supporting facility-specific damage risk assessment and enhancing the reliability of automated structural inspection and disaster response systems.</p>