We present ATC-QA (Air Traffic Control Question-Answering), a novel multi-dimensional benchmark for evaluating large language models (LLMs) in aviation safety compliance. Drawing from 43,264 real-world incident reports from the Aviation Safety Reporting System (ASRS), we develop a comprehensive dataset comprising 47,151 question-answer pairs across seven distinct question formats and four difficulty levels. Unlike general-purpose evaluation frameworks, ATC-QA specifically addresses the unique challenges of aviation safety reasoning, where domain expertise and technical precision are critical. Our benchmark encompasses multiple question structures—from basic classification to complex case analyses requiring expert knowledge—enabling granular assessment of different reasoning capabilities. We implement a rigorous four-stage development methodology including report conversion, format diversification, quality refinement, and standardization to ensure benchmark integrity. Our experimental evaluation of three state-of-the-art LLMs reveals significant performance variations across question types, with strong classification capabilities (94–95% for True/False) but substantial limitations in aviation terminology generation (18–20% accuracy). These findings highlight critical implications for deploying LLMs in safety-critical contexts and demonstrate the value of domain-specific evaluation frameworks. Beyond aviation, our work provides a methodological template for developing similar evaluation benchmarks in specialized domains where precision and domain expertise are essential.