Automated chest X-ray report generation has great potential to improve healthcare efficiency, but rigorous validation is essential for safe clinical adoption. Existing evaluation metrics focus mainly on report-level scores, failing to provide actionable insights for clinicians. In this paper, we present SPEC-CXR (Safety-centered Performance Evaluation in Clinical Report for Chest X-Ray), an evaluation framework that integrates entity-level performance assessment with report-level error analysis using a large language model (LLM). In our approach, the LLM extracts and classifies entities—radiological findings and differential diagnoses—from both generated and reference reports based on a carefully curated entity set. Generated reports are then evaluated on entity presence, location, severity, and prior comparison, yielding structured outputs to calculate detailed entity-level scores (F1 for presence and accuracy for location, severity, and comparison). Our entity-level evaluation shows 91.8% accuracy compared to human evaluation for presence detection and 0.777 Kendall’s tau-b correlation for report-level evaluation. Furthermore, our entity-level performance analysis uncovers critical limitations of current state-of-the-art report generation models across diverse entities, highlighting the urgent need for rigorous, safety-oriented evaluation metrics. Our framework is publicly available and usable: https://github.com/lunit-io/spec-cxr .