Accurate chart detection within digital documents is paramount for advanced AI systems, particularly for chart-based question answering. This paper introduces ChartRec, a high-quality, specialized dataset built from DocLayNet, comprising 2,100 meticulously hand-annotated chart instances across 30,000 documents. To leverage this data, we propose ChartDec, a novel architecture integrating a modified HiFuse backbone with an LW-DETR head, enabling it to effectively capture both fine-grained textual elements and broader chart structures for precise detection. Through extensive experiments, ChartDec demonstrates superior performance against established baselines (LW-DETR, YOLOv11m, YOLOv12m, RetinaNet, Faster R-CNN), particularly excelling on the DocLayNet dataset. Our work further highlights the critical role of pre-training on DocLayNet with CAEv2, followed by targeted fine-tuning, for achieving optimal detection capabilities. Our key contributions include: (1) ChartRec, a high-quality, specialized, re-annotated dataset; (2) ChartDec, an optimized architecture for document chart detection; and (3) a comprehensive evaluation showcasing state-of-the-art chart detection. These advancements pave the way for enhanced visual document analysis and chart-centric downstream applications. Model and dataset are available at https://github.com/Iongshiba/trifuse-lwdetr .