Digitalization of historical document images relies heavily on historical text detection techniques. However, historical document images often feature complex layouts, dense text, and varying degrees of damage caused by time and environmental factors, which present challenges for traditional text detection methods, particularly in terms of ink fading and document aging. To address these issues, we propose a novel historical character-level text detector, termed E-FCOS, based on an enhanced Fully Convolutional One Stage (FCOS) object detection model. The E-FCOS detector incorporates a Fast Fourier Transform Denoising (FFTD) layer and an Adaptive Multi-scale Fusion (AMSF) module into the FCOS object detection model. In particular, the FFTD layer employs the fast Fourier transform to filter out high-frequency background noise and utilizes a Transformer to establish contextual dependencies, thereby enhancing the character structure information. The AMSF module employs bidirectional adaptive fusion to capture multi-scale information, which improves the detection performance of dense text. Our detector achieves state-of-the-art results on three datasets: MTHv2, Kuzushiji, and Nancho, especially achieving F-measure of 97.75% in the MTHv2 datasets. Additional generalization experiments on the competition dataset further confirm the potential applicability of this detector.