The accurate identification and localization of handwritten mathematical expressions (HMEs) in students’ solutions plays a crucial role in enabling intelligent assessment and customized educational support. This work focuses on the problem of HME detection, which is significantly challenging as HMEs in the question-answering area demonstrate unique characteristics such as extreme closeness, excessive aspect ratio, and scribbled writing. However, these unique features can’t be handled completely by existing well-built scene text detectors. To overcome these limitations, we introduce an improved DBNet architecture that integrates the Convolutional Block Attention Module (CBAM) and presents a novel optimization component termed neighbor consistency loss to enhance detection effectiveness. To thoroughly assess the effectiveness of our approach, a novel dataset, called MQHS, was built, which comprises handwritten solutions to math questions. Experimental results on MQHS demonstrate that our method achieves state-of-the-art (SOTA) performance. The integration of CBAM improves the detection accuracy by effectively distinguishing between closely located expressions, while the neighbor consistency loss reduces issues of over-detection of lengthy formulas. The GitHub link of MQHS dataset: https://github.com/Jzliu-dl/MQHSdataset .