Recognition of handwritten subway tunnel disease data refers to the automatic identification of chalk-written subway tunnel structural anomalies, such as cracks and water leakage, from images. It is of great significance to ensuring the safety and reliability of subway operations. Different from existing handwriting recognition tasks, handwritten subway tunnel disease data recognition easily suffers from overfitting and image quality issues. To tackle these problems, we systematically compare two types of frameworks: CNN-based and Transformer-based, and explore an efficient CNN-based architecture which employs the Depthwise Separable Convolution (DSC) for feature enhancement. Through adjusting the number of DSC layers and increasing the types of data pre-processing techniques, we can control model complexity and improve image quality, thereby alleviating the overfitting problem. Besides, we build a dataset specifically for automatic recognition of handwritten subway tunnel disease data. Experimental results on the dataset demonstrate the effectiveness of our proposed method, even outperforming the popular systems on the market.