The diverse and complex morphological characteristics of traditional Chinese medicine herbs make automated image classification of traditional Chinese medicine a critical and challenging task. To address the issue of limited traditional Chinese medicine image datasets, this study constructs a dataset consisting of 16 common traditional Chinese medicine herb categories and a total of 3552 images, covering a diverse range of backgrounds, freshness levels, processing methods, and display arrangements. Based on the characteristics of the dataset, a specialized feature extraction encoder, CSA (ConvNeXtv2 with SE and ACmix), is proposed for traditional Chinese medicine image classification. The CSA encoder employs large-kernel convolutions to extract high-dimensional feature maps from traditional Chinese medicine images. It then enhances the competition among channels in the feature maps using a squeeze-and-excitation (SE) module and improves the network’s ability to process both global and local information through a convolution-attention mixed module. To reduce computational costs, the output dimension of the ACmix module is reduced to one-quarter of the original channel number, followed by a 1 × 1 convolutional layer to increase the channel dimension to z. Experimental results show that the CSA encoder outperforms current mainstream deep learning algorithms, with improvements of 1.56%, 1.81%, 1.69%, and 2.15% in accuracy, precision, recall, and F1 score, respectively. Ablation studies demonstrate that the integration of both the SE and ACmix modules achieves the best classification performance, validating the effectiveness of these modules.