<p>Chestnut classification is essential for improving postharvest processing efficiency and supporting large-scale commercialization; however, conventional manual sorting is labor intensive, inconsistent, and unsuitable for high-throughput operations. To address these challenges, this study proposes a k-means clustering–vision transformer (ViT)–based approach for classifying chestnuts into five cultivars, two size grades, and two rottenness states using a single-view RGB image. A total of 17,797 images were preprocessed using k-means clustering to segment chestnut regions, and four deep learning models—ViT, EfficientNetB0, ResNet-50, and DarkNet-53—were trained for multi-class classification. Model performance was evaluated using accuracy, precision, recall, and F1-score. Among the CNN models, DarkNet-53 achieved the highest performance, followed by ResNet-50 and EfficientNetB0. The ViT model outperformed all CNN models across all classification tasks, demonstrating superior pattern-recognition capability likely attributable to its self-attention mechanism, which effectively captures global contextual relationships within images. These results indicate that the proposed k-means–ViT framework provides a highly accurate and efficient solution for automated chestnut sorting. The approach shows strong potential for enhancing industrial grading systems by enabling reliable, scalable, and data-driven quality assessment.</p>