Explainable Multimodal Learning for Genotype Classification Using Image-Based and Biochemical Features: A Case Study on Black Cumin (Nigella sativa L.)
摘要
Accurate classification of black cumin (Nigella sativa L.) genotypes is critical for plant breeding, quality standardization, and industrial use. However, unimodal approaches based solely on morphological or biochemical data are insufficient to represent the multidimensional nature of genotypic variation. In this study, an explainable and interpretable multimodal learning framework integrating image-based seed phenotyping data with biochemical characteristics was developed and evaluated in the classification of five black cumin genotypes. In this study, a total of five genotypes were used: four local breeding lines of Turkish origin (BC1–BC4) and one standard variety (BC5, cv. Çameli). Seed materials were obtained from field trials conducted in the Dardanos Research Area in Çanakkale, Turkey, during 2023–2024. Image-based physical and color characteristics obtained from a total of 1260 high-resolution individual seed images were integrated with six biochemical parameters: crude fiber, ash, protein, oil, total soluble sugars, and total carbohydrates. Multimodal feature vectors were generated using an early fusion approach at the feature level. Four machine learning and four deep learning models were compared to evaluate classification performance. The highest classification accuracy was achieved with the Simple Logistic (93.6%) and ResNet50 (93.2%) models. Statistical analyses showed significant differences in physical and biochemical characteristics among genotypes (p < 0.05). Explainability analyses (Grad-CAM, feature significance analysis, and t-SNE) revealed that elongation, volume, ash, fiber, and carbohydrate content are strong distinguishing markers in classification. The results of the study indicate that the combined use of image analysis and biochemical analysis provides a reliable, non-destructive, and feasible method for genotype identification and product quality control processes. This approach is proposed as a practical decision support system for use within digital agriculture technologies.