This study’s main goal is to perform a thorough analysis of the functional properties of Deep Learning models used with a particular dataset. The focus of this work is to carefully compare how well five different deep learning models perform in the task of categorizing photographs of gemstones from a dataset that contains eight different classes of gemstones. The focus is on resolving intrinsic dataset issues that come up during the model training phase. Five deep learning models were chosen for assessment: InceptionV3, ResNet50, MobileNetV2, and VGG16. These models were evaluated using the Gemstone Image collection. By doing a comprehensive analysis, we aim to understand the subtleties and capacities of every model and determine how well suited they are to deal with different situations that arise in Gemstone Image classification jobs. Five important performance indicators are included in our evaluation: F1-score, AUC-ROC score, recall, accuracy, and precision. We thoroughly examine each model’s performance in terms of task classification, taking into account trade-offs between recall and precision as well as overall accuracy, prediction accuracy, and the capacity to identify pertinent cases. The models’ ability to discriminate across various thresholds is further elucidated by the AUC-ROC score. Our work attempts to clarify the benefits and drawbacks of these deep learning models by closely examining their performance across several evaluation criteria. This thorough comprehension will enable well-informed choices to be made about which deep learning models to use for gemstone picture classification tasks, hence resolving particular issues raised by the Gemstone Image dataset. The significant results obtained from our approach have led us to pursue a patent for the underlying methodology.