The evaluation of neural network models, particularly for classification tasks, is a critical phase in the development pipeline, ensuring models perform reliably across diverse datasets and real-world scenarios. A robust evaluation framework relies on multiple performance assessment techniques, each addressing different aspects of model accuracy, generalization, and robustness. This paper presents a study on the development and evaluation of a deep learning-based classification system for UML diagrams, focusing on measuring accuracy through various techniques. The research leverages deep learning techniques to classify UML diagrams into predefined categories, such as class, sequence, state, other UML and non-UML diagrams. The system was evaluated using cross-validation with a k-folding approach, to confirm the model does have a reliable performance estimation and minimizing overfitting. Automatic evaluation and self-training methods were explored to enhance the model’s robustness and adaptability to new datasets. The experiments demonstrated an average accuracy of 82.8%, with a recall of 88.8%, indicating an appropriate sensitivity to positive cases. The evaluation metrics, including accuracy, precision, recall, F1-score, and AUC-ROC curves, provide a detailed view of the model’s strengths and limitations.