This article presents a novel approach for training convolutional neural networks (CNNs) using an active learning framework. The core of this methodology lies in iteratively enhancing the training set by incrementally augmenting high-divergent unlabeled samples as labeled data. Initially, a small subset of labeled samples is selected from the dataset for training, while the remaining samples are treated as unlabeled. Three CNN models (with ResNet-18 backbone), each with identical architecture but independent parameter initialization, have been trained separately on this labeled subset of data. While predicting the class of the unlabeled data, diverse data samples are selected from the probability vectors generated as output from each identical model and their ensemble model. Here, the diversity of samples is quantified by the Kullback–Leibler (KL) divergence measure computed using each model’s predictive distribution and the fused predictive distribution of their ensemble version. The fused prediction is derived by averaging the individual models’ probability vectors. At each iteration, samples exhibiting the higher average KL divergence are identified as diverse and are subsequently augmented into the training set. This process involves multiple iterations to ensure the gradual expansion of the training set with high-diversity samples, thereby improving the model’s performance. The effectiveness of this approach is empirically evaluated by assessing the model on the hold-out test set. Experiments have been carried out on CIFAR-10 and CIFAR-100 datasets. Results demonstrate that the proposed approach significantly enhances the model’s generalization capabilities compared to traditional training methods on a fixed dataset. The proposed approach yielded an average accuracy of 90.65% for CIFAR-10 and an average accuracy of 56.67% for CIFAR-100.