Identifying and classifying medicinal plants is pivotal in various fields, including healthcare, pharmaceuticals, and traditional healing practices. With the integration of advanced technologies, this study focuses on the performance evaluation of a novel approach for medicinal leaf classification. Leveraging the DeepLabv3 model for semantic segmentation, we propose a comprehensive methodology that combines deep learning with traditional machine learning (ML) classifiers. The study begins with developing a real-time identification system using the DeepLabv3 network, enabling swift and accurate classification of medicinal plant leaves from images or live video feeds. Semantic features extracted from the segmentation maps are utilized for feature representation. Subsequently, these features are fed into various ML classifiers, including Support Vector Machine (SVM), Random Forest, Logistic Regression, k-nearest Neighbors (KNN), and a Naïve Bayes classifier. The classifiers are trained and evaluated on a dataset comprising five distinct medicinal plant species: Basale, Betel, Guava, Hibiscus, and Tulsi. Performance indicators such as accuracy, precision, F1 score, and recall are extensively analyzed to assess the efficacy of each classifier in medicinal leaf classification. The DeepLabv3 model′s capability for semantic segmentation contributes valuable features, enhancing the discriminatory power of ML classifiers. The results demonstrate the potential of this integrated approach, providing insights into the strengths and limitations of each classifier in the context of medicinal leaf classification.