Medicinal plants have a crucial role in the traditional medical systems of Vietnam. People often consider these resources integral parts of their traditional knowledge. As a result, it becomes critical to use technology to automatically identify these medicinal plants. This paper presents a new method for automating the identification of medicinal plants that utilizes the ViT model’s extracted features and trains SVM classifiers. This study enhances the ability to extract features using ViT’s self-attention mechanism, which successfully captures global relationships within medicinal plant images. It made a contribution by gathering an archive of Vietnamese medicinal herbs and carefully evaluating our method on five distinct datasets. The experimental results clearly demonstrate that the ViT-SVM model outperforms other models, including classic deep learning models and base classifiers.