Image processing practices are emerging day-by-day in the health sector. However, a few old methods are also trendy, effective, and efficient in a corresponding manner. This paper focuses on designing an image classification approach based on the Support Vector Machine to classify different medical image modalities. This method has been applied to different medical image modalities. Hence, the Support Vector Machine can classify all considered medical image modalities accurately. The proposed classification method has noticeable benefits among the four image quality indexes named Statistics, Contrast, Homogeneity, and Energy, respectively, for X-ray images than other modalities such as CT scan, MRI, and Ultrasound. Also, the Support Vector Machine-based classification method gives an accuracy of 94.50% for the considered data set of medical images.