Machine Learning (ML) techniques are widely used for the detection and classification of diseases in medical fields. These algorithms make decisions based on extracted features from images. Therefore, selecting appropriate feature extraction techniques together with appropriate Machine Learning (ML) algorithms is crucial for achieving high accuracy in classification. Through the literature review, we observed a very few information about ML techniques for the classification of Gastrointestinal (GI) diseases. To solve this issue, we collected five classes of GI images from a publicly available Kvasir dataset and extracted textural features like Local Binary Pattern (LBP) and Gray Level Cooccurrence Matrix (GLCM), color features from RGB (Red Green Blue) and HSV (Hue, Saturation, and Value) color spaces and shape information from HOG. These features are combined and dimension was reduced by Principal Component Analysis (PCA). Finally, the selected features are classified with well-known ML classifier One Against All-Support Vector Machine (OAA-SVM) and obtained classification Accuracy-96.8%, Precision-96.74%, Recall-96.75% and F1-Score-96.73%.