This paper proposes a method for the classification of inflammatory bowel disease (IBD) using deep learning and evaluates the performance of different transfer learning models. IBD is a chronic condition affecting millions of people worldwide, and accurate diagnosis is essential for effective treatment. The two main types of IBD are Crohn’s disease and ulcerative colitis; symptoms include weight loss, abdominal pain, and diarrhea. The exact causes of IBD are not yet fully understood, but it is believed to be a combination of genetic, environmental, and immune system factors. The potential benefits of using CAD in the detection of diseases are increased accuracy, efficiency, CAD systems can help standardize the diagnostic process thereby reducing the likelihood of errors, reduction in overall cost and by early detection improve patient outcomes. The proposed method uses a novel convolutional neural network (CNN) architecture to automatically extract features from medical images, followed by classification based on severity of the disease. To validate the performance of CNN, different pre-trained models such as DenseNet, MobileNetV2, and the InceptionResNetV2 were fine-tuned and their scores are compared. The proposed method is evaluated using a large dataset of endoscopic images. The 90% validation and 86% training scores demonstrate that the proposed method achieves high accuracy in the classification of IBD and performs well when compared with the highly advanced pre-trained networks which are trained on millions of such images. The proposed method has potential applications in clinical settings and can assist physicians in the accurate diagnosis and treatment of IBD.