Knowledge Distillation of Vision Transformers and Convolutional Networks to Predict Inflammatory Bowel Disease

Inflammatory bowel disease is a chronic disease of unknown cause that can affect the entire gastrointestinal tract, from the mouth to the anus. It is important for patients with this pathology that a good diagnosis is made as early as possible, so that the inflammation present in the mucosa intestinal is controlled and the most severe symptoms are reduced, thus offering the quality of life to people. Therefore, through this comparative study, we seek to find a way of automating the diagnosis of these patients during the endoscopic examination, reducing the subjectivity that is subject to the observation of a gastroenterologist, using six CNNs: AlexNet, ResNet50, VGG16, ResNet50-MobileNetV2 and Hybrid model. Also, five ViTs were used in this study: ViT-B/32, ViT-S/32, ViT-B/16, ViT-S/16 and R26+S/32. This comparison also consists in applying knowledge distillation to build simpler models, with fewer parameters, based on the learning of the pre-trained architectures on large volumes of data. It is concluded that in the ViTs framework, it is possible to reduce 25x the number of parameters by maintaining good performance and reducing the inference time by 5.32 s. For CNNs the results show that it is possible to reduce 107x the number of parameters, reducing consequently the inference time in 3.84 s.