Gastro-CNN-VIT: Vision Transformer and Deep CNNs for Detecting GI Diseases in WCE Images

The wireless capsule endoscopic (WCE) is a technological breakthrough in gastrointestinal (GI) assessment. The easy-ingestible device comes with a camera, a light source, and optional sensors to capture images as it navigates the digestive tract. Furthermore, Computer-Aided Diagnosis Systems (CADS) supplement this innovation, enabling healthcare professionals to make informed decisions about disease identification and recognition. Integrating the WCE with CADS creates a potent synergy that improves GI diagnostic capabilities. This work proposes a novel approach for the classification of GI diseases by combining the capabilities of both convolutional neural network (CNN) and vision transformer (VIT). With our approach, we seamlessly combine CNN’s local features extraction capabilities with ViT’s self-attention mechanisms to provide global contextual features. The effectiveness of our proposed method is demonstrated by comparative experiments, demonstrating a notable improvement in the accuracy of classifying images related to GI diseases. As a result of this fusion of CNN and ViT, image classification techniques for GI health are expected to be more accurate and effective in the future.