The integration of deep learning tools in gastrointestinal vision holds the potential for significant advancements in diagnosis, treatment, and overall patient care. A major challenge, however, is overconfident predictions, even when encountering unseen or newly emerging disease patterns, which undermines the reliability of such tools. We address this critical issue of reliability in gastrointestinal vision through the lens of out-of-distribution (OOD) detection, which handles previously unseen or emerging diseases as OOD samples. To this end, we hypothesize that the features of an in-distribution example will cluster closer to the centroids of their ground truth class, resulting in a shorter distance between the example and the nearest centroid. In contrast, OOD examples maintain more or less an equal distance from all class centroids. Based on this hypothesis, we propose a novel Nearest-Centroid Distance Deficit (NCDD) score in the feature space for gastrointestinal OOD detection. Evaluations across Resnet, ViT, DeiT and MLPmixer and two publicly available benchmarks, Kvasir2 and Gastrovision, demonstrate the effectiveness of our approach compared to several state-of-the-art methods. The code is available at: bhattarailab/NCDD.