While models like ResNet18, Inception-ResNet-v2 and ViT achieve high accuracy on benchmark datasets, their reliability under real-world shift remains uncertain. This study focuses on analyzing video distortion classification under dataset shift, with particular emphasis on understanding model uncertainty and representation behavior. We investigate how concept and covariate shifts affect model performance using Monte Carlo dropout. Distortions with strong, globally consistent visual features demonstrate greater robustness to concept shift, whereas spatially varying distortions are more prone to confusion. Under covariate shift, training on subtler distortions improves generalization and stabilizes uncertainty, acting as implicit regularization. These findings expose key limitations of current supervised models and motivate the need for more robust, uncertainty-aware approaches for real-world video distortion classification.