This paper discusses the need for trustworthy AI in urban mobility, focusing on high-stakes security applications such as anomaly detection in public transportation. Because the accuracy required to identify potentially dangerous objects often surpasses the capabilities of current models, there is an unavoidable incidence of false positives. We suggest a “learning to defer” approach as a solution. Our technique uses the deep features and label relative importance of a pre-trained classifier (DenseNet/ImageNET-1k) to create a unique item “fingerprint”. We then employ a zero-shot meta-learning approach to calibrate the system, enabling it to distinguish between normal background items and genuine anomalies by assigning a similarity score. This method significantly reduces the false “new object” alarms that would otherwise overwhelm human operators. Our proof-of-concept demonstrates that the system is computationally light and can be easily adapted to specific environments and integrated into existing classification modules.