Deep learning has achieved remarkable success in image classification, yet models often rely on misleading ‘shortcuts’ that do not generalise beyond lab settings, such as watermarks or background cues. It has been proposed that instance-level explanations in explainable AI (XAI) may help reveal such shortcuts without external data, but doing so typically requires examining many individual explanations, making the process labour-intensive and often infeasible. We introduce Counterfactual Frequency (CoF) tables, a novel method that aggregates local explanations into global insights and efficiently exposes shortcuts. This requires semantically meaningful image segments, with appropriate labels which we obtain through pre-trained foundation segmentation models. We demonstrate the effectiveness of CoF tables across multiple datasets, including real-world datasets such as ImageNet, toy datasets constructed for this study, and Biased Action Recognition (BAR), highlighting shortcuts like background reliance and watermarks learned by both models we train and well-studied pre-trained classifiers.