<b>Purpose</b> <p>This work addresses the problem that interpretability methods for medical imaging models are still mostly evaluated by agreement with human annotations, even though such agreement does not necessarily indicate that an explanation is faithful to the model’s actual decision-making. The goal is to quantify fidelity, defined here as the extent to which an explanation reflects the evidence that truly drives model predictions, independent of human intuition, bias, or incomplete annotation.</p> <b>Methods</b> <p>We propose a prediction confidence-based approach to estimate token importance in vision transformers and introduce the confidence-based fidelity score (CFS), which compares token importance signals with interpretability maps to measure how faithfully an explanation matches model-intrinsic reasoning. We evaluate the approach across three medical imaging datasets using vision transformers initialized with both supervised and self-supervised (SSL) pretraining.</p> <b>Results</b> <p>Across datasets, models, and explanation methods, the experiments reveal clear discrepancies between fidelity and human annotation alignment, indicating that explanations can appear plausible or overlap with annotated regions while still failing to reflect confidence-sensitive model evidence.</p> <b>Conclusion</b> <p>The proposed token importance formulation and CFS provide a practical, model-aligned way to assess interpretability fidelity in transformer-based medical imaging, complementing annotation-based evaluation and supporting more reliable auditing and selection of explanation methods for clinical AI workflows.</p>