Data Leakage Concerns in Training and Evaluation Protocols for Oral Cancer Image Classification

Data leakage is a critical issue in deep learning inflating performance and compromise validity, especially in sensitive areas like medical imaging. This study systematically evaluates two common leakage types in oral squamous cell carcinoma classification from histopathology images: (1) preprocessing leakage (global normalization before dataset splitting) and (2) a severe sample-related (patient-related) contamination scenario created by mixing closely related original and augmented images across splits. We trained 11 CNN and Transformer-based models on a public oral cancer histopathology dataset, benchmarking results against published leakage-free baselines. The results obtained show that the configuration with random splitting of original and augmented images (Scenario 2) artificially increased accuracy by up to 18% (mean +14.3%) compared to leakage-free conditions, while the preprocessing-based leakage (Scenario 1) showed smaller deviations (+1.8%). These inflated metrics arise from a combination of cross-split contamination between closely related samples and increased dataset redundancy, rather than genuine gains in generalization ability. Transformers improved leak-free accuracy (+3.9%) but degraded performance in Scenario 2 (−1.4%), revealing sensitivity to sample-specific biases. The observed performance gains under data leakage conditions are methodological artifacts that undermine clinical reliability, with a severe sample-related contamination scenario (Scenario 2) with random splitting of original and augmented images being particularly detrimental due to its promotion of non-generalizable feature learning. The quantitative benchmarks established here-including a mean accuracy gap of 12.5% (Scenario 2 vs. Scenario 1) across 11 models and Transformer architectures’ sensitivity to contamination-reveal fundamental tradeoffs between metric inflation and model trustworthiness. These findings establish quantitative benchmarks for leakage impacts in medical imaging and inform future guidelines for trustworthy AI development in pathology.