<p>Drug synergy prediction holds great promise in accelerating combination therapy development and improving treatment efficacy in cancer and other complex diseases. However, progress in this area is hindered by considerable heterogeneity across experimental datasets, including variability in the number of drug combinations, inconsistencies in synergy scoring methodologies, and differences in data quality. Here, we present the first comprehensive benchmarking framework specifically designed to accommodate inter-dataset heterogeneity. This framework integrates 13 independent datasets encompassing 454,794 retained drug combination–cell line entries, 4247 drugs, and 187 cell lines, all of which are cancer cell lines. Our comparative evaluation of seven computational models for drug synergy prediction reveals that model performance strongly depends on both the scale and quality of the datasets. The graph model JointSyn performed favorably on datasets with larger numbers (&gt; 10,000) of retained drug combinations, while the traditional random forest model performed competitively on smaller-scale datasets. We also observe that the ZIP scoring metric yields the highest accuracy in large-scale data, whereas HSA is more effective in sparse-data scenarios. However, different synergy metrics show significant variability in performance across datasets, suggesting that different synergy metrics capture distinct aspects of drug interactions, and the choice of metric can substantially affect model evaluation and cross‑dataset consistency. Furthermore, we find that well-designed small datasets can match or even surpass the performance of larger benchmarks, suggesting that different metrics are applicable to different datasets/testing scenarios. Our benchmark provides a robust foundation for fair model evaluation and paves the way for the development of more generalizable and preclinically relevant drug synergy prediction methods.</p> Graphical abstract <p></p>