The proliferation of artificial intelligence (AI) in healthcare has triggered the need of fair benchmarking. This, in turn, has inspired the rise of computational challenges, where participants worldwide submit models under standardized evaluation protocols. However, ranking AI models –particularly when declaring winners– raises questions about their difference from the next best (but lower-ranked) model. Here, we present a statistically-aware ranking framework, PermRanker, designed to be agnostic to computational workloads (e.g., segmentation, classification, registration) and underlying data types (e.g., 2D pathology images, 3D MRI scans, or even non-imaging data). PermRanker provides fair and informative benchmarking of AI models, based on two stages: (i) a ranking score, based on case-wise cumulative rankings aggregated across multiple metrics and testing cases, and (ii) a rigorous statistical significance analysis via pairwise permutation testing across the ranked order of the AI models. This framework has served as the official ranking mechanism for over 33 international challenges between 2017 and 2025, including the BraTS, FeTS, and ISLES challenges. While mainly applied in biomedical AI challenges, PermRanker aims to address the unmet need of fair and informative benchmarking of AI models beyond this scope, tackle actual real-world conditions, and hence contribute in streamlining clinical translation of AI models.