Evaluating AI models in clinical decision support requires metrics that go beyond aggregate accuracy to reflect user needs, decision uncertainty, and real-world risk. Traditional metrics such as precision or AUC fail to capture these human-centered concerns, particularly in high-stakes and imbalanced settings. Current evaluation frameworks rarely integrate user hesitation, case relevance, or the asymmetry of error consequences. This abstract summarizes a keynote presented at the 1st Workshop on Artificial Intelligence for Biomedical Data (AIBio) 2025, co-located with ECAI 2025, where we aimed to fill this gap by introducing the weighted Utility (wU) metric–a user-centered evaluation method that incorporates rater-specific hesitation thresholds, relevance weights, and parameters for the asymmetric impact of model errors. The metric generalizes existing decision-theoretic measures such as the Standardized Net Benefit and operationalizes them in a case-dependent, psychometrically informed framework. In clinical case studies on knee MRI classification and ovarian cancer sonography, wU-optimized models improved overall AUC from 0.862 to 0.895 (p < 0.05), with the greatest gains on high-complexity cases (AUC from 0.85 to 0.92, p < 0.05). We also present two online tools, Metimeter and dAIagrams, which support the practical application of wU by enabling data upload, metric computation, and visual decision analysis. These findings suggest that wU offers a more clinically relevant evaluation lens, particularly in contexts where expert judgment is fragile. The metric supports the design and deployment of AI systems that align with human decision-making under uncertainty.