Background <p>Healthcare stakeholders are increasingly seeking comparative provider performance data to enhance data-driven decision-making and quality improvement. Traditional visualisations, like caterpillar plots, are often difficult for end users to understand and interpret. This study aimed to (1) obtain general feedback from end users on a newly proposed design solution for visualising a risk-adjusted hospital comparison and to develop an understanding of the key criteria they rely on in the evaluation process; (2) test the hypothesis that end users will better understand key messages and rate perceived usability higher with the new design solution than with a caterpillar plot.</p> Methods <p>An end user-centred mixed methods study, involving end users of risk-adjusted hospital comparisons across all levels of the Swiss healthcare system, was conducted to evaluate the new design solution. In the qualitative phase, 14 end users from health authorities, insurers, hospital associations, and hospitals were surveyed in 10 semi-structured individual and group interviews, which were analysed using thematic analysis. In the quantitative phase, a non-clinical randomised controlled online trial (A/B testing) was conducted. In total, 200 of the targeted end users, comprising cantonal quality managers, hospital directors, and those responsible for quality and/or the ‘National Prevalence Measurement’ in hospitals, completed the questionnaire. The data were analysed using comparative descriptive and bivariate statistics.</p> Results <p>Thematic analysis revealed three key criteria that end users relied on when evaluating a risk-adjusted hospital comparison: (1) ‘clarity by design’, highlighting strategies for effectively conveying key messages of hospital comparisons; (2) ‘usability by design’, focusing on end user-centred functionalities and presentation elements; (3) ‘suitability for quality development’, addressing the conditions for creating a trustworthy and useful comparison to drive quality improvement. Quantitative analysis confirmed the hypothesis that end users understand key messages better and perceived usability is higher with the new design than with the caterpillar plot.</p> Conclusions <p>The new design solution improves hospital comparison outputs for end users by combining clear displays with additional interactive features. The identified criteria underlying the evaluation should inform further design projects and research dealing with the visualisation of hospital comparisons.</p> Clinical trial number <p>Not applicable.</p>