Evaluating methods of outlier detection when benchmarking clinical registry data–a simulation study

Increasing interest in the public reporting of health provider outcomes and outlier identification using benchmarked clinical registry data makes the accuracy of analyses important. Despite this, little research has robustly investigated the optimal methods, and much remains unknown about the performance of outlier detection techniques within this context. To address this, a parametric simulation study was conducted varying the registry data parameters of outcome prevalence, dispersion, outlier definition, and risk-adjustment fit; the proportion of ‘true’ outliers was set at 0.05 for all simulations to reflect a common flagging percentage. Outliers flagged from a combination of four models (unadjusted, ordinary, fixed effects and random effect logistic regression) and two classification techniques (confidence intervals, control limits) were compared to the simulated ‘true’ underperformers using performance measures such as receiver operator area under the curve. The accuracy of outlier detection varied both between model/method combinations and across the different registry parameters, especially prevalence and dispersion. Overall, risk-adjusted site estimates from logistic regression with 95% exact binomial control limits to flag outliers had the best performance, and estimates from random effects regression performed the worst. Data parameters should be considered when choosing benchmarking and outlier detection techniques for clinical registries. More research evaluating performance under further registry scenarios is needed to determine the optimal methods for implementation and accuracy for communicating to government and stakeholders.