Clinical quality registries for clinician-level reporting: strengths and limitations

We thank Sypek and colleagues for their response to our article1 and their discussion regarding appropriate identification and management of outliers from clinical quality registries. We agree that outliers may occur at both site and clinician level; however, issues of statistical validity are particularly pertinent at the clinician level, where activity volume is generally lower and there are potential consequences for the individual clinician.

We agree that control charts such as cumulative sum control charts and risk-adjusted sequential probability ratio tests2 are useful as early warning detection tools. While these models do not allow relative comparisons between individual clinicians, they allow comparison with an expected (population-level) mean.

Risk-adjusted funnel plots are among the most popular methods used in presenting benchmarked clinician-level data,3-5 yet they have limitations, particularly at low volumes; hence cases should still accumulate over some time before implementation of any outlier policy. Further, the true probability of falling outside of control limits varies depending on the methods used in their construction. The most commonly used control limits for clinician reporting are 95% and 99.8%; however, they are unlikely to be optimal in all circumstances.6 Narrowing the control limits by changing the type 1 error may allow the detection of the clinician (with small patient volume) with a large mortality rate, but then would also detect a potentially large number of false positive outliers. The utility of funnel plots in the presence of small patient numbers and sparse outcomes is not fully understood, and we agree that concurrent application of a Bayesian framework via clinical expert input to define true outliers is an approach that has potential.

We also agree that it is crucial that interpretation of such complex analyses is best undertaken by a multidisciplinary team with input from the relevant subspecialty clinicians, as well as statistical and epidemiological or registry expertise. The considerations raised by Sypek and colleagues — defining acceptable performance, understanding the limitations of the statistical methods employed, and the epidemiological characteristics of the disease or procedure under review — require that this be undertaken by a suitably qualified and independent committee or reference group.