Why proper understanding of confidence intervals and statistical significance is important

In reply: In our medical education article,1 we aimed to provide practical recommendations for more informative interpretation of findings from randomised controlled trials based on considering the range of effects supported by the data. We sought to discourage the types of misinterpretations that are far too common, as illustrated by the two examples we provided. We recommended that, even when the upper or lower limit of the confidence interval (CI) overlaps with the null, researchers avoid conclusions of “non‐statistically significant” in favour of more directive conclusions that consider the width of the CI and whether it overlaps with clinically important differences, suggesting that values towards the limits of the CIs have less support from the data.

Dalton describes our recommendations as “intuitively appealing but incorrect” because the true value of the population parameter is fixed and cannot be more likely in one region of the CI than any other.2 We agree of course — correct interpretation of the CI in terms of a probability is that in a large number of repetitions of the study, 95% of intervals would include the true population difference. However, it is also true that in a large number of repetitions most of the point estimates would cluster in a narrow range around the true value, while point estimates toward the outer limits would occur less frequently. Values near the centre of a CI therefore indicate parameter values that are more consistent with the data.3

Hurley describes our recommendations as leading towards the use of terms such as “a trend to significance” in under‐powered trials.4 We would like to assure readers that this is not our intention. Indeed, in our second example, the CI indicates that a wide range of treatment effects are consistent with the data and the range is so wide that the result is uncertain — up to 5.7% higher risk of mortality or up to 17.3% lower risk of mortality are compatible with the data. We suggest it is misleading to interpret non‐significant findings from small trials as suggestive of no benefit.5

We encourage full interpretation of the CI in conjunction with wider contextual information. Some might align this with a Bayesian approach.