Why proper understanding of confidence intervals and statistical significance is important

To the Editor: The explanation of inference from confidence intervals by Hemming and Taljaard is interesting but unfortunately incorrect.1 The authors may have fallen for the confidence interval variation of the P value fallacy — the mistaken idea that the P value (or confidence interval) can capture both the long term outcomes of an experiment, as commonly reflected in the phrase “a trend to significance (P = 0.06)”, and the evidential meaning of a single result.2

In a frequentist approach, the P value follows from the null hypothesis, which is either accepted or rejected. The calculation of the P value proceeds only because we have accepted the null hypothesis to be true.

Are Hemming and Taljaard confusing Bayesian and frequentist inferential methods?3 The difference between Bayesian and frequentist logic is analogous to the diagnosis of measles for a hypothetical patient presenting with a fever and a rash.4 With frequentist logic, we would consult a text book (the correct textbook being a key assumption), and base our diagnostic inference on a hypothetical cohort of 100 patients presenting to us with an identical rash and fever, to state that 95 of them would have measles. We would not be able to state which of this hypothetical group of individuals had measles. Moreover, a diagnosis of “a trend to measles (P = 0.06)” does not exist in the real world.

By contrast, with Bayesian logic, our hunch (the prior) that the patient in front of us has measles is firmed up (the posterior) by knowing that there is a measles outbreak in the community (the evidence).

Unfortunately, the thinking commonly found in association with P values and 95% confidence intervals, and suggestions that directive conclusions from randomised trials are achievable from borderline P values, leads to terms such as “a trend to significance” for findings from studies that are underpowered.5