Mistakes and misconduct in the research literature: retractions just the tip of the iceberg

To the Editor: Post-publication audits of the quality of medical research studies are vitally important. I support the conjecture of Nath et al1 that the small number of retractions for mistakes and misconduct (about 20 per year for articles published between 1982 and 2002) represents the tip of the iceberg.

I recently wrote a systematic review of studies (published between 1972 and 2005) of growth in children taking stimulant medication for attention deficit hyperactivity disorder (ADHD), and I was astounded by the poor quality of much of the research.2,3 Of the 22 studies reviewed, I felt that 11 were flawed, either because their conclusions were not fully supported by the data, or because of poor methodology, or both. Some had quite subtle mistakes or misinterpretations, such as failure to consider that a child’s height velocity might vary with duration of treatment (two studies). Others were more obvious; for example, a study with a design that introduced systematic errors. Two studies had negative findings associated with inappropriate controls; however, in both of these studies the suitability of the control data was fully discussed. Two studies appeared underpowered, but a full assessment of this could not be made because some of the essential information was either insufficiently detailed or completely lacking. There were three studies lacking any rigorous comparison with control data on which to base their conclusions.

I did not detect any trend for studies with unsupported conclusions or flawed methodology to be published in journals with higher impact factors (median impact factor in both groups, 3.9), suggesting that, while these journals might report a higher rate of retractions (as found by Nath et al1), it may not necessarily reflect a higher rate of mistakes or misconduct among their authors. Likewise, the number of authors and level of funding — pharmaceutical industry or otherwise — appeared to be similar between studies whose conclusions were judged as valid or invalid.

Nearly all of the more rigorously designed studies showed statistically significant slowing of growth in height during the first 1–3 years of treatment with stimulant medication.2 By contrast, the flawed studies as a group supported the notion that stimulant medication does not have any statistically or clinically significant effect on growth in height, the individual studies varying in the extent to which this was emphasised. None of the studies has been retracted, and it is likely that poor quality research has had a substantial influence on clinical opinion in this area.

I do not think that my sample is representative of the quality of medical literature as a whole. I have to admit that my decision to write the review was based on my perception of the poor calibre of many of the studies of growth in children with ADHD. However, I have no reason to believe that the level of poor quality research in my area of interest is unique.

While retractions are important, the medical readership also has a responsibility to evaluate the scientific validity of published studies and, when necessary, correspond with the journals.