Monitoring the quality of medical news reporting: early experience with media doctor

David E Smith, Amanda J Wilson and David A Henry, on behalf of the media doctor study group*
Med J Aust 2005; 183 (4): 190-193. || doi: 10.5694/j.1326-5377.2005.tb06992.x
Published online: 15 August 2005


Objective: To analyse the reviews of medical news articles posted on media doctor, a medical news-story monitoring website.

Design and setting: A descriptive summary of operating the media doctor website between 1 February and 1 September 2004.

Main outcome measures: Consensus scores for 10 assessment criteria for the medical intervention described in the article (novelty, availability in Australia, alternative treatment options given, evidence of “disease mongering”, objective supportive evidence given, quantification of benefits, coverage of harms, coverage of costs, independent sources of information, and excessive reliance on a press release); cumulative article rating scores for major media outlets.

Results: 104 news articles were featured on media doctor in the study period. Both online and print media scored poorly, although the print media were superior: mean total scores 56.1% satisfactory for print and 40.1% for online; percentage points difference 15.9 (95% CI, 8.3–23.6). The greatest differences were seen for the use of independent information sources, quantification of benefits and coverage of potential harms.

Conclusions: Australian lay news reporting of medical advances, particularly by the online news services, is poor. This might improve if journals and researchers became more active in communicating with the press and the public.

The public gains much of its health and medical knowledge from newspapers, radio and television.1-5 Even doctors may find out about new medical developments through the lay press. Medical news reports can have dramatic effects: they can alter consumer behaviour, like the recent wholesale abandonment of hormone replacement therapy after adverse publicity; they can put pressure on politicians and drug subsidy programs; and affect company shares.6

Medical news reports obviously need to be balanced and accurate. One study of television coverage of medical issues in Australia has shown a bias towards bizarre stories, and those that issue moral warnings, discredit well known people, spruik medical “breakthroughs”, or affirm folk remedies.1

Another concern is honesty in the attribution of media reports. For example, media releases about new pharmaceutical products may be prepared by public relations companies and used as the basis of news stories directly and without attribution. A study in the United States found that most articles about pharmaceutical products failed to include complete information about benefits, harms and costs of treatment and did not report financial ties between study groups or experts and pharmaceutical manufacturers.7 Similar findings came from a recent survey of the media in Canada, and there have been calls for more direct and honest reporting of the results of research into the effects of medicinal drugs.8,9

There are many reasons why journalists may find it difficult to write accurate and balanced articles about new medical advances, including lack of time or space, pressure from editors, and difficulty in accessing independent expert opinion.10 Some of these factors may be hard to change, but there is evidence in other professional areas that timely feedback on performance can improve standards.11 We believe that web-based audits of medical news reports have the potential to improve the quality of media articles.

We report here the early experience of operating a website media doctor, which publishes quality assessments and critiques of Australian lay press news articles about medical treatments, including drugs, procedures and diagnostic tests. The primary aim of the critiques is to encourage journalists to report all important information about new treatments, including novelty, availability, benefits, harms, costs, adverse effects and, where possible, the opinions of experts who are free of obvious conflicts of interest. Media doctor does not provide medical advice and does not assess the quality of the evidence on which the reviewed articles are based; instead it concentrates on the articles themselves. In this respect it differs from the “Hitting the headlines” project in the United Kingdom.12

Rating criteria

Our rating instrument ( (Box 1) is an extension of one previously used to assess the quality of medical news reporting in the US.7 We added criteria covering novelty of the treatment, availability, treatment options, presence of elements of “disease mongering”,13 and the reporting of evidence (study methodology). These criteria are consistent with those used in other media surveys and with advice provided by the Australian Press Council.8,15,16


Between 1 February and 1 September 2004, we identified 199 potential articles, but only 104 of these were reviewed. Reasons for non-inclusion were: articles not satisfying the eligibility criteria; the same article appearing in different media outlets; a glut of articles on the same topic (up to four of the first published articles were then included); and no reviewers available to complete the review within an acceptable time period.

Total scores for proportions of items satisfactorily reported were low overall, with statistically significant differences among the five media outlets (F4,103 = 4.16, P = 0.004). This appears to be almost entirely due to the much lower scores for the 53 online articles, which averaged 40.1% satisfactory compared with 56.1% for the 51 print articles, a difference of 15.9 (95% CI, 8.3–23.6) percentage points (Box 2).

The proportions of individual criterion scores that rated as satisfactory ranged from 23% to 94% for print media, and 14% to 89% for online media (Box 3). Seven of the 10 criteria had an average score of less than 50% in the online media, with the lowest scores being for “quantification of benefits” and coverage of harms and costs of treatment. The item scores were higher for print than online media, with the greatest differences seen for the use of independent information sources, quantification of the benefits and the coverage of “harms of treatment”. These were also the items with the poorest scores, apart from “costs of treatment”, for which both media types scored poorly.

Twenty articles scored no stars (overall scores 25% or less) compared with 11 that scored three stars (scores 76% or above). Examples of high and low scoring articles are given in Box 4. These illustrate the fact that news articles can be informative even when brief: the average length of print articles reviewed was 370 words, and 325 for online articles.

The short period of observation and the modest number of articles rated means it was not possible to draw any conclusions about changes in overall scores over time.


These data show that the Australian media outlets we reviewed often do a poor job in conveying important information on new medical treatments to the public. While the print media’s overall mean score was around 55%, that of the online news services was substantially lower, rating just over 40% satisfactory. Analysis of criterion scores showed that the main areas of weakness were inadequate presentation of evidence on benefits and harms of the interventions, failure to mention the costs of treatments (when relevant) or to obtain independent expert commentary.

It is unclear why journalistic standards should be lower for online news. There is no shortage of space that would justify very brief reports. We suspect the sense of “immediacy” that goes with online news reporting may be compromising standards.

We have not presented an analysis of the by-lines of articles (ie, who they were written by) we reviewed in media doctor. Many articles were short and a number were reproduced from overseas outlets without any consideration of local factors (for instance whether a drug or diagnostic test was available in Australia or subsidised under the Pharmaceutical Benefits Scheme/Medical Benefits Scheme). We do not doubt the articles would have rated more highly if written by dedicated health journalists.

Our main findings are in line with those published elsewhere.7-9 For instance, Moynihan et al found that, of 207 news stories covering new drugs in the US, only 30%–47% adequately covered costs and harms.7 Our estimates for the Australian media are close to these values. Similar observations were made by Cassels et al in regard to the Canadian media.8 The main advantages of our work are that it is ongoing, the results are accessible (there are now more than 400 articles reviewed on the website) and a regularly updated series of reviews is available for journalists and the public to read. Our data provide a continuously updated measure of the performance of the main Australian media outlets.

The main weaknesses of our study are that (1) it covers only five major media outlets (which, because of limited resources, were all we were able to cover during the study period), so the results cannot be considered representative of all Australian media; and (2) we concentrate on the evidentiary aspect of news reporting. This has led to criticisms from journalists.

Initial press coverage of the activities of media doctor attracted critical feedback, which is summarised on the media doctor website ( and in a medical journal news article.17 Supportive feedback included “. . . your website is a great new tool, collecting current treatment information together in the same spot and offering another perspective on the contents”. Criticisms from journalists included statements that “They’re trying to turn the media into a medical journal” and “I do get a bit cynical about non-journalists critiquing journalism”. We accept that the skills involved in writing medical news articles and medical journal articles are different. However, we consulted with journalists in establishing media doctor, and there are three experienced ex-journalists/medical writers in the media doctor review team. We make suggestions about what we think should be included in an article about a new medical treatment. We are not trying to advise journalists about how to write the articles. Our concerns about the quality of health reporting seem to be in line with a warning to journalists from the Australian Press Council.15 The “cultural divide” between medical journalism and medical practice may not be that wide. The “detective skills” and capacity for critical analysis required of journalists and doctors are similar.

We recognise that providing simple feedback to media outlets is unlikely to have a major impact on the quality of medical news reporting. Accordingly, we intend to augment the service in a number of ways. In future we will provide an automatic “alert” to media outlets when one of their articles features on the site. We also intend to send an annual “report card” to editors and producers, which summarises the performance of their company and their competitors, and to extend our coverage to other media outlets.

It is important that the responsibility for poor medical news reporting is not borne solely by journalists and editors. In a number of the articles we reviewed, it was apparent that the high quality was due, in part, to the involvement of researchers in disseminating information to the press. We do not have direct evidence but it is likely that many poor articles could have been improved if investigators had taken the trouble to communicate effectively with journalists. A survey of media releases from medical journals showed that “press releases do not routinely highlight study limitations or the role of industry funding”.18 If medical journals do not highlight these issues it is difficult for journalists to do their job properly. In our view both editorial staff and authors should take responsibility for improving this situation.

1 Criteria for reviewing media articles

Media doctor assesses articles for the extent to which they inform readers according to the following 10 criteria:

4 Examples of articles covered on the media doctor website

"US researchers develop new breast cancer treatment": ABC Online, 25 Jul 2004. Rating: 1 of 9

This article was so brief it was difficult to tell exactly what treatment it was describing. A web search revealed a similar US article about a technique in which a probe, inserted into a breast tumour using magnetic resonance imaging, was heated, so destroying the tumour. The US article stressed the need for controlled trials of this technology and revealed that the doctor in the article worked for the company that manufactures the device. The ABC article lacked any useful information yet portrayed the technique as “hope on the horizon”.

"Managing cholesterol: a new approach": ninemsn, 28 Jul 2004. Rating: 2 of 10

This article concerned the launch of a new cholesterol-lowering drug, ezetimibe. The article very closely paralleled a press release written by a public relations company on behalf of the drug manufacturer. The article’s language was promotional and sources cited were limited to those provided by the PR company. There was no mention of research studies, adverse effects or costs of treatment. The drug’s trade name always appeared in upper case as though in an advertisement. The Pharmaceutical Benefits Scheme listing restrictions were not provided even though this information was in the press release.

"World's first skin cancer cream launched": ninemsn, 02 Sep 2004. Rating: 9 of 10

This was the best of several stories on an imiquimod-containing cream used to treat basal cell carcinomas. The article described the type of research used to test efficacy, quantified the benefits and provided accurate information on availability in Australia. It also provided links to detailed information about the drug and prevention of basal cell carcinomas. However, there was no information about the cost of treatment.

"Drug prevents AIDS spread": The Sydney Morning Herald, 11 Jul 2004. Rating: 9 of 10

This described a study showing that a single dose of a generic form of an antiretroviral agent, nevirapine, when taken in combination with another established drug, zidovudine, by pregnant women, reduced transmission of the AIDS virus to the fetus. The article described the form of research used and the size of the treatment benefit in an appropriate and accurate way. The importance of the low cost of the generic drug was discussed and independent comments from experts were provided. However, there was no mention of the adverse effects of treatment.

  • David E Smith1
  • Amanda J Wilson2
  • David A Henry3
  • on behalf of the media doctor study group*

  • 1 South Lakes Medical Group, Dora Creek, NSW.
  • 2 University of Newcastle, Newcastle, NSW.



Media doctor is a project of the Newcastle Institute of Public Health (the population health research arm of Hunter Medical Research Institute), which receives funding from NSW Health through its population research capacity building program.

Competing interests:

None identified.

  • 1. Chapman S, Lupton D. Freaks, moral tales and medical marvels: health and medical stories on Australian television. Media Information Australia 1994; 72: 94-103. Available at: (accessed Jul 2005).
  • 2. Entwistle V. Reporting research in medical journals and newspapers. BMJ 1995; 310: 920-923.
  • 3. Phillips DP, Kanter EJ, Bednarczyk B, Tastad PL. Importance of the lay press in the transmission of medical knowledge to the scientific community. N Engl J Med 1991; 325: 1180-1183.
  • 4. Johnson T. Shattuck Lecture: medicine and the media. N Engl J Med 1998; 339: 87-92.
  • 5. Nelkin D. An uneasy relationship: the tensions between medicine and the media. Lancet 1996; 347: 1600-1603.
  • 6. Lawton B, Rose S, McLeod D, Dowell A. Changes in use of hormone replacement therapy after the report from the Women’s Health Initiative: cross sectional survey of users. BMJ 2003; 327: 845-846.
  • 7. Moynihan R, Bero L, Ross-Degnan D, et al. Coverage by the news media of the benefits and risks of medications. N Engl J Med 2000; 342: 1645-1650.
  • 8. Cassels A, Hughes MA, Cole C, et al. Drugs in the news: an analysis of Canadian newspaper coverage of new prescription drugs. CMAJ 2003; 168: 1133-1137.
  • 9. Schwartz LM, Woloshin S. The media matter: a call for straightforward medical reporting. Ann Intern Med 2004; 140: 226-228.
  • 10. Larsson A, Oxman AD, Carling C, Herrin J. Medical messages in the media — barriers and solutions to improving medical journalism. Health Expect 2003; 6: 323-331.
  • 11. Jamtvedt G, Young JM, Kristoffersen DT, et al. Audit and feedback: effects on professional practice and health care outcomes. Cochrane Database Syst Rev 2003; (3): CD000259.
  • 12. National Health Service. National Electronic Library for Health. Hitting the headlines. Available at: (accessed Oct 2004).
  • 13. Moynihan R, Health I, Henry D. Selling sickness: the pharmaceutical industry and disease-mongering. BMJ 2002; 324: 886-891.
  • 14. Moxey A, O’Connell D, McGettigan P, Henry D. Describing treatment effects to patients: how they are expressed makes a difference. J Gen Intern Med 2003; 18: 948-959.
  • 15. Australian Press Council. General press release No. 245 (April 2001). Reporting guidelines. Available at: (accessed Oct 2004).
  • 16. Schwitzer G. Improving health and medical journalism. Website available at: (accessed Oct 2004).
  • 17. Sweet M. Website gives media the tough treatment. BMJ 2004; 329: 178.
  • 18. Woloshin S, Schwartz LM. Press releases: translating research into news. JAMA 2002; 287: 2856-2858.


remove_circle_outline Delete Author
add_circle_outline Add Author

Do you have any competing interests to declare? *

I/we agree to assign copyright to the Medical Journal of Australia and agree to the Conditions of publication *
I/we agree to the Terms of use of the Medical Journal of Australia *
Email me when people comment on this article

Online responses are no longer available. Please refer to our instructions for authors page for more information.