Evaluating medicines: let’s use all the evidence

Chris W Kelman, Sallie-Anne Pearson, Richard O Day, C D'Arcy J Holman, Erich V Kliewer and David A Henry
Med J Aust 2007; 186 (5): 249-252. || doi: 10.5694/j.1326-5377.2007.tb00883.x
Published online: 5 March 2007
Evolution of pharmacovigilance

Historically, procedures for establishing the effectiveness of medical treatments have been suboptimal, with many commonly used treatments still awaiting rigorous evaluation.4 The safety of many medicines has largely been taken on faith, on the assumption that claimed benefits outweigh (un)known risks. In the early 1960s, the connection between thalidomide and severe birth defects was observed;5 this was a potent catalyst for clinicians and regulators to consider systems for monitoring drug safety.

Limitations of the current system
Premarketing evaluation

There are a number of reasons why premarketing evaluation cannot fully explore the effects of new medicines.7

Postmarketing evaluation

The reporting systems for adverse drug reactions in most countries are voluntary and depend on individuals becoming suspicious that an event is related to a treatment, and then actually reporting it. Identifying medicine-related adverse events is straightforward for unusual reactions such as severe skin rashes or liver failure, and those with a clear temporal relationship, but difficult for common events with well known causes, such as heart attacks and stroke. In either case, risk quantification (rather than mere signal generation) requires measurement of incidence, as well as strength of association between exposure and outcome — such information is not available from adverse event reporting databases.

A recent example of the need for formal ongoing evaluation has been reported by McEwen et al.10 The disease-modifying antirheumatic drug leflunomide was approved in Australia in 1999 for use as monotherapy for rheumatoid arthritis. Actual usage has proven to be different, and the incidence of pancytopenia after use of leflunomide in combination with methotrexate now appears to be more than 1/1000 per person-year. Preregistration examination of such concomitant use was restricted to only 30 patients. While the voluntary reporting scheme has now alerted regulators to this problem, this example emphasises that neither the manner of clinical use nor real-world safety can be assumed when approving new medicines. A routine system for the ongoing examination of actual postmarketing experience seems a wise precaution.

While useful postmarketing information about medicines is sometimes generated by formal studies (a highly desirable but costly approach), observational data about these medicines are already available in Australia as a by-product of administrative processes. These data describe large patient populations and provide a promising new opportunity for monitoring safety of medicines. However, a formal system for examining these data is yet to be implemented, for a number of reasons, including concerns about patient privacy, lack of political will, and restrictions in access to and linkage of the various data collections.

A new approach

The urgent need for increased medicines surveillance has been recently argued by international bodies, such as the International Society for Pharmacoepidemiology, and considered as a component of the new requirements for postmarketing epidemiological studies by the US Food and Drug Administration and the European Agency for the Evaluation of Medicinal Products. Similar discussions are underway locally as legislation is developed for the upcoming Australia New Zealand Therapeutic Products Authority.11

An optimal postmarketing surveillance system should assess all available information about the performance of medicines, drawing on worldwide pre- and postmarketing trials and voluntary reporting, with the addition of findings from regular, systematic and ongoing evaluation using linked observational data.1,7,8,12-14 A number of centres in several countries are already performing high-class pharmacoepidemiological studies of this type.15-19

The utility of linked observational data is well illustrated by the aftermath of the worldwide withdrawal of the cyclooxygenase-2 inhibitor rofecoxib (Vioxx) in response to concerns that the medicine increased heart attack risk. The international literature has since carried a series of over 20 controlled observational and cohort studies evaluating the excess risk associated with use of rofecoxib and related drugs in the United States, Canada and Europe.20 All but two studies employed linked observational data (or linked electronic medical records), permitting rapid evaluation of potential harms. Had these data been analysed cumulatively as soon as they became available (as would occur in a routine surveillance system), rofecoxib might have been withdrawn earlier. A recent meta-analysis of controlled observational studies has indicated that diclofenac has a similar risk profile.21

Australia, as a fortuitous consequence of a complex multi-jurisdictional health care funding system, is replete with observational health care data in linkable databases. They describe the vast majority of health care encounters across the nation, including doctor visits, medicine dispensing, hospital admissions, deaths, and various disease registries. Recent work has shown the practicality of bringing these data together for active monitoring of the outcomes of new drugs while they are being used in the “real world”.22-24 Specific methodologies have been developed to support pharmacoepidemiological analysis of these large datasets.25,26 Encouraging progress has been made over the past few years to improve access to these health datasets. A major linked data resource is now available in Western Australia and a similar resource is being developed in New South Wales.27 To ensure ethical and appropriate use of these data, a nationally accepted protocol has been designed to guarantee both strong privacy protection and accurate linkage.28 While these data have some known limitations and biases, they portray actual practice in use of medicines — and their long-term consequences. With the adoption of appropriate case-inclusion criteria, most of the biases in these data can be controlled.29 Although there are as yet no overseas examples of “routine” medicines monitoring, there have been useful studies based on ad-hoc linkages of medicines and health-outcomes data.30,31

Our preferred approach would support the stated regulatory goals of Australia’s National Medicines Policy.32,33 We propose a staged introduction, incorporating the following features:

  • Chris W Kelman1
  • Sallie-Anne Pearson2
  • Richard O Day2
  • C D'Arcy J Holman3
  • Erich V Kliewer4,5
  • David A Henry6

  • 1 National Centre for Epidemiology and Population Health and ANU Medical School, Australian National University, Canberra, ACT.
  • 2 University of New South Wales, Sydney, NSW.
  • 3 Centre for Health Services Research, University of Western Australia, Perth, WA.
  • 4 Epidemiology and Cancer Registry, CancerCare Manitoba, Winnipeg, Manitoba, Canada.
  • 5 Department of Community Health Sciences, University of Manitoba, Winnipeg, Manitoba, Canada.
  • 6 School of Medicine and Public Health, University of Newcastle, Newcastle, NSW.



We wish to thank Associate Professor Andrea Mant, Dr John McEwen, Dr James Harris and Dr Tom Faunce for their useful comments and encouragement. We also appreciate the support of stakeholders, including industry and consumers, for this proposal.

Competing interests:

Richard Day sits on several advisory boards (mostly related to treating rheumatic disorders with biological agents and anti-inflammatory drugs) that meet ad hoc to review data about new medicines and to give advice regarding their clinical uses. Honoraria are received for these activities, which are unrelated to his contribution to this article.

  • 1. Baciu AS, Stratton K, Burke SP, editors; Committee on the Assessment of the US Drug Safety System. The future of drug safety: promoting and protecting the health of the public. Washington, DC: The National Academies Press, 2006. (accessed Jan 2007).
  • 2. Britton A, McKee M, Black N, et al. Choosing between randomised and non-randomised studies: a systematic review. Health Technol Assess 1998; 2: i-iv and 1-124.
  • 3. Ray WA, Stein CM. Reform of drug regulation — beyond an independent drug-safety board. N Engl J Med 2006; 354: 194-201.
  • 4. Moseley JB, O’Malley K, Petersen NJ, et al. A controlled trial of arthroscopic surgery for osteoarthritis of the knee. N Engl J Med 2002; 347: 81-88.
  • 5. McBride WG. Thalidomide and congenital abnormalities. Lancet 1961; 2: 1358.
  • 6. Australian Government Department of Health and Ageing. Australian adverse drug reactions bulletin 2006. (accessed Aug 2006).
  • 7. Hunter D. First, gather the data. N Engl J Med 2006; 354: 329-331.
  • 8. Strom BL. How the US drug safety system should be changed. JAMA 2006; 295: 2072-2075.
  • 9. McLean AJ, Le Couteur DG. Aging biology and geriatric clinical pharmacology. Pharmacol Rev 2004; 56: 163-184.
  • 10. McEwen J, Purcell PM, Hill RL, et al. The incidence of pancytopenia in patients taking leflunomide alone or with methotrexate. Pharmacoepidemiol Drug Saf 2007; 16: 65-73 [published online ahead of print: 24 Apr 2006]. DOI: 10.1002/pds.1236.
  • 11. Australian Government Department of Health and Ageing. Submissions sought on proposed Australia New Zealand Therapeutic Products Authority regulatory scheme. 23 May 2006. (accessed Sep 2006).
  • 12. Platt R, Ommaya A. A beneficial side effect of the Medicare drug benefit. N Engl J Med 2005; 353: 2742-2743.
  • 13. Nelson MR, Tonkin AM, Cicuttini FM, McNeil JJ. COX-2 inhibitors: exemplars of the drug-safety conundrum [editorial]. Med J Aust 2005; 182: 262-263. <MJA full text>
  • 14. Laupacis A, Paterson M, Mamdani M, et al. Gaps in the evaluation and monitoring of new pharmaceuticals: proposal for a different approach. CMAJ 2003; 169: 1167-1170.
  • 15. Jick H, Vasilakis C, Weinrauch LA, et al. A population-based study of appetite-suppressant drugs and the risk of cardiac-valve regurgitation. N Engl J Med 1998; 339: 719-724.
  • 16. Evans JM, MacDonald TM. Record-linkage for pharmacovigilance in Scotland. Brit J Clin Pharmacol 1999; 47: 105-110.
  • 17. Mangano DT, Tudor IC, Dietzel C; Multicenter Study of Perioperative Ischaemia Research Group; Ischaemia Research and Education Foundation. The risk associated with aprotinin in cardiac surgery. N Engl J Med 2006; 354: 353-365.
  • 18. Vlahakes GJ. The value of phase 4 clinical testing. N Engl J Med 2006; 354: 413-415.
  • 19. PHARMO Institute for Drug Outcomes Research. PHARMO report. Innovations in pharmacotherapy. Utrecht: PHARMO Institute, 2000. .PDF (accessed Dec 2006).
  • 20. Ardoin SP, Sundy JS. Update on nonsteroidal anti-inflammatory drugs. Curr Opin Rheumatol 2006; 18: 221-226.
  • 21. McGettigan P, Henry D. Cardiovascular risk and inhibition of cyclooxygenase: a systematic review of the observational studies of selective and nonselective inhibitors of cyclooxygenase 2. JAMA 2006; 296: 1633-1644.
  • 22. Brameld KJ, Thomas MA, Holman CD, et al. Validation of linked administrative data on end-stage renal failure: application of record linkage to a “clinical base population”. Aust N Z J Public Health 1999; 23: 464-467.
  • 23. Holman CD, Bass AJ, Rouse IL, Hobbs MS. Population-based linkage of health records in Western Australia: development of a health services research linked database. Aust N Z J Public Health 1999; 23: 453-459.
  • 24. Stryer D, Tunis S, Hubbard H, Clancy C. The outcomes of outcomes and effectiveness research: impacts and lessons from the first decade. Health Serv Res 2000; 35: 977-993.
  • 25. Chen J, He H, Li J, et al. Representing association classification rules mined from health data. In: Khosla R, Howlett RJ, Jain LC, editors. Knowledge-based intelligent information and engineering. Lecture Notes in Computer Science. Vol. 3683. Berlin: Springer, 2005: 1225-1231.
  • 26. Li J, Fu AW, He H, et al. Mining risk patterns in medical data. In: Grossman RL, Bayardo R, Bennett K, Vaidya J, editors. Proceedings of the 11th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining; 2005 Aug 21-24; Chicago, Ill. New York: ACM Press, 2005: 770-775.
  • 27. The Sax Institute. Centre for Health Record Linkage. (accessed Jan 2007).
  • 28. Kelman CW, Bass AJ, Holman CD. Research use of linked health data — a best practice protocol. Aust N Z J Public Health 2002; 26: 251-255.
  • 29. Evans JM, MacDonald TM. Misclassification and selection bias in case–control studies using an automated database. Pharmacoepidemiol Drug Saf 1997; 6: 313-318.
  • 30. Pukkala E, Tulenheimo-Silfvast A, Leminen A. Incidence of cancer among women using long versus monthly cycle hormonal replacement therapy, Finland 1994–1997. Cancer Causes Control 2001; 12: 111-115.
  • 31. Sorensen HT, Mellemkjaer L, Olsen JH. Risk of suicide in users of beta-adrenoceptor blockers, calcium channel blockers and angiotensin converting enzyme inhibitors. Br J Clin Pharmacol 2001; 52: 313-318.
  • 32. Australian Government Department of Health and Ageing. National medicines policy 2000.$FILE/nmp2000.pdf (accessed Aug 2006).
  • 33. Best Practice Advocacy Centre. Responsible use of pharmaceuticals. Best practice guidelines. 2006. (accessed Aug 2006).
  • 34. Australian Government Department of Education, Science and Training. National Collaborative Research Infrastructure Strategy (NCRIS). 2005. (accessed Aug 2006).
  • 35. Prime Minister’s Science, Engineering and Innovation Council (PMSEIC) Working Group. From data to wisdom: pathways to successful data management for Australian science. Working group on data for science report to PMSEIC. Canberra: PMSEIC, Dec 2006. (accessed Jan 2007).


remove_circle_outline Delete Author
add_circle_outline Add Author

Do you have any competing interests to declare? *

I/we agree to assign copyright to the Medical Journal of Australia and agree to the Conditions of publication *
I/we agree to the Terms of use of the Medical Journal of Australia *
Email me when people comment on this article

Online responses are no longer available. Please refer to our instructions for authors page for more information.