Evaluating medicines: let’s use all the evidence

Chris W Kelman, Sallie-Anne Pearson, Richard O Day, C D'Arcy J Holman, Erich V Kliewer and David A Henry
Med J Aust 2007; 186 (5): 249-252.
Published online: 5 March 2007

As a requirement for marketing approval, the benefits and risks of medicines are evaluated in formal clinical trials. However, these trials are not necessarily representative of the way medicines are used by consumers. Currently we rely on voluntary reporting schemes to detect adverse events that may occur after marketing, when a much larger patient population is exposed to a medicine.

We describe here the limitations of the current system of monitoring medicines and propose an updated method that takes advantage of the capabilities of information technology, existing data resources and statistical methodologies. Australia is well provided with quality data to support this approach — an approach that is gaining increasing support overseas.1-3

Evolution of pharmacovigilance

Historically, procedures for establishing the effectiveness of medical treatments have been suboptimal, with many commonly used treatments still awaiting rigorous evaluation.4 The safety of many medicines has largely been taken on faith, on the assumption that claimed benefits outweigh (un)known risks. In the early 1960s, the connection between thalidomide and severe birth defects was observed;5 this was a potent catalyst for clinicians and regulators to consider systems for monitoring drug safety.

Premarketing evaluation

Evaluating new medicines for safety and effectiveness involves three phases of testing before market approval: Phase I is the first time the drug is trialled in humans and tests basic safety; Phase II investigates efficacy; and Phase III involves a confirmatory study of both safety and efficacy in larger patient groups. This approach is designed to minimise risks to participants and costs to industry, while maximising potential to discover the range of drug effects.

On completion of testing, an application for marketing approval is made to the appropriate national regulatory authority; in Australia, this is the Therapeutic Goods Administration (TGA). The authority evaluates the quality of the sponsor company’s research in support of the safety and effectiveness of its product. After approval is given and marketing has commenced, the manufacturer conducts studies for some, but certainly not all, medicines, to refine use and determine comparative cost-effectiveness.

Postmarketing evaluation

Cases reported to the national adverse event reporting systems of more than 70 countries are now included in the World Health Organization’s voluntary reporting scheme (WHO Programme for International Drug Monitoring), which can generate appropriate alerts at short notice. In these countries, global experience is reviewed annually in the early years of drug marketing through the requirement for sponsor companies to provide Periodic Safety Update Reports. The voluntary reporting scheme has reached a high level of sophistication both internationally and in Australia, with our Adverse Drug Reactions Unit,6 however, it has important limitations.

Limitations of the current system
Premarketing evaluation

There are a number of reasons why premarketing evaluation cannot fully explore the effects of new medicines.7

  • Trial exposure is short. Most clinical trials are conducted for a period of no more than 12 months. While this may be acceptable for medicines used briefly for acute conditions, it is not adequate for medicines expected to be used continuously by patients with chronic disorders.

  • Power to detect uncommon outcomes is low. The combination of small patient numbers and limited trial duration has the consequence that “events occurring as frequently as 1 in 1000 are predictably undetected”.8

  • Subject selection is biased. Trials of medicines for particular conditions are most conveniently performed in specialist centres, where study subjects are unlikely to be representative of the general population (eg, subjects are often younger and healthier).

  • Drug interactions are not examined. To simplify premarket testing, trial subjects are usually excluded if they are taking other medicines or undergoing other treatment. This is particularly a problem when attempting to generalise findings to older people, 20%–40% of whom are taking more than five drugs concurrently.9

  • Risk is underestimated. Currently accepted best practice for clinical trials is the “intention to treat” approach, which assumes that all subjects allocated at random to the “treatment” arm of a study actually take all of the prescribed medicine. In fact, a proportion of subjects will generally not comply and thus are not exposed to either risks or benefits of the treatment. This will lead to reduced estimates of adverse event rates, and an optimistic side effect profile may emerge.

Postmarketing evaluation

The reporting systems for adverse drug reactions in most countries are voluntary and depend on individuals becoming suspicious that an event is related to a treatment, and then actually reporting it. Identifying medicine-related adverse events is straightforward for unusual reactions such as severe skin rashes or liver failure, and those with a clear temporal relationship, but difficult for common events with well known causes, such as heart attacks and stroke. In either case, risk quantification (rather than mere signal generation) requires measurement of incidence, as well as strength of association between exposure and outcome — such information is not available from adverse event reporting databases.

A recent example of the need for formal ongoing evaluation has been reported by McEwen et al.10 The disease-modifying antirheumatic drug leflunomide was approved in Australia in 1999 for use as monotherapy for rheumatoid arthritis. Actual usage has proven to be different, and the incidence of pancytopenia after use of leflunomide in combination with methotrexate now appears to be more than 1/1000 per person-year. Preregistration examination of such concomitant use was restricted to only 30 patients. While the voluntary reporting scheme has now alerted regulators to this problem, this example emphasises that neither the manner of clinical use nor real-world safety can be assumed when approving new medicines. A routine system for the ongoing examination of actual postmarketing experience seems a wise precaution.

While useful postmarketing information about medicines is sometimes generated by formal studies (a highly desirable but costly approach), observational data about these medicines are already available in Australia as a by-product of administrative processes. These data describe large patient populations and provide a promising new opportunity for monitoring safety of medicines. However, a formal system for examining these data is yet to be implemented, for a number of reasons, including concerns about patient privacy, lack of political will, and restrictions in access to and linkage of the various data collections.

A new approach

The urgent need for increased medicines surveillance has been recently argued by international bodies, such as the International Society for Pharmacoepidemiology, and considered as a component of the new requirements for postmarketing epidemiological studies by the US Food and Drug Administration and the European Agency for the Evaluation of Medicinal Products. Similar discussions are underway locally as legislation is developed for the upcoming Australia New Zealand Therapeutic Products Authority.11

An optimal postmarketing surveillance system should assess all available information about the performance of medicines, drawing on worldwide pre- and postmarketing trials and voluntary reporting, with the addition of findings from regular, systematic and ongoing evaluation using linked observational data.1,7,8,12-14 A number of centres in several countries are already performing high-class pharmacoepidemiological studies of this type.15-19

The utility of linked observational data is well illustrated by the aftermath of the worldwide withdrawal of the cyclooxygenase-2 inhibitor rofecoxib (Vioxx) in response to concerns that the medicine increased heart attack risk. The international literature has since carried a series of over 20 controlled observational and cohort studies evaluating the excess risk associated with use of rofecoxib and related drugs in the United States, Canada and Europe.20 All but two studies employed linked observational data (or linked electronic medical records), permitting rapid evaluation of potential harms. Had these data been analysed cumulatively as soon as they became available (as would occur in a routine surveillance system), rofecoxib might have been withdrawn earlier. A recent meta-analysis of controlled observational studies has indicated that diclofenac has a similar risk profile.21

Australia, as a fortuitous consequence of a complex multi-jurisdictional health care funding system, is replete with observational health care data in linkable databases. They describe the vast majority of health care encounters across the nation, including doctor visits, medicine dispensing, hospital admissions, deaths, and various disease registries. Recent work has shown the practicality of bringing these data together for active monitoring of the outcomes of new drugs while they are being used in the “real world”.22-24 Specific methodologies have been developed to support pharmacoepidemiological analysis of these large datasets.25,26 Encouraging progress has been made over the past few years to improve access to these health datasets. A major linked data resource is now available in Western Australia and a similar resource is being developed in New South Wales.27 To ensure ethical and appropriate use of these data, a nationally accepted protocol has been designed to guarantee both strong privacy protection and accurate linkage.28 While these data have some known limitations and biases, they portray actual practice in use of medicines — and their long-term consequences. With the adoption of appropriate case-inclusion criteria, most of the biases in these data can be controlled.29 Although there are as yet no overseas examples of “routine” medicines monitoring, there have been useful studies based on ad-hoc linkages of medicines and health-outcomes data.30,31

Our preferred approach would support the stated regulatory goals of Australia’s National Medicines Policy.32,33 We propose a staged introduction, incorporating the following features:

  • Establish a pharmacoepidemiology data analysis centre, with staff trained in data management, epidemiology, statistics, and, in particular, the use of linked data. The centre would require start-up funding but should ultimately be supported by registration fees from medicine manufacturers or sponsors. It would logically be based within the TGA, but the work could be contracted to external bodies. It may be appropriate to run a pilot program based on data from WA.

  • All applications for new chemical entities would continue to be evaluated by the TGA; however, initial provisional approval would provide a “Caution – New Medicine” rating for the drug and have expanded requirements for periodic review.

  • Periodic review would be based on accumulated adverse drug reaction reports, recent drug studies, and commissioned Australian observational studies using the linked databases.

  • Provisional approval could be upgraded once significant market experience is gained; for example, a medicine could be given a “gold” safety rating after achieving an agreed benchmark.

  • Progressive assessment of existing medicines could be introduced as evaluation resources and expertise become available.

  • Timely provision of appropriate observational data should be secured under agreements with federal and state government data custodians. These data should be linked according to accepted protocols to ensure adequate privacy protection.

  • Provisionally registered medicines should be eligible for Pharmaceutical Benefits Scheme listing to ensure sufficient uptake for adequate postmarketing evaluation.


We call for more practice-based evidence, to ensure that medicines earn their reputation in the marketplace. A conservative approach should be adopted until sufficient evidence is available to provide confidence in a medicine’s safety and efficacy. Granting a provisional safety rating for initial approval is consistent with this philosophy and allows experience gained over time to be incorporated into the ongoing evaluation of all medicines through the process of periodic review.

Australia is ideally placed to introduce an updated 21st century pharmacovigilance system. Commonwealth infrastructure support is available for this work,34,35 and the system could use and benefit from information that has been dutifully collected for many years. Such a system could provide a model for other countries to follow.

  • Chris W Kelman1
  • Sallie-Anne Pearson2
  • Richard O Day2
  • C D'Arcy J Holman3
  • Erich V Kliewer4,5
  • David A Henry6

  • 1 National Centre for Epidemiology and Population Health and ANU Medical School, Australian National University, Canberra, ACT.
  • 2 University of New South Wales, Sydney, NSW.
  • 3 Centre for Health Services Research, University of Western Australia, Perth, WA.
  • 4 Epidemiology and Cancer Registry, CancerCare Manitoba, Winnipeg, Manitoba, Canada.
  • 5 Department of Community Health Sciences, University of Manitoba, Winnipeg, Manitoba, Canada.
  • 6 School of Medicine and Public Health, University of Newcastle, Newcastle, NSW.



We wish to thank Associate Professor Andrea Mant, Dr John McEwen, Dr James Harris and Dr Tom Faunce for their useful comments and encouragement. We also appreciate the support of stakeholders, including industry and consumers, for this proposal.

Competing interests:

Richard Day sits on several advisory boards (mostly related to treating rheumatic disorders with biological agents and anti-inflammatory drugs) that meet ad hoc to review data about new medicines and to give advice regarding their clinical uses. Honoraria are received for these activities, which are unrelated to his contribution to this article.

  • 1. Baciu AS, Stratton K, Burke SP, editors; Committee on the Assessment of the US Drug Safety System. The future of drug safety: promoting and protecting the health of the public. Washington, DC: The National Academies Press, 2006. (accessed Jan 2007).
  • 2. Britton A, McKee M, Black N, et al. Choosing between randomised and non-randomised studies: a systematic review. Health Technol Assess 1998; 2: i-iv and 1-124.
  • 3. Ray WA, Stein CM. Reform of drug regulation — beyond an independent drug-safety board. N Engl J Med 2006; 354: 194-201.
  • 4. Moseley JB, O’Malley K, Petersen NJ, et al. A controlled trial of arthroscopic surgery for osteoarthritis of the knee. N Engl J Med 2002; 347: 81-88.
  • 5. McBride WG. Thalidomide and congenital abnormalities. Lancet 1961; 2: 1358.
  • 6. Australian Government Department of Health and Ageing. Australian adverse drug reactions bulletin 2006. (accessed Aug 2006).
  • 7. Hunter D. First, gather the data. N Engl J Med 2006; 354: 329-331.
  • 8. Strom BL. How the US drug safety system should be changed. JAMA 2006; 295: 2072-2075.
  • 9. McLean AJ, Le Couteur DG. Aging biology and geriatric clinical pharmacology. Pharmacol Rev 2004; 56: 163-184.
  • 10. McEwen J, Purcell PM, Hill RL, et al. The incidence of pancytopenia in patients taking leflunomide alone or with methotrexate. Pharmacoepidemiol Drug Saf 2007; 16: 65-73 [published online ahead of print: 24 Apr 2006]. DOI: 10.1002/pds.1236.
  • 11. Australian Government Department of Health and Ageing. Submissions sought on proposed Australia New Zealand Therapeutic Products Authority regulatory scheme. 23 May 2006. (accessed Sep 2006).
  • 12. Platt R, Ommaya A. A beneficial side effect of the Medicare drug benefit. N Engl J Med 2005; 353: 2742-2743.
  • 13. Nelson MR, Tonkin AM, Cicuttini FM, McNeil JJ. COX-2 inhibitors: exemplars of the drug-safety conundrum [editorial]. Med J Aust 2005; 182: 262-263. <MJA full text>
  • 14. Laupacis A, Paterson M, Mamdani M, et al. Gaps in the evaluation and monitoring of new pharmaceuticals: proposal for a different approach. CMAJ 2003; 169: 1167-1170.
  • 15. Jick H, Vasilakis C, Weinrauch LA, et al. A population-based study of appetite-suppressant drugs and the risk of cardiac-valve regurgitation. N Engl J Med 1998; 339: 719-724.
  • 16. Evans JM, MacDonald TM. Record-linkage for pharmacovigilance in Scotland. Brit J Clin Pharmacol 1999; 47: 105-110.
  • 17. Mangano DT, Tudor IC, Dietzel C; Multicenter Study of Perioperative Ischaemia Research Group; Ischaemia Research and Education Foundation. The risk associated with aprotinin in cardiac surgery. N Engl J Med 2006; 354: 353-365.
  • 18. Vlahakes GJ. The value of phase 4 clinical testing. N Engl J Med 2006; 354: 413-415.
  • 19. PHARMO Institute for Drug Outcomes Research. PHARMO report. Innovations in pharmacotherapy. Utrecht: PHARMO Institute, 2000. .PDF (accessed Dec 2006).
  • 20. Ardoin SP, Sundy JS. Update on nonsteroidal anti-inflammatory drugs. Curr Opin Rheumatol 2006; 18: 221-226.
  • 21. McGettigan P, Henry D. Cardiovascular risk and inhibition of cyclooxygenase: a systematic review of the observational studies of selective and nonselective inhibitors of cyclooxygenase 2. JAMA 2006; 296: 1633-1644.
  • 22. Brameld KJ, Thomas MA, Holman CD, et al. Validation of linked administrative data on end-stage renal failure: application of record linkage to a “clinical base population”. Aust N Z J Public Health 1999; 23: 464-467.
  • 23. Holman CD, Bass AJ, Rouse IL, Hobbs MS. Population-based linkage of health records in Western Australia: development of a health services research linked database. Aust N Z J Public Health 1999; 23: 453-459.
  • 24. Stryer D, Tunis S, Hubbard H, Clancy C. The outcomes of outcomes and effectiveness research: impacts and lessons from the first decade. Health Serv Res 2000; 35: 977-993.
  • 25. Chen J, He H, Li J, et al. Representing association classification rules mined from health data. In: Khosla R, Howlett RJ, Jain LC, editors. Knowledge-based intelligent information and engineering. Lecture Notes in Computer Science. Vol. 3683. Berlin: Springer, 2005: 1225-1231.
  • 26. Li J, Fu AW, He H, et al. Mining risk patterns in medical data. In: Grossman RL, Bayardo R, Bennett K, Vaidya J, editors. Proceedings of the 11th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining; 2005 Aug 21-24; Chicago, Ill. New York: ACM Press, 2005: 770-775.
  • 27. The Sax Institute. Centre for Health Record Linkage. (accessed Jan 2007).
  • 28. Kelman CW, Bass AJ, Holman CD. Research use of linked health data — a best practice protocol. Aust N Z J Public Health 2002; 26: 251-255.
  • 29. Evans JM, MacDonald TM. Misclassification and selection bias in case–control studies using an automated database. Pharmacoepidemiol Drug Saf 1997; 6: 313-318.
  • 30. Pukkala E, Tulenheimo-Silfvast A, Leminen A. Incidence of cancer among women using long versus monthly cycle hormonal replacement therapy, Finland 1994–1997. Cancer Causes Control 2001; 12: 111-115.
  • 31. Sorensen HT, Mellemkjaer L, Olsen JH. Risk of suicide in users of beta-adrenoceptor blockers, calcium channel blockers and angiotensin converting enzyme inhibitors. Br J Clin Pharmacol 2001; 52: 313-318.
  • 32. Australian Government Department of Health and Ageing. National medicines policy 2000.$FILE/nmp2000.pdf (accessed Aug 2006).
  • 33. Best Practice Advocacy Centre. Responsible use of pharmaceuticals. Best practice guidelines. 2006. (accessed Aug 2006).
  • 34. Australian Government Department of Education, Science and Training. National Collaborative Research Infrastructure Strategy (NCRIS). 2005. (accessed Aug 2006).
  • 35. Prime Minister’s Science, Engineering and Innovation Council (PMSEIC) Working Group. From data to wisdom: pathways to successful data management for Australian science. Working group on data for science report to PMSEIC. Canberra: PMSEIC, Dec 2006. (accessed Jan 2007).


remove_circle_outline Delete Author
add_circle_outline Add Author

Do you have any competing interests to declare? *

I/we agree to assign copyright to the Medical Journal of Australia and agree to the Conditions of publication *
I/we agree to the Terms of use of the Medical Journal of Australia *
Email me when people comment on this article

Responses are now closed for this article.