To the Editor: We read with interest the article by Koczwara and colleagues proposing a national exit examination for all Australian medical school graduates.1 It is refreshing to see interest in educational outcomes, a distinctly different trend from earlier reforms that shifted curricular focus from content to the learning process, exemplified by problem-based learning (PBL). Although the early process-focused programs were based on sound pedagogy current at their time, their educational outcomes have been relatively disappointing, with marginal or no demonstrable improvements in knowledge structures, clinical skills, or generic capabilities such as self-direction.2 Rather than an indictment of PBL, the results may reflect what was missing in those programs: explicit focus on educational outcomes, alignment of assessments with outcomes, and attention to the learning environment.

There is widespread agreement on the outcomes desired by medical schools. They include teamwork, effective communication, critical evaluation and reflective practice, as well as more traditional outcomes.3 Unfortunately, assessment methods have been slow to match curricular reforms, as these outcomes require new approaches, such as group and assignment work, peer assessment and portfolio examination, which are only now emerging in Australia.4 A national exit examination for Australian graduates is unlikely to adequately measure this range of outcomes.

While Koczwara and colleagues recognise that a national examination “might need to include a clinical component” and “would necessarily entail the explicit statement of professional values and expectations”, they support a multiple-choice question examination, suggesting such performance “can correlate well with clinical skills and future performance in multiple disciplines”.1 While this might “complement rather than replace” other medical school assessments, the message sent by its failure to address personal and professional attributes would be invidious.

In recognition of the limitations of multiple-choice questions, national examinations in North America now include a clinical component.5 This has major resource implications and, like all high-stakes assessments, uses relatively reliable, but much less valid measures — standardised or simulated clinical encounters. This is at odds with current initiatives in medical schools, which are moving to clinical assessments with higher face validity, such as the mini-CEX (mini-clinical examination exercise).6 It would be near impossible to adequately measure generic outcomes, such as teamwork, communication and reflection, in a single national examination. Koczwara et al recognise that insufficient attention has been paid to ensuring that achievement of educational outcomes is embedded in reform of medical curricula. Their solution is overly simple for a highly complex set of issues.

To the Editor: Foreign medical graduates sitting for the Australian Medical Council (AMC) examinations are expected to achieve a standard comparable to that of Australian medical students. If we do not measure the level of knowledge and problem-solving ability nationally, there is no reasonable basis for presuming that the AMC examination is fair. A uniform examination-based assessment should be passed by all potential medical practitioners before registration in Australia. I would be in favour of a national exit examination.1

To the Editor: The recent article by Koczwara and colleagues proposing a national exit examination for medical students1 prompted me to recall a 1970 trial of a national examination in surgery.2 Seven of the then eight medical schools participated. Interstate differences were wide for some questions; separate analyses of the 15 teaching hospitals showed variation to be even wider within a university than between universities. Do local differences still undermine the validity of a national examination?

It is still uncertain what is actually tested by questions on paper. Context-free, standardised questions and answers assume clinical teaching and practice are standardised. However, clinical teachers writing examination items know well that many colleagues choose the “wrong” answer. Consensus may be imposed on those who differ. Teachers then forget their disparity, but expect candidates to choose only one “true” answer!

Clinical performance is interactive, multifaceted, situation-specific and value-laden. Complex judgement and decision-making cannot be measured by ticking predetermined boxes. Clinical experts develop personal subsets of specific evidence, and seek different data for diagnosis and management. But separate, context-free tasks, as in an objective structured clinical examination (OSCE), naively assume they do not.3 OSCE even standardises scoring; examiners become recorders rather than assessors.

Reductionist standardisation reflects a pseudoscientific attempt to apply objectivity, consistency and precision to complex human interactions around incomplete evidence and uncertainty, approximations, judgements, trade-offs and locally-determined decisions.4 Internal consistency of measuring instruments does not confer external validity in real world clinical practice.

The inexorable growth of medical knowledge and technological opportunities continuously expands “what every doctor should know”. Medical learning today embraces a mix of science-based, problem-based and work-based learning experiences, with community-based experiences5 increasingly included. In their recent article, Koczwara and colleagues identified gaps in oncology education,2 a field ranging from molecular processes to euthanasia. Is oncology managed and taught consistently across different medical schools and hospitals across Australia? Which facets would you test in a national exit examination?6

Clinical performance today includes patient/person management, case management, health system management and self-management. Clinicians can judge student performance consistently. However, formal clinical examinations lack the range of cases and open-ended time that allow examiners to observe all the patient-care skills espoused by today’s curricula.7 Assessment of performance in case management and procedural skills within hospital practice can be conducted simpy by paired examiners.8

In reply: We appreciate the insightful responses to our proposal.1 Lawson-Smith alludes to one of the most significant justifications for a national examination — fairness. One cannot expect foreign medical graduates to attain a standard comparable with that of Australian graduates if we do not measure this standard. We propose that we owe fairness not only to foreign graduates coming to Australia, but also to Australian medical students who have a right to confidently expect an education that will lead to similar knowledge, skills and attitudes, irrespective of which university they choose. And finally, we owe fairness to society, which would also expect the same standards of graduates irrespective of where they come from. Unless we consider what are acceptable standards, we operate within an environment where standards of outcome differ from place to place and as we do not measure outcomes uniformly, we do not know how they differ nor have a system to address potential deficiencies.

McNeil and Grimm point out that medical education has focused less on outcomes and more on process, and raise concerns that assessment methods lag in the sophistication necessary to assess outcomes. We wonder whether the reason for this lack of sophistication lies in the relative lack of interest in this field, and also in the lack of agreement on what constitutes acceptable outcomes. The process of outcome assessment is indeed complex, and the first step is national consensus on appropriate outcomes to be uniformly achieved.

McNeil and Grimm also point out that some of the desirable outcomes, such as teamwork, effective communication, critical evaluation and reflective practice, may be harder to test than medical knowledge. While this is certainly the case, reliable assessment methods do exist, such as the Moral Judgment Interview2 and Rest’s Defining Issues test.3 The mini-CEX (mini-clinical examination exercise) that McNeil and Grimm refer to, allows testing of judgement, professionalism, communication, organisation and efficiency.4

Cox questions whether a written examination can test the complex judgement and decision-making process that is better tested in the clinical setting by experienced clinicians. We propose that the national examination is not meant to replace clinician-based assessments and ongoing learning and feedback. Furthermore, a national examination does not need to be conducted only in the written form. Specialty exams already conducted nationally incorporate a clinical element and are conducted in multiple locations. The main objective of a national examination is to ensure the comparison of outcomes against agreed acceptable national standards. This objective should not impose specific limitations on the structure of the examination.

Cox warns of the risk of “reductionist standardisation” and asks whether oncology is taught consistently across different medical schools in Australia. We acknowledge that uncertainty is ever present in medical decision-making, but remain hopeful that there exist core knowledge, skills and attitudes that patients can expect and that can provide a foundation for national standards. Oncology is not taught consistently across different medical schools in Australia today. While its mode of delivery may differ, we propose that its outcomes should not. And we hope that agreement on national outcome standards and a national process of assessment of these outcomes will be a first step in achieving that objective.