In the past decade, some 15 000 students entered Australian medical schools, and in the United Kingdom, four times that number were admitted. Such a large number should imply that much is known about what to select on, how to select and whom to select. The sad reality is that surprisingly little is known. Instead, strongly held opinions are rife, inertia predominates, and change occurs more because of necessity, external pressure, political force or mere whim, than because of coherent evidence-based policy or theorising. Selection sometimes seems more to ensure the correct number of entrants on day one, than to identify those best suited to the course and profession. As if to illustrate the problem, the University of Adelaide recently reduced its emphasis on selection interviews, the University of Sydney extended its use of interviews, the University of Queensland may be ending interviews, and a meta-analysis in Medical Teacher suggested that selection interviews have only “modest” predictive validity and “little” or “limited” practical value.1 However, interviews differ in many ways, and although the meta-analysis found no moderating effect of factors such as interview method, structure, training, or scoring, some forms of interview may still be valid, as found outside of medicine,2 particularly for situational interviewing.3

An innovation in selection is written tests of “cognitive aptitude”, such as the Undergraduate Medicine and Health Sciences Admission Test and the Graduate Australian Medical School Admissions Test (GAMSAT) in Australia, and the BioMedical Admissions Test, GAMSAT and the UK Clinical Aptitude Test in the UK — tests whose enthusiastic adoption belies the absence of published evidence on predictive validity.4 In this issue of the Journal, an important article by Groves et al5 evaluates GAMSAT, used in nine Australian and four UK universities and which claims to measure “the mastery and use of concepts in basic science as well as the acquisition of more general skills in problem solving, critical thinking and writing”.6,7 The study by Groves et al is not large, but like another small study,8 addresses the predictive validity of GAMSAT, as well as further questioning the value of interviews.

Cognitive aptitude tests4 measure either intellectual ability, general mental ability or intelligence (all effectively synonymous), as in GAMSAT’s first section, or academic achievement or attainment (typically with specific scientific content, as in GAMSAT’s third section, “Reasoning in Biological and Physical Sciences”). Although intellectual ability does predict outcome in primary and secondary schools, its prediction of university outcome is poor, tertiary-level performance depending more on motivation and specific knowledge. Consequently, achievement measures, such as A-levels in the UK, are more effective predictors of medical school outcome4,9,10 than are measures of intellectual ability, which predict little10 — a result known since the 1970s.11 For GAMSAT, Groves et al found a significant correlation with Year 2 examination results only for the biological and physical sciences section assessing achievement, and not for the sections assessing “more general skills in problem solving, critical thinking and writing”.5,6

It has to be acknowledged that even measures of achievement are not strong indicators of medical school outcome, with much variance still to be explained.9 A major problem for people developing and evaluating selection criteria is the lack of adequate outcome measures reflecting the broad and complex day-to-day behaviour of doctors. A rare and important exception is the study by Papadakis et al,12 who found professional misconduct was related both to unprofessional behaviour in medical school and to poor academic achievement before and during medical training. In medical school and beyond, most measures of competence assess knowledge, whereas being a competent, safe and effective doctor probably depends to an equal extent on behaviour, attitudes and approaches. Adequate knowledge is a necessary part of good medicine, but is far from sufficient for competent practice. Without robust, objective and valid measures of other professional skills to complement our robust measures of knowledge, any post-hoc assessment of selection criteria will be lopsided. The study by Groves et al is significant for including clinical reasoning skill as another outcome measure, but here there was a negative correlation with GAMSAT.5

Students from only two universities were included in the study by Groves et al, and the authors comment, “there is an urgent need for collaborative studies to explore these issues further”.5 Inadequate numbers of patients are the bane of clinical trials, so collaborations between hospitals are now standard when testing new drugs. Large-scale evaluations of medical education should be relatively easy, but few studies have been done. Randomised controlled trials (RCTs) should surely also have a place in assessing selection methods, but reasons are too easily found for them being impractical or unethical, despite being the ideal way to answer questions on the validity of novel selection procedures. If RCTs are ethical when assessing the effectiveness of drugs given to patients, they are surely also ethical for assessing the efficacy of tests used for selecting the doctors giving those drugs to patients.

A more general issue raised by the very scarcity of studies like that of Groves et al is why, despite educationalists proliferating in medical schools, so little is known of what does or does not work in selection. The psychologist William James, more than a century ago, characterised his own nascent discipline as “A string of raw facts; a little gossip and wrangle about opinions; a little classification and generalization on the mere descriptive level [but] not a single proposition from which any consequence can causally be deduced”, concluding, “This is no science, it is only the hope of a science”.13 Recently, Schuwirth and van der Vleuten were equally critical of much medical education research, with its often “poorly performed or poorly reported” studies, problems that need solving “if the specialty wants to be taken seriously”.14 Selection encapsulates most of the important questions that medical education faces. The problems are not insoluble, and the intellectual tools mostly exist. The problem is in finding the collaborative will to solve them, and to confront negative answers when they arise.