Is the UCAT appropriate for selecting undergraduate medical students?

Non‐academic personal qualities are also desirable in doctors: higher mental abilities, empathy, ethics, creativity

Medicine is an enduringly popular career choice for several reasons, including the desire to heal the sick and to improve people’s health, but for many, no doubt, also the prospect of guaranteed employment with a satisfactory income and the elevated social status enjoyed by doctors. The net result is that for most medical schools there have been, and still are, far more applicants than places available.1,2,3 This poses an annual selection dilemma.

The selection challenge has been met in a variety of ways. For many years, most medical schools chose the simple and straightforward option of raising the academic threshold for entry so that the number of eligible applicants matched the number of places available for the next intake.2 Certainly, high prior academic achievement does predict success in medical school,4 but several published studies have questioned the link between academic prowess and the skills required of a good doctor.2,5 Further, using prior academic achievement as the principal selection criterion disadvantages people from lower socio‐economic groups and those who attend less academically focused schools.1 This works against the widening of access that many educational organisations encourage, or even prescribe.6

In the 1970s, the newly established medical school at the University of Newcastle sought to reduce the emphasis on prior academic achievement. A battery of psychometric tests was developed to measure non‐academic personal qualities considered desirable in future doctors: higher mental abilities, empathy, ethics, creativity.7 These tests were subsequently further developed, and from 1997 were gradually adopted by other medical schools in Australia and New Zealand.8

In 1997, the test battery, renamed the Undergraduate Medicine and Health Science Admissions Test (UMAT), was offered commercially by the Australian Council for Education Research (ACER) on behalf of the user schools. However, by the year 2000 its content had changed from that originally used in Newcastle;7,9 cognitive skills measurement (logical reasoning, problem solving and non‐verbal reasoning) were now favoured over unspecified non‐cognitive qualities (understanding people).

In 2019, the pen‐and‐paper UMAT was discontinued by the UMAT users’ consortium and replaced by the online University Clinical Aptitude Test (UCAT). UCAT comprises four cognitive skills subtests and a situational judgement test purported to measure non‐cognitive qualities such as “integrity, perspective taking, team involvement, resilience and adaptability”,10 although most situational judgement test validity studies have assessed non‐specific outcomes, such as “performance in general practice… foundation training as a junior doctor and medical school admissions”.11

The authors of the research study reported in this issue of the MJA12 compared data for 2018 intake applicants, the last to sit the UMAT, with those for the 2019 intake, the first to sit the UCAT. They found that male applicants, and those from higher socio‐economic status areas and major cities, performed better on both the UMAT and UCAT than other applicants.

Another finding of the study, consistent with other reports,13 is that coaching (mainly undertaken by students from the groups who already do well) had a negligible effect on the cognitive test performance of those selected for interview. This finding suggests that medical schools could let applicants know that the advantage afforded by expensive training is questionable.

Further care is needed in distinguishing between test preparation and unnecessary (and costly) coaching. The burden of preparation required by the UCAT system is considerable. Applicants need to familiarise themselves with the computer‐based application and testing system, and the instructions also make it clear that those who have not completed the large sets of practice items are likely to be at a disadvantage.10 Moreover, the specific instructions for the situational judgement test recommend that applicants refer to two extensive and different guidelines for Australian and New Zealand doctors,10 which, however, are likely to be of little assistance with the questions in the test.

The findings of Griffin and her colleagues12 should ring alarm bells for medical schools. The UCAT clearly favours some groups of applicants. Further, the UCAT predominantly assesses cognitive skills, with scant focus on other personal qualities important to medical practice. Assessment of these qualities might provide a more even playing field for all applicants.

Finally, in light of the findings of Griffin and colleagues12 — not unexpected given the content of the UCAT — one important question needs to be answered: does the test actually measure the qualities that medical schools wish for future doctors? The inclusion of “clinical” in “UCAT” suggests that it does. If not, one interpretation could be that institutions employ the test merely as an administrative convenience to reduce the large number of academically qualified applicants, with the advantage that it costs the medical school nothing in terms of effort or expense.3