Predictive validity of the UMAT for medical students’ academic performance

To the Editor: By focusing on the observed low correlations between the Undergraduate Medicine and Health Sciences Admission Test (UMAT) and academic scores at the University of Queensland (UQ), Wilkinson and colleagues1 conclude that the UMAT is a poor predictor of medical student performance. Alternatively, one could focus on the very high mean and low variance grade point average (GPA) scores of the UQ student cohort and conclude that the use of UMAT scores was clearly very successful in identifying a high-performing group of students.

However, before making any conclusions based on these data, we should draw on the vast literature that highlights common problems that significantly affect validity coefficients.2 Range restriction is one such problem, and Wilkinson and colleagues rightly used Thorndyke case 2 to correct for this in their UMAT scores. Although they achieved little change in observed correlation values, a recalculation using the mean standard deviation of the entire UMAT cohort for the period 2005–2009 actually takes the 0.14 correlation between Section 1 and Year 1 GPA to 0.27. Despite this considerable increase, Thorndyke case 2 only accounts for direct range restriction of the predictor (UMAT scores). However, in the context of medical student selection, there are other important sources of error that need to be accounted for in order to better appreciate predictive validity.

First, use of a high cut-off on the high school examination score is very likely to add indirect range restriction on the UQ UMAT scores. This may be further attenuated by not including measures of non-cognitive ability (interviews). Second, selection on cognitive ability tests (UMAT and high school results) also creates range restriction on the largely cognitive criterion (GPA) that would further reduce observed validity correlations. The UQ study illustrates such restriction. Third, the unreliability of the criterion itself needs to be accounted for, especially when more subjective ratings are included, such as ratings of clinical performance.

The problem of relatively small samples in selection studies is well documented.2 Given the evidence of extreme range restriction of predictor and criterion variables used in medical school selection research, more primary studies (followed by meta-analysis) are necessary before accurate conclusions can be made about the use of the UMAT, or of any other predictor used in this high-stakes selection context.