Results: Mean overall UMAT score at entry was 60/100 and mean GPA during university study was 6.1 (range, 1–7), with a correlation coefficient of 0.15 (P = 0.005). This relationship existed only in the first year of university study. For UMAT Section 1 score, the correlation coefficient was 0.14 (P = 0.01); for UMAT Section 2, the correlation coefficient was 0.06 (P = 0.29); and for UMAT Section 3, the correlation coefficient was 0.09 (P = 0.11). UMAT overall score for men (60.2) and women (59.8), and GPA for men (6.1) and women (6.2) were similar. However, men performed better in Section 1 (mean score 61.6 v 61; P = 0.05) and Section 3 (63.2 v 60.7; P < 0.001), whereas women performed better in Section 2 (58.5 v 55.8; P = 0.009). In multivariate analysis, only correlation between GPA and UMAT Section 1 score remained significant but was weak and lasted for 1 year of university study.
Selection of students into medical degree programs is highly visible and competitive. On one hand, there are many more suitable candidates than there are places. This requires the selection process to manage supply and demand in a defensible and transparent manner, usually focusing on academic ability. On the other hand, there is an understandable desire to select medical students who are most likely to become good doctors. This entails considering non-academic factors.1
In Australia and New Zealand, selection of students into medical study directly from high school typically uses a combination of academic performance at school, performance on a standardised test (the Undergraduate Medicine and Health Sciences Admission Test [UMAT]) and an interview. In 2009, 14 universities in Australia and New Zealand used the UMAT as part of their selection processes.
Devised by the Australian Council for Educational Research, the UMAT comprises three parts: Section 1 (logical reasoning and problem solving), Section 2 (understanding people) and Section 3 (non-verbal reasoning).2 In Section 1, students are required to exercise reasoning and problem-solving skills. Section 2 assesses a student’s ability to understand and think about people. Items are based on passages of text representing specific interpersonal situations. Section 3 consists of abstract items that are designed to evaluate a student’s ability to exercise non-verbal reasoning skills.3
There are little published data on the predictive validity of UMAT for medical student academic performance,4 although the impact of coaching and the association with emotional intelligence have been explored.3,5 This lack of data is surprising given the widespread use of UMAT.
The medical degree program at the School of Medicine, University of Queensland, provides for entry through two distinct streams: directly from high school (using the UMAT) and graduate entry (using the Graduate Australian Medical School Admissions Test). Overall, between 2005 and 2009, the program admitted 298 Australian students each year.
The school-leaver entry stream was introduced in 2005 and the number of places in it has increased annually. Most students study through a dual-degree program — usually a combined Bachelor of Science (BSc)–Bachelor of Medicine and Bachelor of Surgery (MB BS), although some students combine their MB BS with a Bachelor of Health Sciences (BHSc) or Bachelor of Arts (BA).
For entry into the school-leaver stream, the university considers only students with the highest possible high school examination score (overall position 1). From these students, it then ranks those who score a minimum of 50/100 in each of three UMAT exam sections by their overall UMAT score, and makes offers accordingly. There is no interview.
Students undergo a range of assessments during the various courses within the 2 years of their first degree. According to university policy, assessment occurs midway through each course and at the end of each course, and uses a mix of assessment methods. The degrees studied are typically structured as 2-unit courses with a study load of 16 units each year.
In the medical degree, students also undergo mid- semester and end-of-semester (written and clinical) exams during the first 2 years of the program, and every 8 weeks at the end of each clinical rotation during the final 2 years of the program. Assessment scores for each course are combined, according to university policy, to generate a grade point average (GPA) on a scale of 1 to 7.
For each student, we tabulated overall and section scores for the UMAT as well as overall GPA calculated at the end of the most recent semester (cumulative since entry to the university). We calculated the Pearson correlation coefficient (PCC) between GPA and overall UMAT score, as well as between UMAT section scores, for all students and for men and women separately. We also examined how the correlation changed over the 4 years of the study program. We constructed a generalised linear model, adjusting to account for the effect of three UMAT sections and sex, with all variables included in the model.
Students with very high academic performance at high school and high UMAT scores typically achieve a high GPA. We therefore explored correction of restriction of range in correlation analysis (using Thorndike case 2 for explicit selection method).6 We present the uncorrected results, as the correction increased correlation by a maximum of only 0.03 (eg, from 0.14 to 0.17 for UMAT Section 1).
We studied data from all 339 students (169 men and 170 women) consecutively enrolled directly from high school at the University of Queensland between 2005 and 2009. Of these, 45 (13%) enrolled in 2005, 29 (9%) in 2006, 34 (10%) in 2007, 73 (22%) in 2008, and 158 (47%) in 2009.
Most students took the BSc–MB BS dual degree (265, 78%), with 15 (4%) taking the BA–MB BS option, 14 (4%) taking the BHSc–MB BS option, and the balance (14%) combining an MB BS with another undergraduate degree such as law or business.
Mean overall UMAT score was 60 (standard deviation [SD], 4.7) and mean GPA attained during the program was 6.1 (SD, 0.61; range, 1–7) (Box 1). The correlation coefficient for these two scores was 0.15 (P = 0.005).
For UMAT Section 1 score, the correlation coefficient was 0.14 (P = 0.01); for UMAT Section 2, the correlation coefficient was 0.06 (P = 0.29); and for UMAT Section 3, the correlation coefficient was 0.09 (P = 0.11) (Box 2).
The statistically significant relationship between UMAT Section 1 score and GPA existed only in the first year of study (Box 3).
There were no significant differences between men and women in overall UMAT score (60.2 v 59.8) or GPA (6.1 v 6.2). However, men performed better than women in Section 1 (mean score, 61.6 v 61; P = 0.05) and Section 3 (mean score, 63.2 v 60.7; P < 0.001), whereas women performed better in Section 2 (mean score, 58.5 v 55.8; P = 0.009).
Only UMAT Section 1 (B = 0.01; P = 0.019) and sex (B = − 0.14; P = 0.04) were statistically significantly correlated with GPA (Box 4). The relationship between UMAT Section 1 score, sex and GPA was statistically significant only in the first year of university study (P < 0.002) (Box 5).
These are the first peer-reviewed, published data reporting predictive validity of UMAT for academic performance at university. Our study shows only weak correlation between UMAT overall score and program GPA. Further, the weak correlation did not persist beyond the first year of university study, and in multivariate analysis, correlation was limited to UMAT Section 1 score.
What might explain this lack of predictive validity? The students in our analysis are, like most students entering medical programs, highly selected and high performing. Predictably, these students perform very well on the UMAT. At university, most of our students also perform very well; the mean GPA in our study was 6.1 out of a possible 7. Similar factors occur when studying other aspects of selection,1,7 and range restriction is an important limitation to this type of analysis. Statistical manipulation to control for this is possible, but we found that the correlation coefficient increased by no more than 0.03. This suggests that range restriction is not a major limitation here. Statistical power in our study was limited by the number of students available, especially in the later years of the study program. Larger studies, including those combining data from several medical schools would be beneficial.
Most students in our study have not yet completed their medical program. It is important to continue and extend our analysis throughout the medical program as it is possible that correlations between UMAT and performance during clinical training may emerge. Further, we have only reported on overall GPA, and have not explored correlation between UMAT scores and individual components of assessment such as knowledge, clinical skills or professionalism. This further research is important and will be done. Long-term follow-up into specialty training and clinical practice would shed more light on the value of UMAT in selection, as its value may become more evident later.
Selection of students into medical programs continues to be controversial, with demand outstripping supply, and with a widespread desire to select those students deemed more likely to become better doctors. UMAT does not seem to have useful validity in terms of predicting academic performance at university. Further research is needed to determine whether UMAT is predictive of performance during clinical training at medical school, in postgraduate training environments, and in clinical practice.
1 Correlation between overall UMAT mean score and undergraduate GPA
2 Correlation (unadjusted and partial) between undergraduate grade point average (GPA) and Undergraduate Medicine and Health Sciences Admission Test (UMAT) score, by UMAT section
3 Correlation (unadjusted and partial) between undergraduate grade point average (GPA) and Undergraduate Medicine and Health Sciences Admission Test (UMAT) Section 1–3 scores, by program year
4 Multivariate relationship between undergraduate grade point average and Undergraduate Medicine and Health Sciences Admission Test (UMAT) score, by UMAT section and sex
5 Multivariate relationship between undergraduate grade point average and Undergraduate Medicine and Health Sciences Admission Test (UMAT) Section 1–3 scores and sex, by program year
Provenance: Not commissioned; externally peer reviewed.
Received 16 September 2010, accepted 16 January 2011
- 1. Ferguson E, James D, Madeley L. Factors associated with success in medical school: systematic review of the literature. BMJ 2002; 324: 952-957.
- 2. Australian Council for Educational Research. UMAT. Undergraduate Medicine and Health Sciences Admission Test. http://umat.acer.edu.au/ (accessed May 2010).
- 3. Carr SE. Emotional intelligence in medical students: does it correlate with selection measures? Med Educ 2009; 43: 1069-1077.
- 4. Mercer A. Selecting medical students: an Australian case study [PhD thesis]. Perth: Murdoch University, 2007.
- 5. Griffin B, Harding DW, Wilson IG, Yeomans ND. Does practice make perfect? The effect of coaching and retesting on selection tests used for admission to an Australian medical school. Med J Aust 2008; 189: 270-273. <MJA full text>
- 6. Wiberg M, Sundström A. A comparison of two approaches to correction of restriction of range in correlation analysis. Pract Assess Research Eval [Internet] 2009; 14 (5). Epub 2009 Mar 18. http://pareonline.net/getvn.asp?v=14&n=5 (accessed Feb 2011).
- 7. Wilkinson D, Zhang J, Byrne GJ, et al. Medical school selection criteria and the prediction of academic performance. Evidence leading to change in policy and practice at the University of Queensland. Med J Aust 2008; 188: 349-354. <MJA full text>
Publication of your online response is subject to the Medical Journal of Australia's editorial discretion. You will be notified by email within five working days should your response be accepted.