Connect
MJA
MJA

Predictive validity of the Undergraduate Medicine and Health Sciences Admission Test for medical students’ academic performance

David Wilkinson, Jianzhen Zhang and Malcolm Parker
Med J Aust 2011; 194 (7): 341-344.
Published online: 4 April 2011

Selection of students into medical degree programs is highly visible and competitive. On one hand, there are many more suitable candidates than there are places. This requires the selection process to manage supply and demand in a defensible and transparent manner, usually focusing on academic ability. On the other hand, there is an understandable desire to select medical students who are most likely to become good doctors. This entails considering non-academic factors.1

In Australia and New Zealand, selection of students into medical study directly from high school typically uses a combination of academic performance at school, performance on a standardised test (the Undergraduate Medicine and Health Sciences Admission Test [UMAT]) and an interview. In 2009, 14 universities in Australia and New Zealand used the UMAT as part of their selection processes.

Devised by the Australian Council for Educational Research, the UMAT comprises three parts: Section 1 (logical reasoning and problem solving), Section 2 (understanding people) and Section 3 (non-verbal reasoning).2 In Section 1, students are required to exercise reasoning and problem-solving skills. Section 2 assesses a student’s ability to understand and think about people. Items are based on passages of text representing specific interpersonal situations. Section 3 consists of abstract items that are designed to evaluate a student’s ability to exercise non-verbal reasoning skills.3

There are little published data on the predictive validity of UMAT for medical student academic performance,4 although the impact of coaching and the association with emotional intelligence have been explored.3,5 This lack of data is surprising given the widespread use of UMAT.

Methods
Setting

The medical degree program at the School of Medicine, University of Queensland, provides for entry through two distinct streams: directly from high school (using the UMAT) and graduate entry (using the Graduate Australian Medical School Admissions Test). Overall, between 2005 and 2009, the program admitted 298 Australian students each year.

The school-leaver entry stream was introduced in 2005 and the number of places in it has increased annually. Most students study through a dual-degree program — usually a combined Bachelor of Science (BSc)–Bachelor of Medicine and Bachelor of Surgery (MB BS), although some students combine their MB BS with a Bachelor of Health Sciences (BHSc) or Bachelor of Arts (BA).

Selection of medical students

For entry into the school-leaver stream, the university considers only students with the highest possible high school examination score (overall position 1). From these students, it then ranks those who score a minimum of 50/100 in each of three UMAT exam sections by their overall UMAT score, and makes offers accordingly. There is no interview.

Assessment during university study

Students undergo a range of assessments during the various courses within the 2 years of their first degree. According to university policy, assessment occurs midway through each course and at the end of each course, and uses a mix of assessment methods. The degrees studied are typically structured as 2-unit courses with a study load of 16 units each year.

In the medical degree, students also undergo mid- semester and end-of-semester (written and clinical) exams during the first 2 years of the program, and every 8 weeks at the end of each clinical rotation during the final 2 years of the program. Assessment scores for each course are combined, according to university policy, to generate a grade point average (GPA) on a scale of 1 to 7.

Analysis

For each student, we tabulated overall and section scores for the UMAT as well as overall GPA calculated at the end of the most recent semester (cumulative since entry to the university). We calculated the Pearson correlation coefficient (PCC) between GPA and overall UMAT score, as well as between UMAT section scores, for all students and for men and women separately. We also examined how the correlation changed over the 4 years of the study program. We constructed a generalised linear model, adjusting to account for the effect of three UMAT sections and sex, with all variables included in the model.

The statistically significant difference of the means was set at 0.05. SPSS, version 17 (SPSS Inc, Chicago, Ill, USA) was used for data analysis.

Students with very high academic performance at high school and high UMAT scores typically achieve a high GPA. We therefore explored correction of restriction of range in correlation analysis (using Thorndike case 2 for explicit selection method).6 We present the uncorrected results, as the correction increased correlation by a maximum of only 0.03 (eg, from 0.14 to 0.17 for UMAT Section 1).

Ethics

Ethics approval was received from the University of Queensland Behavioural and Social Sciences Ethical Review Committee. No funding was sought for our study.

Results
Participant characteristics

We studied data from all 339 students (169 men and 170 women) consecutively enrolled directly from high school at the University of Queensland between 2005 and 2009. Of these, 45 (13%) enrolled in 2005, 29 (9%) in 2006, 34 (10%) in 2007, 73 (22%) in 2008, and 158 (47%) in 2009.

Most students took the BSc–MB BS dual degree (265, 78%), with 15 (4%) taking the BA–MB BS option, 14 (4%) taking the BHSc–MB BS option, and the balance (14%) combining an MB BS with another undergraduate degree such as law or business.

UMAT score and GPA

Mean overall UMAT score was 60 (standard deviation [SD], 4.7) and mean GPA attained during the program was 6.1 (SD, 0.61; range, 1–7) (Box 1). The correlation coefficient for these two scores was 0.15 (P = 0.005).

For UMAT Section 1 score, the correlation coefficient was 0.14 (P = 0.01); for UMAT Section 2, the correlation coefficient was 0.06 (P = 0.29); and for UMAT Section 3, the correlation coefficient was 0.09 (P = 0.11) (Box 2).

The statistically significant relationship between UMAT Section 1 score and GPA existed only in the first year of study (Box 3).

Sex

There were no significant differences between men and women in overall UMAT score (60.2 v 59.8) or GPA (6.1 v 6.2). However, men performed better than women in Section 1 (mean score, 61.6 v 61; P = 0.05) and Section 3 (mean score, 63.2 v 60.7; P < 0.001), whereas women performed better in Section 2 (mean score, 58.5 v 55.8; P = 0.009).

Multivariate analysis

Only UMAT Section 1 (B = 0.01; P = 0.019) and sex (B =  0.14; P = 0.04) were statistically significantly correlated with GPA (Box 4). The relationship between UMAT Section 1 score, sex and GPA was statistically significant only in the first year of university study (P < 0.002) (Box 5).

Discussion

These are the first peer-reviewed, published data reporting predictive validity of UMAT for academic performance at university. Our study shows only weak correlation between UMAT overall score and program GPA. Further, the weak correlation did not persist beyond the first year of university study, and in multivariate analysis, correlation was limited to UMAT Section 1 score.

What might explain this lack of predictive validity? The students in our analysis are, like most students entering medical programs, highly selected and high performing. Predictably, these students perform very well on the UMAT. At university, most of our students also perform very well; the mean GPA in our study was 6.1 out of a possible 7. Similar factors occur when studying other aspects of selection,1,7 and range restriction is an important limitation to this type of analysis. Statistical manipulation to control for this is possible, but we found that the correlation coefficient increased by no more than 0.03. This suggests that range restriction is not a major limitation here. Statistical power in our study was limited by the number of students available, especially in the later years of the study program. Larger studies, including those combining data from several medical schools would be beneficial.

Most students in our study have not yet completed their medical program. It is important to continue and extend our analysis throughout the medical program as it is possible that correlations between UMAT and performance during clinical training may emerge. Further, we have only reported on overall GPA, and have not explored correlation between UMAT scores and individual components of assessment such as knowledge, clinical skills or professionalism. This further research is important and will be done. Long-term follow-up into specialty training and clinical practice would shed more light on the value of UMAT in selection, as its value may become more evident later.

Selection of students into medical programs continues to be controversial, with demand outstripping supply, and with a widespread desire to select those students deemed more likely to become better doctors. UMAT does not seem to have useful validity in terms of predicting academic performance at university. Further research is needed to determine whether UMAT is predictive of performance during clinical training at medical school, in postgraduate training environments, and in clinical practice.

1 Correlation between overall UMAT mean score and undergraduate GPA


GPA = grade point average.
R2 = coefficient of determination for multivariate analysis.
UMAT = Undergraduate Medicine and Health Sciences Admission Test.

2 Correlation (unadjusted and partial) between undergraduate grade point average (GPA) and Undergraduate Medicine and Health Sciences Admission Test (UMAT) score, by UMAT section

GPA

UMAT Section 1

UMAT Section 2

UMAT Section 3


UMAT Section 1

PCC (P)

0.14 (0.01)

1.00

0.14 (0.01)

0.21 (< 0.001)

pPCC (P)

0.11 (0.04)

0.16 (0.003)

0.23 (< 0.001)

df (pdf)

337 (335)

0

339 (336)

339 (336)

UMAT Section 2

PCC (P)

0.06 (0.29)

0.14 (0.01)

1.00

0.11 (0.04)

pPCC (P)

0.05 (0.38)

0.16 (0.003)

0.14 (0.009)

df (pdf)

337 (335)

339 (336)

0

339 (336)

UMAT Section 3

PCC (P)

0.09 (0.11)

0.21 (< 0.001)

0.11 (0.04)

1.00

pPCC (P)

0.07 (0.23)

0.23 (< 0.001)

0.14 (0.009)

df (pdf)

337 (335)

339 (336)

339 (336)

0


pdf = degree of freedom for partial correlations.
PCC = Pearson correlation coefficient.
pPCC = partial Pearson correlation coefficient (adjusted for the other UMAT sections).

3 Correlation (unadjusted and partial) between undergraduate grade point average (GPA) and Undergraduate Medicine and Health Sciences Admission Test (UMAT) Section 1–3 scores, by program year

GPA

UMAT Section 1

UMAT Section 2

UMAT Section 3


Year 1

UMAT Section 1

PCC (P)

0.24 (0.002)

1.00

0.17 (0.04)

0.38 (< 0.001)

pPCC (P)

0.20 (0.01)

0.21 (0.008)

0.40 (< 0.001)

df (pdf)

161 (159)

0

161 (160)

161 (160)

UMAT Section 2

PCC (P)

0.10 (0.20)

0.17 (0.04)

1.00

0.07 (0.36)

pPCC (P)

0.07 (0.39)

0.21 (0.008)

0.15 (0.06)

df (pdf)

161 (159)

161 (160)

0

161 (160)

UMAT Section 3

PCC (P)

0.11 (0.16)

0.38 (< 0.001)

0.07 (0.36)

1.00

pPCC (P)

0.03 (0.68)

0.40 (< 0.001)

0.15 (0.06)

df (pdf)

161 (159)

161 (160)

161 (160)

0

Year 2

UMAT Section 1

PCC (P)

0.08 (0.48)

1.00

0.12 (0.33)

0.23 (0.05)

pPCC (P)

0.13 (0.30)

0.17 (0.16)

0.26 (0.03)

df (pdf)

71 (69)

0

71 (70)

71 (70)

UMAT Section 2

PCC (P)

0.11 (0.376)

0.12 (0.33)

1.00

0.19 (0.12)

pPCC (P)

0.05 (0.68)

0.17 (0.16)

0.22 (0.06)

df (pdf)

71 (69)

71

0

71 (70)

UMAT Section 3

PCC (P)

0.20 (0.10)

0.23 (0.05)

0.19 (0.12)

1.00

pPCC (P)

0.21 (0.08)

0.26 (0.03)

0.22 (0.06)

df (pdf)

71 (69)

71 (70)

71 (70)

0

Year 3

UMAT Section 1

PCC (P)

0.08 (0.52)

1.00

0.17 (0.16)

0.03 (0.80)

pPCC (P)

0.06 (0.65)

0.17 (0.17)

0.004 (0.98)

df (pdf)

69 (67)

0

69 (68)

69 (68)

UMAT Section 2

PCC (P)

0.14 (0.26)

0.17 (0.16)

1.00

0.16 (0.20)

pPCC (P)

0.13 (0.28)

0.17 (0.17)

df (pdf)

69 (67)

69 (68)

0

69 (68)

UMAT Section 3

PCC (P)

0.03 (0.83)

0.03 (0.80)

0.16 (0.20)

1.00

pPCC (P)

0.05 (0.69)

0.004 (0.98)

df (pdf)

69 (67)

69 (68)

69

0

Year 4

UMAT Section 1

PCC (P)

0.25 (0.16)

1.00

0.02 (0.90)

0.30 (0.10)

pPCC (P)

0.27 (0.15)

0.003 (0.99)

0.30 (0.11)

df (pdf)

30 (28)

0

30 (29)

30 (28)

UMAT Section 2

PCC (P)

0.02 (0.91)

0.02 (0.90)

1.00

0.07 (0.71)

pPCC (P)

0.02 (0.91)

0.003 (0.99)

0.15 (0.21)

df (pdf)

30 (28)

30 (29)

0

30 (28)

UMAT Section 3

PCC (P)

0.01 (0.94)

0.30 (0.10)

0.07 (0.71)

1.00

pPCC (P)

0.09 (0.62)

0.30 (0.11)

0.15 (0.21)

df (pdf)

30 (28)

30 (28)

30 (28)

0


df = degree of freedom.
pdf = degree of freedom for partial correlations.
PCC = Pearson correlation coefficient.
pPCC = partial Pearson correlation coefficient (adjusted for the other UMAT sections).

4 Multivariate relationship between undergraduate grade point average and Undergraduate Medicine and Health Sciences Admission Test (UMAT) score, by UMAT section and sex

Hypothesis test


B (95% CI)

SE

χ2

df

P


UMAT Section 1

0.011 (0.002 to 0.020)

0.0045

5.498

1

0.019

UMAT Section 2

0.002 ( 0.008 to 0.013)

0.0054

0.213

1

0.65

UMAT Section 3

0.006 ( 0.002 to 0.013)

0.0039

1.972

1

0.16

Sex

0.014 ( 0.271 to 0.006)

0.0677

4.176

1

0.04

5 Multivariate relationship between undergraduate grade point average and Undergraduate Medicine and Health Sciences Admission Test (UMAT) Section 1–3 scores and sex, by program year

Hypothesis test


B (95% CI)

SE

χ2

df

P


Year 1

Sex

0.247 (0.055 to 0.438)

0.0978

6.353

1

0.012

UMAT Section 1

0.024 (0.008 to 0.039)

0.0077

9.383

1

0.002

UMAT Section 2

0.001 (0.014 to 0.017)

0.0081

0.034

1

0.853

UMAT Section 3

0.003 (0.008 to 0.013)

0.0054

0.244

1

0.622

Year 2

Sex

0.073 (0.344 to 0.198)

0.1383

0.281

1

0.596

UMAT Section 1

0.010 (0.026 to 0.007)

0.0083

1.365

1

0.243

UMAT Section 2

0.002 (0.024 to 0.019)

0.0110

0.045

1

0.833

UMAT Section 3

0.017 (0.002 to 0.036)

0.0097

3.214

1

0.073

Year 3

Sex

0.257 (0.054 to 0.568)

0.1588

2.620

1

0.105

UMAT Section 1

0.006 (0.014 to 0.027)

0.0107

0.368

1

0.544

UMAT Section 2

0.012 (0.011 to 0.034)

0.0116

1.039

1

0.308

UMAT Section 3

0.007 (0.011 to 0.025)

0.0091

0.541

1

0.462

Year 4

Sex

0.217 (0.561 to 0.128)

0.1758

1.520

1

0.218

UMAT Section 1

0.023 (0.001 to 0.047)

0.0123

3.460

1

0.063

UMAT Section 2

0.001 (0.025 to 0.028

0.0136

0.009

1

0.923

UMAT Section 3

0.009 (0.036 to 0.017)

0.0137

0.475

1

0.491


Provenance: Not commissioned; externally peer reviewed.

Received 16 September 2010, accepted 16 January 2011

  • David Wilkinson1
  • Jianzhen Zhang2
  • Malcolm Parker3

  • School Of Medicine, University of Queensland, Brisbane, QLD.

Correspondence: jenny.zhang@uq.edu.au

Competing interests:

We are staff members of the School of Medicine, University of Queensland.

  • 1. Ferguson E, James D, Madeley L. Factors associated with success in medical school: systematic review of the literature. BMJ 2002; 324: 952-957.
  • 2. Australian Council for Educational Research. UMAT. Undergraduate Medicine and Health Sciences Admission Test. http://umat.acer.edu.au/ (accessed May 2010).
  • 3. Carr SE. Emotional intelligence in medical students: does it correlate with selection measures? Med Educ 2009; 43: 1069-1077.
  • 4. Mercer A. Selecting medical students: an Australian case study [PhD thesis]. Perth: Murdoch University, 2007.
  • 5. Griffin B, Harding DW, Wilson IG, Yeomans ND. Does practice make perfect? The effect of coaching and retesting on selection tests used for admission to an Australian medical school. Med J Aust 2008; 189: 270-273. <MJA full text>
  • 6. Wiberg M, Sundström A. A comparison of two approaches to correction of restriction of range in correlation analysis. Pract Assess Research Eval [Internet] 2009; 14 (5). Epub 2009 Mar 18. http://pareonline.net/getvn.asp?v=14&n=5 (accessed Feb 2011).
  • 7. Wilkinson D, Zhang J, Byrne GJ, et al. Medical school selection criteria and the prediction of academic performance. Evidence leading to change in policy and practice at the University of Queensland. Med J Aust 2008; 188: 349-354. <MJA full text>

Author

remove_circle_outline Delete Author
add_circle_outline Add Author

Comment
Do you have any competing interests to declare? *

I/we agree to assign copyright to the Medical Journal of Australia and agree to the Conditions of publication *
I/we agree to the Terms of use of the Medical Journal of Australia *
Email me when people comment on this article

Responses are now closed for this article.