Medical school selection criteria and the prediction of academic performance

Selection of students into medical school is a high-profile, contentious, and high-stakes exercise. Selection typically serves two distinct purposes: to reduce the large number of otherwise qualified and capable applicants to match the number of places available, and to enrol students thought most likely to succeed in what is an arduous program of study and to subsequently become effective members of the profession.

In Australia, most medical schools use a combination of prior academic performance (prior degree grade point average [GPA] for graduate-entry programs), performance on a specific admissions test (GAMSAT [Graduate Australian Medical School Admissions Test]) and an interview or other psychometric technique.1 There is little consistency between schools in the combination of, or the weight given to, each component in the decision-making process.1 Similar variation in practice has been reported from the United Kingdom.2 A systematic review3 indicated that prior academic performance is the best predictor of subsequent academic performance and that interviews add little to the selection process. There is only very limited published literature on the value of GAMSAT.

Amid concerns that the selection process for the University of Queensland MB BS program may not be optimal, the School of Medicine reviewed selection as part of the regular process of quality assurance. This article reports on the relationship between the school’s selection criteria (combination of GPA, GAMSAT and interview) and subsequent student performance during the medical program.

The MB BS program at the School of Medicine, University of Queensland, is Australia’s largest, admitting 375 students in 2007. This is a 4-year graduate-entry program, and entry is based on selection according to an initial hurdle of attaining a GPA of 5 or more in any prior degree (any Masters or PhD degree is deemed as meeting this requirement). Final ranking and an offer of a place is based on a combination of GAMSAT and interview scores (Box 1).

GAMSAT is a written examination developed by the Australian Council for Educational Research with the consortium of graduate-entry medical schools,1 and was designed to assess the capacity of students to undertake high-level intellectual studies in a demanding course. It evaluates the nature and extent of abilities and skills gained through prior experience and learning, including the mastery and use of concepts in basic science as well as the acquisition of more general skills in problem solving, critical thinking, and writing.1 GAMSAT Section 1 focuses on reasoning in humanities and social sciences, Section 2 focuses on written communication, and Section 3 on reasoning in biological and physical sciences.

The interview is semistructured and is conducted by a pair of academic and/or medically qualified members of staff. The interview is designed to assess selected personal attributes considered appropriate to a career in medicine, including the ability to communicate, cognitive style, cooperativeness, evidence of active participation, motivation to practise medicine, open-mindedness, self-confidence, stable self-identity, and ability to contribute to the health services in Australia. All interviewers undergo formal training, which includes a didactic presentation, detailed written material and video presentations of actual interviews from previous years. Prospective interviewers are required to achieve a predefined standard on trial interviews before being accepted onto the panel. Interviewers are also required to undergo biennial recertification based on their ratings of videotaped interviews. Before each daily interview session, interviewers are briefed once more on the essential components of the interview. Interviewers are required to independently rate the eight distinct applicant attributes before conferring with the co-interviewer to arrive at final individual ratings for each attribute. Interviewers rate each attribute according to a calibrated yardstick for that attribute. The 16 individual ratings generated by this process are averaged to produce a final mark. If sufficient agreement between interviewers is not achieved, a different pair of interviewers reinterview the applicant.

Students in the MB BS program take examinations midway through and at the end of Year 1 and Year 2, and (at the time of this study) at the end of Year 4.

For the purpose of our analyses, we used scores obtained in all examinations in all 4 years of the program combined (we called this the overall total score), Year 1 examinations, and Year 4 examinations. We did this to explore the relationship between selection criteria and total academic performance, performance shortly after entry to the program, and performance at the end of the program, respectively. We also used scores in the three major components of the Year 1 and Year 4 assessments, namely the clinical examination, the ethics examination, and the written examination. Year 2 results were similar to Year 1 results (data not shown).

We studied three consecutive, complete student cohorts (2001–2003 entry years) comprising 706 students, over the 4-year MB BS program.

Our findings confirm, and importantly extend, the existing literature on factors that are associated with academic performance in medical school, and hence that may have value in the selection of students.3 Our results show that the selection criteria we (and many other schools) use predict about 20% (ranging from about 10% to 30%) of student academic performance (as measured by examination), depending on year within the program and examination component.

The largest and most recent systematic review on this topic concluded that prior academic performance accounted for 23% of variance in undergraduate medical performance,3 a figure consistent with our findings. It is important to stress therefore that most variation in academic performance is not explained by selection criteria and is presumably a consequence of both intrinsic personal factors and the effect of the teaching itself.

The systematic review3 also concluded that further studies on the value of the interview are needed, and indicated that, in the studies reviewed, interviews seemingly added little or nothing to the selection process. At best, they were associated with only weak to modest independent prediction (0.11–0.14) of performance. In our study, the interview was correlated with overall total examination performance and performance in each Year 4 component (Box 6), but only at modest levels. The high levels of statistical significance (low P values) reflect the large dataset we studied, and it is important to focus on the absolute value of the adjusted correlation coefficient when interpreting our findings. For the interview, these ranged from 0.05 to 0.22, and were consistently substantially lower than the adjusted coefficients for GPA, except for Year 4 clinical and ethics examination performance, in which they were similar.

Although widely used in Australia, and now used by some schools in the UK and Ireland, there is only limited literature on the value of GAMSAT in predicting medical school performance. A PubMed search identified only two studies (one that is 10 years old and outlines a rationale for GAMSAT, and a second that explores association with clinical reasoning skills in a small sample of students). There are many more articles that validate the North American equivalent, the Medical College Admission Test.4 Our data indicate that GAMSAT is poor in predicting academic performance; all of the adjusted correlation coefficients (Box 6) for GAMSAT total score are close to zero. These are the first published data on the validity of GAMSAT in an entire student cohort. Our findings suggest that GAMSAT may have only limited value in predicting academic performance.

An exploratory meta-analysis showed that the predictive power of interviews for academic success was only 0.06, and for clinical success (after graduation) was 0.17, indicating a modest effect.5 Part of the reason for this may be that interviews are inherently unreliable. The authors of a literature review and empirical study called into doubt the fairness of interviews as a highly influential component of an admissions process.6 We acknowledge that the interview process we used may, in and of itself, have influenced the results of our analysis. For example, the training and standardisation that we sought may have limited the ability to discriminate between candidates. However, the data in Box 3 indicate that a wide range of scores was attained in our interviews, and our results are consistent with those previously reported.3

Another important limitation is that our analysis only included students with a relatively high GAMSAT score (mean, 66.2) and so our findings do not test the whole range of GAMSAT scores. Nevertheless, this is a “real world” use of GAMSAT and our findings should be interpreted in that light. A strength of our study is that we examined a range of selection criteria in association with each other, not in isolation (which is a problem in many previously published studies examining individual components of the selection process3).

One stated desire of a selection process is to seek to include non-academic, non-cognitive factors. It is important to acknowledge that academic ability and other key (non-cognitive) attributes are not necessarily inversely correlated,7 or mutually exclusive. Indeed, there is evidence that the two are positively correlated.8 Selecting on academic performance alone, or predominantly, may in fact also lead to the admission of students with attractive non-cognitive attributes.

We acknowledge that other approaches to selection do exist or are being developed, such as the Personal Qualities Assessment9 and the Multiple Mini-Interview.10 These methods may have value, but need to be formally assessed in longitudinal studies. We also acknowledge that selection criteria may influence the learning behaviour of potential applicants (such as studying particular material in preparation for GAMSAT, which may then influence future performance at medical school); this may be seen as a useful or useless influence. Further, the “threat” of an interview may dissuade some potential applicants (such as those with inherently poor communication skills) from even applying to medical school.

Our findings are not merely of academic interest. Our judgement from these data is that GPA is the strongest and most defensible (although not perfect) performance predictor, that interviews add only a small amount, and that GAMSAT has little predictive value. As a result, the school has removed the interview from its selection process. We will diligently monitor the effect of changes to our selection criteria on performance in our medical program, and will report our experience in the future.

4 Regression modelling of the relationship between selection criteria and academic performance

6 Scatter plots for overall total score against grade point average (GPA), Graduate Australian Medical School Admissions Test (GAMSAT) total score, and interview score