Connect
MJA
MJA

Selecting medical students for academic and attitudinal outcomes in a Catholic medical school

Julie A Quinlivan, Lawrence T Lam, Siu hong Wan and Rodney W Petersen
Med J Aust 2010; 193 (6): 347-350.
Published online: 20 September 2010

Abstract

Objectives: To evaluate whether the four criteria used by the University of Notre Dame Australia (UNDA) to select medical students are successful in selecting for graduates with the desired outcomes of academic excellence and Catholic “mission fit”.

Design, setting and participants: Prospective cohort study of medical students selected for 2008 and 2009 entry to UNDA in Sydney, New South Wales.

Main outcome measures: The statistical association between the two academic selection criteria of the Graduate Australian Medical School Admissions Test (GAMSAT) and grade point average (GPA) compared with the outcome of medical school examination performance, and the two mission selection criteria of a portfolio score and interview score compared with the outcome of a positive attitude towards serving underserved communities as measured using the Medical Student Attitudes Toward the Underserved (MSATU) test.

Results: A total of 223 students were enrolled. GAMSAT section 3, GPA and the interview scores were significantly positively associated with academic performance (P < 0.05). However, none of the selection variables were significantly associated with a positive attitude towards serving underserved communities, as measured by the MSATU score.

Conclusion: None of the four selection tools used were significantly associated with medical students who had a positive attitude towards serving underserved communities.

Selection of medical students has been a topical subject for many years.1 In Australia, criterion-based approaches based on academic grades have traditionally been used in medical student selection. However, initiatives by the University of Newcastle, New South Wales, in the 1970s, followed by other Australian universities, set the scene for diversification in the criteria used to select medical students. Selection criteria went beyond simple grade levels, with the adoption of interviews designed to assess personal qualities of applicants.2,3 The broadening of the criteria for selection was intended to address perceived deficiencies in communication skills and emotional intelligence, which had been reported as criticisms of some medical practitioners in consumer surveys.4

In a Catholic medical school such as the University of Notre Dame Australia (UNDA), the outcomes of social justice rank equally with academic outcomes and formed a key part in planning a selection policy for medical students. In Australia, the health disparities in some communities mean there is a need for a committed workforce of doctors who will work all their lives in those communities.5 Delivering doctors to the health care workforce with a positive attitude towards underserved communities was identified as a strategy by which the Catholic mission of the medical school could be achieved.

To balance the competing needs of having graduates who achieve academic excellence and having graduates with positive attitudes towards serving underserved communities, UNDA implemented a selection process that involved two academic and two attitudinal criteria to generate a selection score. The aim of this study is to report on whether these academic and attitudinal criteria successfully improved the University’s selection of appropriate students.

Methods

Participants were medical students enrolled in the MB BS program at UNDA, Sydney medical school, in 2008 and 2009. Participants gave informed consent.

The student selection criteria included four equally weighted components: a grade point average (GPA), the Graduate Australian Medical School Admissions Test (GAMSAT), an interview score and a portfolio score.

GPA: A weighted GPA was calculated from the applicant’s primary Bachelor degree.

GAMSAT score: The GAMSAT (http://www.gamsat.edu.au) was designed by the Australian Council for Educational Research with a consortium of graduate-entry medical schools, and has been used as a quantified measure of academic ability to undertake demanding academic courses. The GAMSAT test is divided into three sections: section 1 (humanities and social sciences), section 2 (written communication) and section 3 (reasoning in biological and physical sciences).

Interview score: The interview was a semi-structured vignette-based interview lasting 20 minutes and designed to assess non-cognitive personal attributes. It was based on existing interview styles designed to attract more generalist career students.6-8 The interview was conducted by a pair of interview-trained medical academic and/or community volunteer staff interviewers, who retrain every 2 years and are briefed on the day of the interviews by the Head of Selection. Interviewers individually scored students based on their responses to the vignettes, using a predefined and calibrated set of possible outcomes. They also scored candidates for general communication and group-working characteristics. At the conclusion of the interview, members of the interview panel finalised their marks and conferred with their co-interviewer. During this time, the candidates waited in a secure waiting area. If consensus between interviewers was not achieved, the candidate returned to the interview room and a further 10-minute interview was conducted, using additional vignettes.

Portfolio score: The portfolio score was derived from the UNDA supplementary information form.9 The form collected the following information: rural background; higher research degrees and research experience; work experience; sporting, music, church and other participation or leadership activities; experience working with charitable agencies or with underserved communities; languages spoken other than English; curriculum vitae; two written references; and a personal statement about why the applicant wished to study medicine at UNDA.

Staff members were trained in reading and evaluating the portfolios in a didactic session using cases as examples. Their first 20 portfolios were scored under supervision and the scores were not used in final assessments. Portfolios were scored using a criterion-referenced scale, and the scores were provided to the Head of Selection. The Head of Selection received at least two portfolio scores from separate reviewers, and in some cases a third evaluation was conducted for audit purposes.

Outcomes

The key outcomes were academic performance in medical school assessment measures and a positive attitude towards working with underserved communities.

The selected academic measures were written examination performances, multiple clinical assessment task (MCAT) performances and total examination score.

The outcome of selecting graduates with positive attitudes towards serving underserved communities was the central Catholic mission outcome. It was quantified by use of the Medical Student Attitudes Toward the Underserved (MSATU) test.10 The MSATU test is the only reliable instrument that measures these attitudes;10 its reliability, validity, factor analysis and internal consistency coefficients are reported elsewhere.10,11 The MSATU test contains two subscales that use a five-point psychometric Likert scale indicating the extent to which respondents agree or disagree with statements. The first subscale measures “attitudes regarding professional roles and responsibilities”, and has the two subscale factors of societal expectations and professional responsibility. The second subscale is “attitudes towards the basic to extensive care services to which patients should have access”, and has the two subscale factors of basic services and expensive procedures.11 The total MSATU score is the combination of the attitudes and services subscales.

Statistical analysis

Data were analysed using Stata, version 10.2 (StataCorp, College Station, Tex, USA) statistical software. Descriptive statistics are summarised with frequencies, percentages, means, standard deviations and medians. Bivariate analyses were conducted between students’ demographic characteristics, selection variables and all key outcome variables. As all outcome variables were continuous by nature, comparisons of means across groups were applied using t tests or one-way analysis of variance for categorical demographic variables. Correlation analyses were employed for continuous demographic variables as well as all selection variables, as they were also continuous by nature.

To select potential predictors for further analyses, a selection criterion of P < 0.2 was applied to the test statistics in the bivariate analyses. Multiple linear regression modelling was applied to the data to identify potential predictors for each of the key outcome variables. Backward stepwise procedures were used for model reduction to achieve the most parsimonious model with all significant variables retained. Model diagnostics were applied to the final model of each study outcome for identifying problems with collinearity and violation of basic assumptions. A significance level of 0.05 was used for retaining variables in the regression analyses.

Ethics approval

Institutional ethics committee approval was obtained from UNDA.

Results

A total of 223 students were enrolled in the Sydney medical school of UNDA: 111 (50%) in 2008 and 112 (50%) in 2009. The students’ demographic characteristics, selection scores and the key study outcomes are summarised in Box 1 and Box 2.

Results obtained from the bivariate analyses suggested some potential predictor variables for outcomes.

Academic outcomes

In terms of student characteristics, sex and undergraduate degree were significantly associated with the MCAT (sex, t218 = 3.87, P < 0.001; undergraduate degree, F(2,217) = 7.06, P = 0.001) (Box 3). Female students scored significantly higher than males, with a mean MCAT score of 128.5 (SD, 9.3) and 123.4 (SD, 10.5), respectively. Students who came from a biomedical or sciences background or from a humanities or social sciences background also scored significantly higher than students with other undergraduate degrees, with mean MCAT scores of 126.4 (SD, 9.8), 126.9 (SD, 9.5) and 114.9 (SD, 13.5), respectively.

In terms of the two academic selection variables, the GPA was positively and significantly associated with all three academic outcomes. The GAMSAT total score did not significantly correlate to the academic outcomes. However, the section 3 score of the GAMSAT was significantly and positively associated with written examination marks. Interview scores correlated significantly and positively with MCAT scores (r = 0.22, P = 0.001).

Catholic mission outcome

The only bivariate predictor of a high MSATU score was undergraduate degree. Students with a background in the humanities and social sciences had the highest scores of 3.80 (SD, 0.28). This compared favourably with scores of 3.66 (SD, 0.31) for students with a biomedical or sciences background, and 3.49 (SD, 0.24) for students with another undergraduate background.

The two Catholic mission selection criteria designed to select for students with high MSATU scores (interview score and portfolio score) did not correlate significantly with MSATU score.

Multivariate analysis

Results obtained from the multiple linear regression analyses are summarised in Box 4. After adjusting for other potential confounding variables, GPA, GAMSAT section 3 score, and interview score were found predictive of different outcomes. The GAMSAT section 3 scores were predictive of the written examination marks (P = 0.020) and the final examination marks (P = 0.015). The interview scores were found to be predictive of the MCAT marks (P = 0.002) as well as the final examination marks (P = 0.025). The GPA was also predictive of the final examination marks (P = 0.001).

In terms of modelling for variance, section 3 score in the GAMSAT, adjusted for year level, was associated with 34.2% of variance in written examination performance.

Student sex, undergraduate degree and interview performance, adjusted for year level, were associated with 16.6% of variance in MCAT performance.

However, the combination of GPA, GAMSAT section 3 score and interview score, adjusted for year level, was only associated with 8.0% of variance in final examination score.

The only useful modelling to predict MSATU score, the Catholic mission outcome, was undergraduate degree adjusted for year level. This was associated with only 6.6% of variance.

Discussion

Our findings demonstrate that selecting medical students who will perform well in written examinations or in MCAT can be promoted through a combination of academic and student selection predictors. We found that 34.2% of variance in written examination marks was statistically associated with the section 3 score in GAMSAT. We also found that sex, undergraduate degree and interview score predicted 16.6% of variance in MCAT performance. However, written assessment and the MCAT measure different skills in medical practice. Written assessment relies heavily on memory and knowledge of facts and their interpretation, and the MCAT relies on verbal intelligence, examination and practical and communication skills. Given the divergent skills under evaluation, it is not surprising that a single combination of selection variables was not associated with a large percentage of the variance in total examination performance.

We were disappointed that we were unable to identify a significant association between the current University portfolio and interview system, and students who had positive attitudes towards serving underserved communities, as measured by the MSATU score. This outcome was selected by the study designers to fulfil the Catholic mission values of the University, as it encompassed the primary service mission of Catholic Health Australia (CHA). The foundation principle of service, as stated by CHA (http://www.cha.org.au/site.php?id=26), is:

The provision of health care is conducted out of a spirit of service and solidarity with those in need. Health care is a social good. The degree to which health care is driven by a genuine compassionate concern for others and a selfless commitment to the well being of people, will be the measure by which a community can gauge its maturity and sense of integrity.

The only useful modelling to predict MSATU score was undergraduate degree in combination with year level. This only accounted for 6.6% of variance, but suggested that a humanities or social sciences background was associated with higher scores than other degrees.

Other universities trying to achieve non-academic and socially oriented outcomes in their medical graduates report similarly disappointing outcomes. A comparison study from Drew University in the United States that aimed to determine measurable factors for selecting students who would then work to service underserved communities found that these outcomes were driven by individual values and subjective external influences, and not by standard predictors.5 A second US study that compared a customary committee review selection process with an actuarial system found the two groups were ultimately not reliably differentiated on any outcome.12 In this case, the committee selection system used aptitude tests as well as basic demographic data, information on extracurricular activities, vocational interests, work experience, letters of recommendation, personality test profiles and interview impressions. This was very similar to the interview and supplementary information portfolio used in our study.

However, other studies have reported that these strategies are successful in selecting graduates who have a greater focus on serving the community. New York Medical College has had considerable success in meeting its targets since implementing a detailed portfolio and pre-admission questionnaire for students.6

Given the results of our trial, UNDA will need to review its selection system again, to attempt to identify students who meet both the academic and mission goals of the University. This process will start with community consultation and evaluation of the service goals to which the University would like its graduates to aspire. This process has now begun.

1 Descriptive statistics of student characteristics

Student characteristic

Frequency (n = 223) (%)


Year of intake

2008 

111 (50%)

2009

112 (50%)

Sex

Male

111 (50%)

Female

112 (50%)

Age group*

< 25 years

149 (68%)

25–29 years

54 (25%)

30+ years

17 (7%)

State of residence

New South Wales

202 (91%)

Victoria

9 (4%)

Queensland

9 (4%)

Other

3 (1%)

Undergraduate degree

Biomedical/health sciences

176 (79%)

Humanities/social sciences

36 (16%)

Other

11 (5%)

Years since graduation*

< 3

140 (64%)

3–5

48 (22%)

6+

32 (14%)

Postgraduate training

Yes

25 (11%)

No

198 (89%)


* Totals are not 223 due to three blank responses.

2 Descriptive statistics of selection variables and outcomes

Selection variable

Mean (SD)

Median (range)


GPA

5.96 (0.58)

6.01 (3.95–7.00)

GAMSAT

Section 1

Section 2

Section 3

Total*

 

58.2 (5.0)

63.7 (6.6)

60.4 (7.9)

60.7 (4.7)

 

58 (46–76)

64 (49–80)

59.5 (42–100)

60 (52–79)

Portfolio score

16.0 (3.0)

16 (9–25)

Interview score

33.6 (4.0)

34 (16–40)

Study outcomes

Written examination raw marks

220.93 (26.50)

223.25 (98.88–279.50)

MCAT

125.89 (10.24)

127.50 (78.00–141.00)

Final examination marks

70.18 (6.55)

70.61 (32.44–85.34)

MSATU score

3.67 (0.31)

3.68 (2.51–4.49)


GPA = grade point average. GAMSAT = Graduate Australian Medical School Admissions Test. MCAT = multiple clinical assessment task. MSATU = Medical Student Attitudes Toward the Underserved questionnaire. * Total GAMSAT score is the average of the scores on the three component sections.

3 Results of tests of association between variables of interest and study outcomes

Variables of interest

Written marks

MCAT

Final marks

MSATU


Student characteristics

Result

P

Result

P

Result

P

Result

P


Sex

t218, 1.02

0.31

t218, 3.87*

< 0.001*

t218, 1.83

0.07

t177, 0.63

0.53

Age

r, – 0.09

> 0.05

r, – 0.09

> 0.05

r, – 0.09

> 0.05

r, – 0.08

> 0.05

Undergraduate degree

F(2,217), 0.45

0.64

F(2,217), 7.06*

0.001*

F(2,217), 1.92

0.15

F(2,176), 4.05*

0.02*

Postgraduate

t218, 1.78

0.08

t218, 1.08

0.28

t218, 1.22

0.22

t177, 0.50

0.62

Selection variables

r

P

r

P

r

P

r

P


GPA

0.15*

0.03*

0.18*

0.01*

0.19*

0.005*

– 0.07

0.35

GAMSAT

Section 1

0.11

0.10

– 0.01

0.95

0.12

0.01

0.01

0.96

Section 2

– 0.09

0.18

– 0.03

0.71

– 0.08

0.21

0.08

0.32

Section 3

0.13*

0.05*

– 0.09

0.18

0.11

0.12

0.06

0.43

Total

0.11

0.11

– 0.09

0.19

0.10

0.16

0.07

0.37

Portfolio score

– 0.14

0.03

– 0.02

0.81

– 0.07

0.32

– 0.05

0.51

Interview score

– 0.01

0.87

0.22*

0.001*

0.11

0.09

0.06

0.46


MCAT = multiple clinical assessment task. MSATU = Medical Student Attitudes Toward the Underserved questionnaire. GPA = grade point average. GAMSAT = Graduate Australian Medical School Admissions Test. * Significant correlation between variables.

4 Results obtained from final multiple regression models of selected variables on study outcomes

Variables retained in the final model

β

SE

t

P

Variance explained*


Written examination marks

Year

30.20

2.93

10.32

< 0.001

34.2%

GAMSAT section 3 score

0.44

0.19

2.35

0.020

MCAT

Year

2.77

1.28

2.16

0.032

16.6%

Sex

– 4.62

1.29

– 3.59

< 0.001

Humanities/social sciences undergraduate degree

– 0.49

1.77

– 0.28

0.781

Other undergraduate degree

– 10.10

2.97

– 3.40

0.001

Interview score

0.50

0.16

3.10

0.002

Final examination marks

Year

0.96

0.88

1.09

0.277

8.0%

GPA

2.45

0.76

3.22

0.001

GAMSAT section 3 score

0.14

0.06

2.45

0.015

Interview score

0.26

0.11

2.25

0.025

MSATU

Year

– 0.09

0.05

– 2.02

0.045

6.6%

Humanities/social sciences undergraduate degree

0.14

0.06

2.25

0.026

Other undergraduate degree

– 0.15

0.11

– 1.46

0.147


GAMSAT = Graduate Australian Medical School Admissions Test. MCAT = multiple clinical assessment task. GPA = grade point average. MSATU = Medical Student Attitudes Toward the Underserved questionnaire. * Percentage of outcome variance explained by the input variables.

  • Julie A Quinlivan1
  • Lawrence T Lam1
  • Siu hong Wan1
  • Rodney W Petersen2

  • 1 School of Medicine, University of Notre Dame Australia, Fremantle, WA.
  • 2 Joondalup Health Campus, Edith Cowan University, Perth, WA.

Correspondence: juliequinlivan@nd.edu.au

Competing interests:

None identified.

  • 1. Campbell EF, Rank BK, Sinclair AJM. Selection of medical students — a burning question. Med J Aust 1974; 1: 785-788.
  • 2. Vinson T, Cooney G, Turnbull J. Admission to medical school: the Newcastle experiment. Innov Educ Teach Int 1979; 16: 277-287.
  • 3. Ewan C, Melville P. Interviews of medical school entrants: to what purpose? Med J Aust 1982; 2: 233-236.
  • 4. Greengross S. What patients want from their doctors. In: Allen I, Brown P, Hughes P, editors. Choosing tomorrow’s doctors. London: Policy Studies Institute, 1997.
  • 5. Bazargan M, Lindstrom RW, Dakak A, et al. Impact of desire to work in underserved communities on selection of specialty among fourth year medical students. J Natl Med Assoc 2006; 98: 1460-1465.
  • 6. Juster F, Levine JK. Recruiting and selecting generalist-orientated students at the New York Medical College. Acad Med 1999; 74: S45-S48.
  • 7. Donnon T, Paulucci EO. A generalizability study of the medical judgement vignettes interview to assess students’ noncognitive attributes for medical school. BMC Med Educ 2008; 8: 58.
  • 8. Donnon T, Oddone-Paolucci E, Violato C. A predictive validity study of medical judgment vignettes to assess students’ noncognitive attributes: a 3-year prospective longitudinal study. Med Teach 2009; 31: e148-e155.
  • 9. University of Notre Dame Australia. Supplementary information form. Sydney: UNDA, 2010.
  • 10. Crandall SJ, Volk RJ, Loemker V. Medical students’ attitudes toward providing care for the underserved. Are we training socially responsible physicians? JAMA 1993; 269: 2519-2523.
  • 11. Crandall SJ, Volk RJ, Cacy DS. A longitudinal investigation of medical student attitudes toward the medically indigent. Teach Learn Med 1997; 9: 254-260.
  • 12. Schofield W, Garrard J. Longitudinal study of medical students selected for admission to medical school by actuarial and committee methods. Br J Med Educ 1975; 9: 86-90.

Author

remove_circle_outline Delete Author
add_circle_outline Add Author

Comment
Do you have any competing interests to declare? *

I/we agree to assign copyright to the Medical Journal of Australia and agree to the Conditions of publication *
I/we agree to the Terms of use of the Medical Journal of Australia *
Email me when people comment on this article

Responses are now closed for this article.