Objective: To investigate whether interviewer personality, sex or being of the same sex as the interviewee, and training account for variance between interviewers’ ratings in a medical student selection interview.
Design, setting and participants: In 2006 and 2007, data were collected from cohorts of each year’s interviewers (by survey) and interviewees (by interview) participating in a multiple mini-interview (MMI) process to select students for an undergraduate medical degree in Australia. MMI scores were analysed and, to account for the nested nature of the data, multilevel modelling was used.
Main outcome measures: Interviewer ratings; variance in interviewee scores.
Results: In 2006, 153 interviewers (94% response rate) and 268 interviewees (78%) participated in the study. In 2007, 139 interviewers (86%) and 238 interviewees (74%) participated. Interviewers with high levels of agreeableness gave higher interview ratings (correlation coefficient [r] = 0.26 in 2006; r = 0.24 in 2007) and, in 2007, those with high levels of neuroticism gave lower ratings (r = − 0.25). In 2006 but not 2007, female interviewers gave higher overall ratings to male and female interviewees (t = 2.99, P = 0.003 in 2006; t = 2.16, P = 0.03 in 2007) but interviewer and interviewee being of the same sex did not affect ratings in either year. The amount of variance in interviewee scores attributable to differences between interviewers ranged from 3.1% to 24.8%, with the mean variance reducing after skills-based training (20.2% to 7.0%; t = 4.42, P = 0.004).
Conclusion: This study indicates that rating leniency is associated with personality and sex of interviewers, but the effect is small. Random allocation of interviewers, similar proportions of male and female interviewers across applicant interview groups, use of the MMI format, and skills-based interviewer training are all likely to reduce the effect of variance between interviewers.
- 1. Harasym PH, Woloschuk W, Mandin H, Brundin-Mather R. Factors affecting the selection of students for medical school. Acad Med 1996; 71: S40-S42.
- 2. Roberts C, Walton M, Rothnie I, et al. Factors affecting the utility of the multiple mini interview in selecting candidates for graduate-entry medical school. Med Educ 2008; 42: 396-404.
- 3. Eva KW, Reiter HI, Rosenfeld J, Norman GR. The relationship between interviewers’ characteristics and ratings assigned during a multiple mini-interview. Acad Med 2004; 79: 602-609.
- 4. Digman J. Personality structure: emergence of the five-factor model. Annu Rev Psychol 1990; 41: 417-440.
- 5. McCrae RR, Costa PT. A five-factor theory of personality. In: John LaPaop, editor. Handbook of personality: theory and research. 2nd ed. New York: Guilford Press, 1999: 139-153.
- 6. Pulakos ED, Schmitt N, Whitney D, Smith M. Individual differences in interviewer ratings: the impact of standardization, consensus discussion, and sampling error on the validity of a structured interview. Personnel Psychol 1996; 49: 85-102.
- 7. Donnellan MB, Oswald FL, Baird BM, Lucas RE. The mini-IPIP scales — tiny yet effective measures of the big five factors of personality. Psychol Assess 2006; 18: 192-203.
- 8. International Personality Item Pool: a scientific collaboratory for the development of advanced measures of personality and other individual differences. http://ipip.ori.org/ipip (accessed Aug 2010).
- 9. Sacco JM, Scheu CR, Ryan AM, Schmitt N. An investigation of race and sex similarity effects in interviews: a multilevel approach to relational demography. J Appl Psychol 2003; 88: 852-865.
- 10. Wilson I, Harding D, Yeoman N, et al. Lack of biases in the MMI. Med Teach 2009; 31: 959-960.
- 11. Cohen J, Cohen P, West SG, Aiken LS. Applied multiple regression/correlation analysis for the behavioral sciences. 3rd ed. Fort Worth: Harcourt Brace College Publishers, 2003.
- 12. McManus IC, Thompson M, Mollon J. Assessment of examiner leniency and stringency (‘hawk–dove effect’) in the MRCP(UK) clinical examination (PACES) using multi-facet Rasch modeling. BMC Med Educ 2006; 6: 42.
- 13. Lawson DM. Applying generalizability theory to high-stakes objective structured clinical examinations in a naturalistic environment. J Manipulative and Physiol Ther 2006; 29: 463-467.
- 14. Harasym PH, Woloschuk W, Cunning L. Undesired variance due to examiner stringency/leniency effect in communication skill scores assessed in OSCEs. Adv Health Sci Educ 2008; 13: 617-632.
- 15. Graziano WG, Eisenberg NH. Agreeableness: a dimension of personality. In: R Hogan, Johnston J, Briggs S, editors. Handbook of personality psychology. San Diego, Calif: Academic Press, 1997: 795-824.
- 16. Gustavo C, Okun MA, Knight GP, de Guzman MRT. The interplay of traits and motives on volunteering: agreeableness, extraversion and prosocial value motivation. Pers Individ Dif 2005; 38: 1293-1305.
- 17. Laurence CO, Turnbull CO, Briggs NE, Robinson JS. Applicant characteristics and their influence on success: results from an analysis of applicants to the University of Adelaide Medical School, 2004–2007. Med J Aust 2010; 192: 212-216. <MJA full text>
- 18. Mann WC. Interviewer scoring differences in student selection interviews. Am J Occup Ther 1979; 33: 235-339.
- 19. Iramaneerat C, Yudkowsky R. Rater errors in a clinical skills assessment of medical students. Eval Health Prof 2007; 30: 266-283.
- 20. Roberts C, Rothnie I, Zoanetti N, Crossley J. Should candidate scores be adjusted for interviewer stringency or leniency in the multiple mini-interview? Med Educ 2010; 44: 690-698.
Publication of your online response is subject to the Medical Journal of Australia's editorial discretion. You will be notified by email within five working days should your response be accepted.