Objective: To evaluate the effectiveness of formal assessments of prevocational medical trainees (ie, interns, first-year residents and international medical graduates undergoing supervised training).
Design and setting: Retrospective review of structured assessment forms that are completed by all prevocational trainees and their supervisors in New South Wales public hospitals. We reviewed the first 3390 assessment forms, representing assessments of 1072 trainees at 43 training sites (83% of all prevocational trainees in NSW) since January 2009.
Results: At the end of term, 43% of trainees assessed themselves as performing “at expected level” on all 19 rating items. Nearly 99% of trainees were assessed by their supervisors as performing at or above the expected level on all assessment items. Written comments by supervisors were generally short and encouraging, but lacked specific feedback that trainees might use to guide improvements in performance.
Conclusions: As currently used by trainees and supervisors, the assessment forms may underreport trainee underperformance, do not discriminate strongly between different levels of performance of trainees or the training system, and do not provide trainees with enough specific feedback to guide their professional development.
In New South Wales and other Australian states, medical graduates are given a conditional training (internship) registration in their first postgraduate year of training, and they must demonstrate satisfactory performance to qualify for general medical registration. In the past, medical registration was overseen by state medical boards, but in July 2010, a national registration scheme overseen by the new Medical Board of Australia was instituted.
Assessment processes vary in each state, but are based on assessments made by senior clinicians acting as term supervisors. So far, the Medical Board of Australia has endorsed the intern-assessment processes of each state, but a nationally consistent approach to internship standards is in development.1
In NSW, the training of interns (postgraduate year 1) and residents (postgraduate year 2) — collectively known as prevocational medical trainees — is overseen by the Clinical Education and Training Institute (CETI). CETI introduced new progress review forms for assessing prevocational trainees in January 2009. The new forms were intended to:
promote assessment of trainees against the competency framework of the Australian Curriculum Framework for Junior Doctors;2
encourage trainees to develop skills in self-assessment, an essential professional skill; and
encourage dialogue between supervisors and trainees on the specifics of the assessment so as to provide trainees with more feedback on their performance.
There are two assessment forms: “Mid Term Formative Appraisal” and “End of Term Summative Assessment”. They can be viewed online at http://www.ceti.nsw.gov.au by searching for “progress review form”.
The forms ask trainees and supervisors to rate performance on eight clinical management, six communication and four professionalism competencies, plus one overall rating of performance; 19 ratings in all.
Clearly below expected level (the trainee is substantially below the standard expected for someone at this stage of training);
Borderline/requires assistance (the trainee requires further development relative to the average trainee performing this term at this stage of training);
At expected level (the trainee is performing at the level expected of someone at this stage of training);
Clearly above expected level (the trainee is performing at a level that should be recognised and highly commended).
Our sample of 3390 assessment forms included 1674 mid-term forms and 1716 end-term forms from 1072 trainees (about 83% of all prevocational trainees in NSW) from 43 of the 52 prevocational training sites.
On nearly 43% of end-term forms, trainees self-assessed their performance as “at expected level” on all 19 items. No trainees rated themselves as performing below expected level on any item at end of term. The item most likely to be rated at borderline by trainees was procedural skills (on 1.4% of end-term forms).
There was a strong correlation between trainee self-assessment and supervisor assessment. Supervisor ratings for individual rating items matched the trainee’s self-assessment 66.9% of the time, were higher by one point 26.3% of the time and lower by one point 3.1% of the time.
The sample included forms for 1046 trainees for whom there were one or more end-term assessments. One thousand and thirty trainees (98.5%) were rated by themselves and by their supervisors as performing at or above the expected level of performance on every item in every term. The Box shows the distribution of overall performance ratings by trainee self-assessment and supervisor assessment at mid term and end of term. The “below expected level” rating is virtually unused, and the “borderline” rating is rarely used. Further, at the end of term, supervisors are more likely to rate trainees as “above expected level” than “at expected level”.
On the summary overall performance rating at the end of term, only two trainees (in one term each; 0.2%) were assessed as performing below expected level by their supervisors. Eleven trainees (0.9%) were assessed as borderline.
Supervisors were mostly likely to comment on the strengths of the trainee, usually in generous but non-specific language (most commonly, the single word “excellent”, or phrases such as “reliable and conscientious”, “sound clinical knowledge”, “enthusiastic and keen to learn” and “no issues”). Few comments contained specific feedback to direct the trainee’s professional development.
to protect patient safety by identifying underperformance issues;
to guide the trainee’s professional development by providing feedback to the trainee on how to improve performance (formative assessment);
to document satisfactory performance so that the trainee can progress to higher training (summative assessment); and
to identify strengths and weaknesses of the education and training process (systemic assessment).
In our study, assessment identified only two trainees at end of term (0.2%) as performing below the expected level and 11 (0.9%) as performing at a borderline level. Reports from directors of prevocational training and estimates by the NSW Junior Medical Officer Forum (a peer representative body) suggest that real rates of underperformance are much higher.
a small proportion (3%–7%) of JMOs [prevocational trainees] in training programs are ‘difficult’ or ‘problem’ doctors who require the intervention of someone in authority [but] a higher proportion of JMOs are simply unable to cope with the stress of the job and exhibit signs of being ‘in difficulty’, ‘stressed’ or ‘distressed’.3
In a recent study validating the mini peer assessment tool in the United Kingdom Foundation Programme (a 2-year training program that forms the bridge between medical school and specialist/general practice training), 5.6% of Foundation-year 2 trainees were assessed as being below expectations for Foundation-year 2 completion.4
Failure to identify underperformance during formal assessments is a missed opportunity to remediate gaps in learning, and potentially (if the underperformance is not detected and managed through one of the training hospital’s other processes) a risk to patient safety.
To provide safe and effective patient care, doctors must recognise their strengths and limitations. Most vocational training programs use self-assessment as a key part of their assessment process5-7 because skill in self-assessment is fundamental to self-directed continuing medical education, which in turn is essential for maintaining competence throughout a clinical career.
It has been noted that self-assessment has limitations.8 In our study, 43% of trainees rated their performance as “at expected level” on all 19 items. We have discussed this result with trainees. Many feel constrained by the potential implications of their self-assessment and deliberately opt for a safe but meaningless “straight down the line” set of ratings.
In our sample, most supervisor assessments offered encouragement to trainees in the form of high ratings and short general comments to the effect that they are doing a good job and will improve with experience. However, trainees request and value specific feedback about particular strengths and weaknesses, coupled with suggestions about actions they should take to improve. Such feedback was rarely documented in these assessments.
Our assessment data show that nearly 99% of trainees perform at or above the expected level. What does this actually say about their developing competency? If a trainee does a core medical, surgical or emergency term in Term 1, performing “at expected level” indicates a lower level of performance than if the term was completed in Term 5. The phrases “at expected level” or “above expected level” do not indicate a specific level of competence.
Good assessment processes can reveal areas for improvement in the training program. We found that the assessment results were strangely uniform across training sites, training terms, and learning domains. An assessment process that was more sensitive to differences in training outcomes would be more informative.
There is an extensive body of literature on assessment methods that we cannot review here. The UK has implemented a much more complex assessment scheme: “most aspects of the curriculum are assessed by more than one method and by more than one assessor, and all assessments are carried out several times”.9 This “triangulation” of assessment has the potential to be more reliable and informative, but implementing such a system in NSW would require additional resources.
The apparent weakness of the current process may be a measure of the time and engagement that trainees and supervisors bring to written assessment. There is always a risk that filling in forms will be seen as a mere formality. Future research on assessment methods that are sustainable in the clinical environment and supported by supervisors and trainees would be valuable. Changes in the workplace to facilitate adequate time for feedback and assessment may be required, and supervisors may need training in assessment methods.
Received 2 February 2011, accepted 27 July 2011
- 1. Medical Board of Australia. Proposed registration standard for granting general registration as a medical practitioner to Australian and New Zealand medical graduates on completion of intern training. Melbourne: Australian Health Practitioner Regulation Agency, 2011. http://www.medicalboard.gov.au/News/Current-Consultations.aspx (accessed Jul 2011).
- 2. Confederation of Postgraduate Medical Education Councils. Australian curriculum framework for junior doctors project. http://www.cpmec.org.au/Page/acfjd-project (accessed Jan 2011).
- 3. Lake F, Ryan G. Teaching on the run tips 11: the junior doctor in difficulty. Med J Aust 2005; 183: 475-476. <MJA full text>
- 4. Archer J, Norcini J, Southgate L, et al. mini-PAT (Peer Assessment Tool): a valid component of a national assessment programme in the UK? Adv Health Sci Educ Theory Pract 2008; 13: 181-192.
- 5. Royal Australasian College of Physicians. Multi-source feedback. A formative assessment tool. 2009. http://www.racp.edu.au/index.cfm?objectid=FB515CEF-F3CD-A9ED-2A852247C5CB068F (accessed Sep 2011).
- 6. Royal Australasian College of Physicians. Learning needs analysis. http://www.racp.edu.au/page/learning-needs-analysis (accessed Sep 2011).
- 7. Royal Australasian College of Surgeons. Approval of CPD activities and accreditation of educational courses. 2009. http://www.surgeons.org/media/14249/POL_2009-06-26_Approval_of_CPD_Activities_and_Accreditation_of_Educational_Courses_V1.pdf (accessed Sep 2011).
- 8. Epstein RM. Assessment in medical education. N Engl J Med 2007; 356: 387-396.
- 9. Holsgrove G, Davies H. Assessment in the Foundation Programme. In: Jackson N, Jamieson A, Khan A, editors. Assessment in medical education and training. Oxford: Radcliffe Publishing, 2007: 41-51.