A review of prevocational medical trainee assessment in New South Wales

On nearly 43% of end-term forms, trainees self-assessed their performance as “at expected level” on all 19 items. No trainees rated themselves as performing below expected level on any item at end of term. The item most likely to be rated at borderline by trainees was procedural skills (on 1.4% of end-term forms).

There was a strong correlation between trainee self-assessment and supervisor assessment. Supervisor ratings for individual rating items matched the trainee’s self-assessment 66.9% of the time, were higher by one point 26.3% of the time and lower by one point 3.1% of the time.

The sample included forms for 1046 trainees for whom there were one or more end-term assessments. One thousand and thirty trainees (98.5%) were rated by themselves and by their supervisors as performing at or above the expected level of performance on every item in every term. The Box shows the distribution of overall performance ratings by trainee self-assessment and supervisor assessment at mid term and end of term. The “below expected level” rating is virtually unused, and the “borderline” rating is rarely used. Further, at the end of term, supervisors are more likely to rate trainees as “above expected level” than “at expected level”.

On the summary overall performance rating at the end of term, only two trainees (in one term each; 0.2%) were assessed as performing below expected level by their supervisors. Eleven trainees (0.9%) were assessed as borderline.

Supervisors were mostly likely to comment on the strengths of the trainee, usually in generous but non-specific language (most commonly, the single word “excellent”, or phrases such as “reliable and conscientious”, “sound clinical knowledge”, “enthusiastic and keen to learn” and “no issues”). Few comments contained specific feedback to direct the trainee’s professional development.

In our study, assessment identified only two trainees at end of term (0.2%) as performing below the expected level and 11 (0.9%) as performing at a borderline level. Reports from directors of prevocational training and estimates by the NSW Junior Medical Officer Forum (a peer representative body) suggest that real rates of underperformance are much higher.

Published data on rates of underperformance among junior doctors are few, but it has been suggested that:

In a recent study validating the mini peer assessment tool in the United Kingdom Foundation Programme (a 2-year training program that forms the bridge between medical school and specialist/general practice training), 5.6% of Foundation-year 2 trainees were assessed as being below expectations for Foundation-year 2 completion.4

Failure to identify underperformance during formal assessments is a missed opportunity to remediate gaps in learning, and potentially (if the underperformance is not detected and managed through one of the training hospital’s other processes) a risk to patient safety.

To provide safe and effective patient care, doctors must recognise their strengths and limitations. Most vocational training programs use self-assessment as a key part of their assessment process5-7 because skill in self-assessment is fundamental to self-directed continuing medical education, which in turn is essential for maintaining competence throughout a clinical career.

It has been noted that self-assessment has limitations.8 In our study, 43% of trainees rated their performance as “at expected level” on all 19 items. We have discussed this result with trainees. Many feel constrained by the potential implications of their self-assessment and deliberately opt for a safe but meaningless “straight down the line” set of ratings.

In our sample, most supervisor assessments offered encouragement to trainees in the form of high ratings and short general comments to the effect that they are doing a good job and will improve with experience. However, trainees request and value specific feedback about particular strengths and weaknesses, coupled with suggestions about actions they should take to improve. Such feedback was rarely documented in these assessments.

One effect of concentrating performance ratings on only two points of a four-point scale is that the assessments do not discriminate between different levels of trainee performance.

Our assessment data show that nearly 99% of trainees perform at or above the expected level. What does this actually say about their developing competency? If a trainee does a core medical, surgical or emergency term in Term 1, performing “at expected level” indicates a lower level of performance than if the term was completed in Term 5. The phrases “at expected level” or “above expected level” do not indicate a specific level of competence.

Good assessment processes can reveal areas for improvement in the training program. We found that the assessment results were strangely uniform across training sites, training terms, and learning domains. An assessment process that was more sensitive to differences in training outcomes would be more informative.

There is an extensive body of literature on assessment methods that we cannot review here. The UK has implemented a much more complex assessment scheme: “most aspects of the curriculum are assessed by more than one method and by more than one assessor, and all assessments are carried out several times”.9 This “triangulation” of assessment has the potential to be more reliable and informative, but implementing such a system in NSW would require additional resources.

The apparent weakness of the current process may be a measure of the time and engagement that trainees and supervisors bring to written assessment. There is always a risk that filling in forms will be seen as a mere formality. Future research on assessment methods that are sustainable in the clinical environment and supported by supervisors and trainees would be valuable. Changes in the workplace to facilitate adequate time for feedback and assessment may be required, and supervisors may need training in assessment methods.