Connect
MJA
MJA

A review of prevocational medical trainee assessment in New South Wales

Craig M Bingham and Roslyn Crampton
Med J Aust 2011; 195 (7): 410-412. || doi: 10.5694/mja11.10109
Published online: 3 October 2011

In New South Wales and other Australian states, medical graduates are given a conditional training (internship) registration in their first postgraduate year of training, and they must demonstrate satisfactory performance to qualify for general medical registration. In the past, medical registration was overseen by state medical boards, but in July 2010, a national registration scheme overseen by the new Medical Board of Australia was instituted.

Assessment processes vary in each state, but are based on assessments made by senior clinicians acting as term supervisors. So far, the Medical Board of Australia has endorsed the intern-assessment processes of each state, but a nationally consistent approach to internship standards is in development.1

In NSW, the training of interns (postgraduate year 1) and residents (postgraduate year 2) — collectively known as prevocational medical trainees — is overseen by the Clinical Education and Training Institute (CETI). CETI introduced new progress review forms for assessing prevocational trainees in January 2009. The new forms were intended to:

  • promote assessment of trainees against the competency framework of the Australian Curriculum Framework for Junior Doctors;2

  • encourage trainees to develop skills in self-assessment, an essential professional skill; and

  • encourage dialogue between supervisors and trainees on the specifics of the assessment so as to provide trainees with more feedback on their performance.

Here, we report a review of the new assessment forms.

Methods

There are two assessment forms: “Mid Term Formative Appraisal” and “End of Term Summative Assessment”. They can be viewed online at http://www.ceti.nsw.gov.au by searching for “progress review form”.

The forms ask trainees and supervisors to rate performance on eight clinical management, six communication and four professionalism competencies, plus one overall rating of performance; 19 ratings in all.

The rating scale is:

  • Clearly below expected level (the trainee is substantially below the standard expected for someone at this stage of training);

  • Borderline/requires assistance (the trainee requires further development relative to the average trainee performing this term at this stage of training);

  • At expected level (the trainee is performing at the level expected of someone at this stage of training);

  • Clearly above expected level (the trainee is performing at a level that should be recognised and highly commended).

The forms also provide prompts for written comments on the strengths and weaknesses of the trainee.

We reviewed the first 3390 progress review forms completed by prevocational trainees and their supervisors that were returned to CETI since January 2009.

Results

Our sample of 3390 assessment forms included 1674 mid-term forms and 1716 end-term forms from 1072 trainees (about 83% of all prevocational trainees in NSW) from 43 of the 52 prevocational training sites.

Ratings of performance

On nearly 43% of end-term forms, trainees self-assessed their performance as “at expected level” on all 19 items. No trainees rated themselves as performing below expected level on any item at end of term. The item most likely to be rated at borderline by trainees was procedural skills (on 1.4% of end-term forms).

There was a strong correlation between trainee self-assessment and supervisor assessment. Supervisor ratings for individual rating items matched the trainee’s self-assessment 66.9% of the time, were higher by one point 26.3% of the time and lower by one point 3.1% of the time.

The sample included forms for 1046 trainees for whom there were one or more end-term assessments. One thousand and thirty trainees (98.5%) were rated by themselves and by their supervisors as performing at or above the expected level of performance on every item in every term. The Box shows the distribution of overall performance ratings by trainee self-assessment and supervisor assessment at mid term and end of term. The “below expected level” rating is virtually unused, and the “borderline” rating is rarely used. Further, at the end of term, supervisors are more likely to rate trainees as “above expected level” than “at expected level”.

On the summary overall performance rating at the end of term, only two trainees (in one term each; 0.2%) were assessed as performing below expected level by their supervisors. Eleven trainees (0.9%) were assessed as borderline.

Written comments on forms

Supervisors were mostly likely to comment on the strengths of the trainee, usually in generous but non-specific language (most commonly, the single word “excellent”, or phrases such as “reliable and conscientious”, “sound clinical knowledge”, “enthusiastic and keen to learn” and “no issues”). Few comments contained specific feedback to direct the trainee’s professional development.

Discussion

We consider that there are four main reasons for assessing medical trainees:

  • to protect patient safety by identifying underperformance issues;

  • to guide the trainee’s professional development by providing feedback to the trainee on how to improve performance (formative assessment);

  • to document satisfactory performance so that the trainee can progress to higher training (summative assessment); and

  • to identify strengths and weaknesses of the education and training process (systemic assessment).

How well does the NSW assessment process meet these purposes?

Patient safety

In our study, assessment identified only two trainees at end of term (0.2%) as performing below the expected level and 11 (0.9%) as performing at a borderline level. Reports from directors of prevocational training and estimates by the NSW Junior Medical Officer Forum (a peer representative body) suggest that real rates of underperformance are much higher.

Published data on rates of underperformance among junior doctors are few, but it has been suggested that:

a small proportion (3%–7%) of JMOs [prevocational trainees] in training programs are ‘difficult’ or ‘problem’ doctors who require the intervention of someone in authority [but] a higher proportion of JMOs are simply unable to cope with the stress of the job and exhibit signs of being ‘in difficulty’, ‘stressed’ or ‘distressed’.3

In a recent study validating the mini peer assessment tool in the United Kingdom Foundation Programme (a 2-year training program that forms the bridge between medical school and specialist/general practice training), 5.6% of Foundation-year 2 trainees were assessed as being below expectations for Foundation-year 2 completion.4

Failure to identify underperformance during formal assessments is a missed opportunity to remediate gaps in learning, and potentially (if the underperformance is not detected and managed through one of the training hospital’s other processes) a risk to patient safety.

Formative assessment: the role of self-assessment

To provide safe and effective patient care, doctors must recognise their strengths and limitations. Most vocational training programs use self-assessment as a key part of their assessment process5-7 because skill in self-assessment is fundamental to self-directed continuing medical education, which in turn is essential for maintaining competence throughout a clinical career.

It has been noted that self-assessment has limitations.8 In our study, 43% of trainees rated their performance as “at expected level” on all 19 items. We have discussed this result with trainees. Many feel constrained by the potential implications of their self-assessment and deliberately opt for a safe but meaningless “straight down the line” set of ratings.

Formative assessment: providing feedback to trainees

In our sample, most supervisor assessments offered encouragement to trainees in the form of high ratings and short general comments to the effect that they are doing a good job and will improve with experience. However, trainees request and value specific feedback about particular strengths and weaknesses, coupled with suggestions about actions they should take to improve. Such feedback was rarely documented in these assessments.

Summative assessment: measuring readiness to progress

One effect of concentrating performance ratings on only two points of a four-point scale is that the assessments do not discriminate between different levels of trainee performance.

Our assessment data show that nearly 99% of trainees perform at or above the expected level. What does this actually say about their developing competency? If a trainee does a core medical, surgical or emergency term in Term 1, performing “at expected level” indicates a lower level of performance than if the term was completed in Term 5. The phrases “at expected level” or “above expected level” do not indicate a specific level of competence.

Systemic assessment: guiding improvements in training

Good assessment processes can reveal areas for improvement in the training program. We found that the assessment results were strangely uniform across training sites, training terms, and learning domains. An assessment process that was more sensitive to differences in training outcomes would be more informative.

Improving assessment

There is an extensive body of literature on assessment methods that we cannot review here. The UK has implemented a much more complex assessment scheme: “most aspects of the curriculum are assessed by more than one method and by more than one assessor, and all assessments are carried out several times”.9 This “triangulation” of assessment has the potential to be more reliable and informative, but implementing such a system in NSW would require additional resources.

The apparent weakness of the current process may be a measure of the time and engagement that trainees and supervisors bring to written assessment. There is always a risk that filling in forms will be seen as a mere formality. Future research on assessment methods that are sustainable in the clinical environment and supported by supervisors and trainees would be valuable. Changes in the workplace to facilitate adequate time for feedback and assessment may be required, and supervisors may need training in assessment methods.

Overall mid-term and end-term performance ratings by trainee self-assessment and supervisor assessment

Received 2 February 2011, accepted 27 July 2011

  • Craig M Bingham1
  • Roslyn Crampton2

  • Clinical Education and Training Institute, Sydney, NSW.

Correspondence: cbingham@ceti.nsw.gov.au

Competing interests:

No relevant disclosures.

Author

remove_circle_outline Delete Author
add_circle_outline Add Author

Comment
Do you have any competing interests to declare? *

I/we agree to assign copyright to the Medical Journal of Australia and agree to the Conditions of publication *
I/we agree to the Terms of use of the Medical Journal of Australia *
Email me when people comment on this article

Responses are now closed for this article.