Workplace-based assessment for international medical graduates: at what cost?

Australia, like many developed countries, has a shortage of doctors. While this is disputed by some,1 the literature highlights current and projected shortages of skilled clinicians relative to the demand.2-15 One response to this imbalance is to utilise international medical graduates (IMGs).

The Medical Board of Australia is responsible for the registration of all doctors in Australia and sets the relevant standards, codes and guidelines. The Australian Medical Council (AMC) oversees assessment before this. Doctors who have trained overseas can take one of two paths to registration. Those who attained medical qualifications in selected countries (United Kingdom, United States, Canada, New Zealand and Ireland) can fast-track registration under a competent authority pathway. IMGs who obtained their medical accreditation in other countries follow a standard pathway.16 In the standard pathway, proficiency is assessed by a 3.5-hour, computer-based, multiple choice examination and an integrated multidisciplinary structured clinical assessment. Clinical skills in medicine, surgery, obstetrics, gynaecology, paediatrics and psychiatry are assessed at 16 stations using role-players and one or two real patients. Candidates have 2 minutes of reading time and 8 minutes to complete the task at each station. A pass requires satisfactory performance in 12 or more stations, including one obstetrics and gynaecology station and one paediatrics station.

The waiting period to sit for the structured clinical assessment is long, and often fewer than half of the candidates are successful.9,17 Combined with a low pass rate, this restricts employment and subsequent opportunities for many IMGs. From an IMG’s perspective, the traditional examinations provide a very limited opportunity to understand and be integrated into the Australian health care system.

Evidence suggests that this traditional pathway for registering and integrating IMGs into the Australian health care system has not been ideal. A recent study found that IMGs are more likely than Australian-trained doctors to have complaints made against them to medical boards.18 The medical boards in Australia have also been more likely to make adverse findings against IMGs than against Australian-trained doctors.18 The authors of this study suggested a rethink on the regulation of IMGs in Australia. A logical starting place for reviewing this would be at the start of the IMGs’ journey.

The Workplace Based Assessment (WBA) Program was developed in 2010 by a team of academics from Hunter New England Health (HNEH) and the University of Newcastle (UN) to overcome some of the difficulties associated with assessing IMGs and integrating them into the health care system. It was accredited by the AMC in 2010 as an alternative to the AMC clinical examination and to give an opportunity to study its feasibility. The WBA Program is based on a 6-month assessment process. Entry into the WBA Program has the same eligibility requirements as the standard pathway, including verification of qualifications and passes in the AMC English language test and the multiple choice exam. However, the WBA candidates have limited medical registration to work in accredited Australian hospitals.9 Details of the program have been reported previously,17 and are summarised in Box 1. A recent parliamentary enquiry recommended a national rollout.9

Health system resources provided by HNEH and UN to deliver the program include:

• 91 assessors from medicine, surgery, paediatrics, obstetrics and gynaecology, mental health, emergency and general practice;

• academics to conduct calibration and recalibration of assessors;

• administrative staff to support and coordinate the program;

• staff members for the program committees (governance committee; appeals committee; 360° review panel); and

• the Director of the program.

Our aim in this study was to report an estimate of the resources required to deliver the WBA Program from the perspective of the health care system.

We assessed 2012 costs by stipulating:

• the resources considered appropriate for inclusion in the costing;

• the measurement of these resources;

• how monetary units were applied.19

The costing in this study reflects the direct and indirect resources required to deliver the program to a cohort of 15 IMGs — the average number of candidates per cohort in the WBA Program over 2010–2012.

The total labour-related cost of delivering the WBA Program for a cohort of 15 IMGs was $240 236. After adding consumables ($3148), the per candidate cost was $16 226. Labour costs comprised 98.7% of total costs.

The analysis allowed for the examination of cost by program stage, cross-tabulated by activity (Box 4). The most resource-intensive stage was program delivery ($190 566). Of this amount, $176 027 was for assessment. Major cost components for assessment were mini clinical evaluation exercises ($122 691) and case-based discussions ($43 396).

Revenue collected by the WBA Program was $6000 per candidate, totalling $90 000. After receipt of fees, the program had a deficit of $153 384 ($10 226 per candidate).

Based on our cost analysis, the WBA Program for IMGs had a deficit of $153 384 ($10 226 per candidate), which represents the contribution made by the health care organisation.

This investment has produced several positive outcomes. Out of 95 IMG candidates who have undertaken the program, all have passed. Therefore, compared with the structured clinical assessment and its approximate 50% pass rate, the program maximises the possible supply of well prepared and assessed IMGs into the Australian medical workforce.

The program provides an extended window for observing and assessing IMGs and allows a rigorous assessment process by a range of senior clinicians who take ownership of the program and provide assessment. From the assessors’ perspective, it gives them an opportunity to observe clinical performance and workplace interaction. From the candidates’ perspective, there is a 6-month period to learn to work within the Australian health care system.

Although the program does not provide training, regular feedback allows candidates to self-assess, self-correct and reflect as needed. Successful integration of IMGs into the health care system is not solely based on clinical expertise; it requires the individual to appropriately interact and communicate with patients and staff. This is a core component of the program and is assessed through the multisource feedback from co-workers at all levels of the health system who would otherwise not routinely be invited to comment.

The program helps to offset the cost of recruiting doctors for health services, as the candidates are retained in the settings in which they have worked while gaining registration. Over the long term, IMGs who successfully complete the WBA Program may provide benefits to the Australian health care system during their working lifetime. In this context, the cost of the program to the health system is relatively small.

Projections indicate that the supply of doctors in Australia is likely to fall short of demand until at least 2025, suggesting IMGs will remain an important source of skilled clinicians in this country.24 The WBA Program can be delivered in regional Australia (where there are substantial doctor shortages). It makes good sense to ensure that IMGs seeking registration in Australia are provided with an efficient and fair opportunity to register.

As a preliminary study to describe the costs associated with delivering the WBA Program from the perspective of the health system, our analysis had some limitations. The study captured some costs that did not create a monetary burden for the health system, and it might appear that this would create an upward bias in our estimates. For example, there was evidence that assessors and administrative staff contributed unpaid time to the program. However, we had aimed to capture opportunity cost, and this rightly includes the unpaid overtime spent on the program because it represents time that could have been spent on other health service functions.

We have identified the resources used to deliver the program, and yet it is up to health system administrators to determine whether this investment represents value for money. Our study was not designed to examine the effectiveness of the program nor to compare its cost against benefit. Further work is being planned to compare the cost and effectiveness of the program against the standard pathway.

Our program constitutes an innovative response to improving the assessment pathway for IMGs and we have now shown that the cost per candidate for the program is small. IMGs who have their performance assessed in the workplace and receive longer-term, constructive feedback will benefit the health care system. We believe this program is cost-effective and the cost is a small price to pay for doctors who will provide many years of safe and effective service.

2 Schematic of the Workplace Based Assessment (WBA) Program for international medical graduates

AHPRA = Australian Health Practitioner Regulation Agency. AMC = Australian Medical Council. HNEH = Hunter New England Health. * National program for doctors and medical educators, aimed at improving the quality of the teaching and supervision of trainee doctors and students. ◆