Programmatic assessment: the process, rationale and evidence for modern evaluation approaches in medical education

Learners in a programmatic assessment environment make better use of feedback

Common approaches to the assessment of competence in undergraduate and post-graduate medicine are modular — each module is completed with an assessment at the end, often resulting in a grade. Learners are passed for that module when they exceed a minimum passing level. Completing all modules results in graduation under the assumption that combining, often disparate, modules in this way assures entrustment to professional practice or ongoing training.

This traditional summative approach to assessment has been perceived to have many disadvantages: encouraging poor learning styles, particularly shallow or rote learning;1 unwanted effects such as “grade hunting”; and learners consistently barely exceeding the pass mark. This approach also seems to promote extrinsic motivation, with learning being dictated by the curriculum and the requirements to pass. Feedback is largely ignored by learners, particularly by those who need it the most.2 The system is also uninformative: two learners may receive a pass but display varying strengths and weaknesses. Grades alone are uninformative about the individual learner and are the poorest form of feedback; they may demonstrate learning problems, but provide no detail of their nature.3 Qualitative, narrative information is more powerful than quantitative, particularly when complex skills are assessed, such as communication or professionalism.4 Finally, the traditional approach no longer aligns with modern learning theories that emphasise agency in learning and ongoing feedback for durable learning. The modular approach stems from a “mastery” model of learning, where performance demonstrated once is assumed to last forever. Forgetting curves from psychology clearly show this is unfounded.5 Traditional systems cannot link phases of learning with patient outcomes, resulting in disconnection between undergraduate training, post-graduate training and clinical outcomes.

Contemporary learning programs, instead, foster deeper learning styles. Holistic, authentic tasks encourage learners to acquire and construct their knowledge.6 Learners and learning processes are more central, with learners expected to self-direct their learning, unlike traditional assessment approaches, which usually hamper self-direction.2 In modern education, competency frameworks play an important role internationally, asserting that learners achieving competencies are better prepared for modern health care. While competency frameworks recognise the importance of knowledge and skills, they typically also include professionalism, communication, teamwork and interprofessional collaboration. These complex and practically important competencies cannot be learned in a single course with an end-of-course assessment. They develop continuously as a function of practice, rich feedback, role modelling and guidance. Traditional summative assessment approaches do not evaluate development over time or extend to assessment of integrated cognitive, behavioural and attitudinal aspects of practice, and are therefore deemed inadequate for the assessment of such complex skills.7

There is a felt need for an alternate assessment approach,8 which is outlined further in this article.

Programmatic assessment

In programmatic assessment, each assessment activity is considered to be one data point (Box). Each data point is optimised for learning, not purely for decision making, which means that every assessment activity must provide information on learner performance and meaningful feedback, with an embedded expectation that learners will use feedback for learning. Any assessment method may be used — traditional or modern, objective or subjective — and feedback may be quantitative, qualitative or both. The choice is determined by the educational justification at a certain point in the curriculum and its contribution to the assessment program as a whole. For example, an oral exam could evaluate and stimulate verbalisation skills. A written essay might require synthesis of the literature and may assess critical thinking and writing skills. Aspects of both assessments can be combined to evaluate the learner’s ability to communicate scholarly insights. In this way, assessment can evaluate across methods, much like laboratory tests and pathology reports may be combined to diagnose a patient. Each individual assessment is only informative and no high stakes decisions are taken, so the reliability of a single assessment activity is not especially relevant to the choice of methods. Assessments in programmatic assessment will be partly modular, but will also include longitudinal assessment, such as progress testing. Here, all learners are assessed with tests attuned to graduate level multiple times throughout the year.9 Behavioural longitudinal assessment could include periodic multisource feedback, in which teachers, peers and the learners themselves evaluate performance with narrative feedback.10 The arrangement and choice of methods is planned purposefully in a master plan so there is coherence in the educational messages that assessments convey to learners.

In programmatic assessment, the learner collates evidence of progress in a portfolio with reference to a verified database of assessment activity information. The portfolio becomes the hub of analysis of feedback, collation of learning plans to address problems, and evidence of progress, and especially comes into its own when building a case for outcomes such as professionalism and communication. This portfolio also offers tantalising promise as a future bridge between the stages of learning, subsequent professional practice and even patient outcomes. For example, a medical student’s portfolio may form the basis of internship training, continue through specialist training, and be maintained through professional life as part of their registration. This has the potential to provide an incredible database for professional practice and for research into the connections between training, the learner’s response to learning opportunities and clinical outcomes.

Traditionally, exams are summative or formative with pass–fail decisions. In programmatic assessment, this binary process is replaced by a continuum ranging from low stakes to high stakes. Any individual data point is low stakes, but the stakes and the number of data points are proportionally related. Higher stakes decisions require more data points and very high stakes decisions need many data points. For example, a mid-year progress decision uses an aggregation of all available data points, usually across a competency framework. The learners receive progress decisions with areas of strengths and weaknesses identified and may be required to remediate certain areas. Later, a final end-of-year high stakes decision is taken on progression to the next year. For this decision, many data points are available and the emerging “picture” of the learner has clarity. Metaphorically, data points are pixels in a picture; with increasing pixels, the image becomes clear. The review of data points and decision making is done by a committee of experts. For the majority of learners, a progress decision will be a simple task due to the clarity of information; thus, the committee will only need to deliberate extensively on a few learners. Consensus is justified as the decision is based on evidence provided across multiple data points. This efficient due process of applying considerable expertise to assess the “close calls” makes it a trustworthy process. Committee members are trained and have standards guiding their judgement, with final judgements based on consensual professional processes.

Programmatic assessment promotes learning within a framework in which the use of feedback is an explicit expectation. However, the follow-up on feedback and self-directed learning needs additional support.11 Learners in programmatic assessment are supported by learning coaches to effectively use the feedback they receive to plan for ongoing learning. Learning coaches guide learners over an extended period of time, sometimes throughout the curriculum, and have access to assessment data and regularly meet with their learners. Learners prepare for meetings by self-assessment based on available feedback; learning coaches may encourage learners to identify strategies for remediation and to develop learning processes. They are not soft companions, but like sports coaches, they will try to push the learner to optimise their learning. To protect the relationship between learning coach and learner, learning coaches may contribute to but are essentially disconnected from the final progression decision making.

Current status and future of programmatic assessment

Programmatic assessment in medical education began with a small number of flagship programs that gathered early evidence and experience of this innovative assessment approach. These programs have accrued supportive educational evidence over many years.12,13 Their successes and the logic of programmatic assessment in implementing educational theory have encouraged other programs to adopt aspects of this approach in their curriculum. Universities that have recently implemented full programmatic assessment within health professional courses include the University of Groningen and Utrecht University, Netherlands, for medicine and veterinary medicine; McMaster University and the University of Toronto, Canada, for undergraduate and post-graduate medicine; Université de Montréal, Canada, for veterinary medicine; and Flinders University, Australia, for graduate-entry medicine. These and other more recent adopters are building on early research and adding to the knowledge about implementing programmatic assessment in different contexts.14,15

Research on programmatic assessment is important to understand the mechanisms and circumstances of how it may work best. For example, research shows that while low stakes assessment is often perceived differently by learners,16 learners are very satisfied with the approach.17 Learners in a programmatic assessment environment make better use of their feedback than learners in a traditional assessment culture, and their knowledge score is substantially higher at the end of training.13 Experience in the cost of programmatic assessment is also emerging and is relevant to most training programs. There is a cost attached to all assessments, and programmatic assessment is affordable when resource allocation is reprioritised to favour assessment with the maximum educational impact.18

Although conceptually appealing, implementing programmatic assessment is not an easy task, especially when traditional assessment models are firmly established. Implementation requires a partial overhaul of the assessment program, but, more importantly, a mind shift away from the summative culture towards an informative learning culture. Achieving this mind shift requires change management, staff and learner development, and ample discussion. Once implemented, the greatest challenge is to provide regular, high quality usable feedback for learners, as this is essential for optimal development of expertise.11

Like many new innovations, programmatic assessment will be implemented in a range of contexts and capacities. Some implementation will be more successful than others. It is important for the future of programmatic assessment that research and evaluation continue to inform the ongoing development of this important educational concept.

Conclusion

Programmatic assessment is a modern vision of assessment and learning. We know assessment drives learning, but learning drives programmatic assessment. It aligns with modern education and it optimises both the learning and decision-making functions of assessment. Increasingly, universities and post-graduate institutions internationally are using programmatic assessment and are advancing experience and research.