Connect
MJA
MJA

Professional monitoring and critical incident reporting using personal digital assistants

Paul D Bent, Stephen N Bolsin, Bernie J Creati, Andrew J Patrick and Mark E Colson
Med J Aust 2002; 177 (9): 496-499.
Published online: 4 November 2002

Abstract

Objective: To assess the practicality of using personal digital assistants (PDAs) for the collection of logbook data, procedural performance data and critical incident reports in anaesthetic trainees.

Design: Pilot study.

Setting: Two tertiary referral centres (in Victoria and New Zealand) and a large district hospital in Queensland.

Participants: Six accredited Australian and New Zealand College of Anaesthetists (ANZCA) registrars and their ANZCA training supervisors.

Interventions: Registrars and supervisors underwent initial training for one hour, and supervisors were provided with ongoing support.

Main outcome measures: Reliable use of the program, average time for data entry and number of procedures logged.

Results: ANZCA trainees reliably enter data into PDAs. The data can be transferred to a central database, where they can be remotely analysed before results are fed back to trainees.

Conclusions: This technology can be used to monitor professional performance in ANZCA trainees.

The recent failure of professional monitoring systems in the United Kingdom, particularly in Bristol, has highlighted the need for improved monitoring of training and performance.1,2 The culture of medicine needs to change to embrace the feedback of data on personal and unit performance.3 External assessment of competency is a difficult task and agreement about what clinical outcomes are acceptable is open to debate.4 Self-assessment can include logbooks, maintenance of professional standards programs and self-audit.5

The recent introduction of handheld computers, or personal digital assistants (PDAs), has given medical practitioners the ability to accurately collect data at the point of care. The potential uses for medical practitioners have recently been highlighted.6,7 We investigated the feasibility of using a PDA software package custom-designed for anaesthesiology trainees. It incorporates a bedside logbook, a self-assessment tool using the cumulative summation (cusum) technique for particular anaesthetic procedures, and the ability to collect data on anaesthetic adverse events.

Methods

The PDA software was developed from a handwritten blueprint, with the emphasis of the program being on ease of use. In association with a software developer (Sync International, Victoria, Australia) over a period of four months and five revisions, a definitive version of the program was produced which included a logbook (Box 1) and cusum self-assessment tool. The software allowed users to review all past data entries, filter this information to obtain a summary of caseload or procedural exposure, annotate interesting cases and make handwritten notes on the PDA.

Six first- or second-year registrars from two tertiary referral centres (in Victoria and New Zealand) and a large district hospital in Queensland and their training supervisors were recruited for data collection over a four- to seven-month period, from March to October 2000. Each trainee was supplied with a Palm Ille PDA (Palm Computing Incorporated, California, USA) loaded with the experimental software. Registrars and their supervisors had a one-hour educational session before beginning data collection. Ongoing technical support via the training supervisors was provided as required.

The registrars recorded logbook entries for each anaesthetic procedure undertaken. Eight anaesthetic critical event categories (Box 2) were included as drop-down menu items in the software. If a defined anaesthetic incident occurred it was recorded using the menu along with the outcome of the event (eg, uneventful, minor morbidity, major morbidity or death). In this way the true incidence of adverse events could be determined.

The data from the PDAs were uploaded onto a secure departmental desktop computer for storage and the generation of cusum graphs with Microsoft Access and Microsoft Excel software.

Cusum

Four specific procedures were selected for cusum analysis: 8,9 spinal anaesthesia, epidural anaesthesia, central venous catheter (CVC) insertion, and arterial line insertion. Success or failure for these procedures was recorded based on defined criteria (Box 3).

The cusum technique lends itself to analysis of continuous processes. To use cusum, one needs to determine appropriate acceptable and unacceptable failure rates, which, for this project, were declared to be 10% and 30%, respectively. One also needs to determine a nominal alpha and beta. Alpha is the probability of making a type one error, equivalent to the chance of wrongly declaring a trainee's failure rate to be unacceptable. Conversely, beta is the probability of making a type two error, which, in this context, would be wrongly declaring a trainee's failure rate to be acceptable. For technical and practical reasons, when using cusum analysis alpha and beta are normally both set at 0.1.9

Knowing these four parameters, a given sequence of success/failure data can be presented on a cusum plot. A cusum plot is a graphical representation of the trend in the outcome of a series of consecutive procedures. If the trend in performance is negative, the cusum plot slopes upward. A trend towards acceptable performance is shown by a downwards slope on the cusum plot. The benefit of the cusum plot is that it readily identifies any change in performance (Box 4).

Results

During the study period 1690 anaesthetic cases were logged: 62% were performed with level 1 supervision as defined by the Australian and New Zealand College of Anaesthetists10 (Box 5), 22% with level 2 supervision, 15% with level 3 supervision and 1.3% with level 4 supervision. Twenty-seven per cent of procedures were performed out of hours. The main operation classes were general surgery (22%), orthopaedics (17%), plastic surgery (15%) and obstetrics (10%). Eleven per cent of procedures were performed in remote locations outside the operating suite (general ward, radiology, cardiology, and magnetic resonance imaging).

Four hundred and eighty procedures were logged for cusum analysis. Ninety of these were excluded from analysis, as the supervisor of the case performed them (Box 6). The cusum plots showed variation in the failure rates between registrars, but all were within the acceptable range according to the predefined parameters.

Forty-two critical incidents were recorded, giving a critical incident rate of 2.5%. Twenty-one critical incidents were airway/respiratory events, 17 were cardiovascular, and one each of pharmacological, severe hypothermia, equipment, and regional-procedural (dural tap). Of these 42 events, 19 had uneventful outcomes, eight had minor morbidity, 14 had major morbidity and there was one death.

An estimated 10% of data were lost owing to operator and equipment errors. These problems related to PDA breakage or battery failure. Data loss occurred because desktop computer software for backing up data was not introduced until midway into the project.

Discussion

PDAs are a technology in their infancy. Released in 1995, their adoption has been remarkable, with over 15 million units sold worldwide (Palm Inc Australia, personal communication). The medical profession has embraced this new technology, with an estimated 26% of practitioners using PDAs, a number estimated to grow to 50% by 2005.6

We encountered significant practical problems resulting in major data loss with the model of PDA used in our trial. Our PDAs are now protected physically with a durable external case, which increases their robustness for theatre use and has prevented further breakages. We now have a reliable backup system using an Internet-based program to upload data from the PDA to a secure server, which can be accessed using any modem or compatible mobile phone. This also gives us the ability to download updated software to the PDA. Information and analysis of the data can be viewed via a website that is accessible only with a password.

The use of logbooks does not provide a qualitative analysis of performance, and indeed this has been one of their major criticisms.10 Logbooks do provide evidence of exposure to cases and techniques, and one might hope that the greater the exposure the higher the level of performance. The Australian and New Zealand College of Anaesthetists training guidelines stipulate supervision requirements for various levels of training,11 as well as exposure parameters for practices such as labour ward epidurals, neurosurgical cases, double-lumen tubes and "out of hours" procedures.12 This portable logbook can easily provide this information, as well as details on exposure to subtypes of operation, age ranges of patients, American Society of Anesthesiologists scores, amount of out-of-hours work, and types and numbers of critical incidents.

The desirable depth of information recorded in a logbook is a contentious issue, and we have found opinions vary widely. Our logbook has more depth than that of the UK Royal College of Anaesthetists, but, despite this, data entry by an experienced user took less than 45 seconds for an average case.

While exposure to procedures may lead to improved performance, this may not always be the case, as shown by the cusum graphs (Box 4). From the first graph it can be seen that the registrar has shown a continuous improvement in performance in the study period. The second graph reveals an initial unacceptable performance, followed by a continued stable level of performance. This is indicative of a learning curve and one would hope that, with continued experience, an improvement in performance would be indicated by the graph trending downwards.

The advantage of serially reviewing performance is that even a small deviation (especially deleterious) is revealed early, allowing timely intervention. Likewise, excellent performance can identify practitioners capable of training more junior trainees, and this can enhance training programs.

Although our pilot study limited cusum analysis to four basic procedures, our software has been designed to allow cusum audit of any other procedure designated by the operator. All one need do is define success and failure for a procedure and commence data collection.

Objectively defining success or failure for a task is probably the most difficult part of the process. Our objective criteria (Box 3) were obtained by consensus of the working group. This is the aspect of cusum analysis that is highly interpretive and does warrant further debate. Previous studies have used different criteria for determining success and failure.8

Procedural analysis is only one aspect of training and will only ever be one component in competency assessment. Throughout training, academic knowledge is tested rigorously, but assessment of procedural competency, professionalism and crisis-management skills may be neglected. Competency assessment would be aided by cusum analysis of these less rigorously analysed areas.

The recording of adverse incidents in anaesthesia is at present a generally haphazard affair. The major advantages of data entry at the point of care are that events can be recorded soon afterwards, and the incidence of given events can be rapidly tabulated with both a numerator and denominator. In our study we found an adverse incident rate of 2.5%. Having both a numerator and denominator can only improve the relevance of data for daily clinical practice.

There are few published prospective studies of the rate of anaesthetic incidents, with reported incident rates ranging from 0.76% to 6.70%.13,14 One of the difficulties of tabulating an incident rate is the lack of universal definitions for a critical incident.15 Our aim was to collect data on incidents that resulted in, or had the potential to result in, patient morbidity or mortality. While, in theory, this list could be quite exhaustive, we restricted our drop-down menu to major items, with the ability to write notes for unusual or rare events.

Finally, two philosophical issues need to be addressed. Firstly, the ownership of the data collected is important. As the primary aim is to allow a practitioner to independently review his or her own performance, we believe the data should remain confidential to the individual anaesthetist. Our main aim is for self-analysis with secure data. Secondly, there is the problem of reliability of the data. Any self-assessment tool relies on the integrity of the person entering data. This is the case with the ANZCA Maintenance of Professional Standards program. The ability to obtain maximal value from a system such as this is highly dependent on the operator. We adhere to the concept that Australasian anaesthetists want to do the very best for their patients, and we have aimed to give them a tool to help them achieve this.

1: Logbook data fields

  • Date and time stamp

  • Out-of-hours work

  • Unit record number and date of birth

  • American Society of Anesthesiologists (ASA) status

  • Supervision level

  • Type and subtype of surgery

  • Anaesthesia technique

  • Regional techniques

  • Procedures

A screenshot of the PDA logbook.

2: Anaesthetic complications menu items

  • Airway

  • Cardiovascular

  • Respiratory

  • Central nervous system

  • Equipment

  • Pharmacological

  • Regional/procedural

  • Temperature

3: Definitions of successful procedures for cusum analysis

Epidural anaesthesia

  • Loss of resistance felt and catheter fed with functioning epidural.

  • Complete independence or only minimal intervention by supervising consultant.

  • Less than 30 minutes from time of skin infiltration to the application of catheter dressing.

  • Sterile technique maintained.

  • No significant morbidity.

  • Doctor–patient interaction was adequate.

Spinal anaesthesia

  • Cerebrospinal fluid flowed and there was onset of a functioning block.

  • Complete independence or only minimal intervention by supervising consultant.

  • Less than 30 minutes from time of skin infiltration to removal of the needle.

  • Sterile technique maintained.

  • No significant morbidity.

  • Doctor–patient interaction was adequate.

Central venous catheter insertion

  • Less than 30 minutes from time of skin infiltration to application of catheter dressing with correct placement of line.

  • Sterile technique maintained.

  • No significant morbidity, including large-needle cannulation of artery.

  • Doctor–patient interaction was adequate.

Arterial line insertion

  • Less than 30 minutes from time of skin infiltration to application of catheter dressing with functional arterial line.

  • Sterile technique maintained.

  • No significant morbidity, including large haematoma.

  • Doctor–patient interaction was adequate.

4: Cusum plots for two registrars

Note horizontal boundary lines. A cusum plot which spans the gap between two boundary lines in the downward direction is proof of acceptable performance according to the defined parameters. Conversely, a plot which spans the gap between two boundary lines in the upward direction is proof of unacceptable performance according to the defined parameters for the intervening series of procedure attempts.The first plot (of a registrar performing epidural anaesthesia) shows a steady trend of improving performance; the second (a registrar inserting arterial lines) shows a more difficult progress, with alternating stages of acceptable and unacceptable performance.

5: Supervision levels for trainees of the Australian and New Zealand College of Anaesthetists

1 Supervisor rostered to one trainee and available solely for that trainee.

2 Supervisor rostered to supervise two trainees who are in anaesthetising locations in close proximity.

3 Supervisor available in the medical institution but is not exclusively available for a specific trainee.

4 Supervisor is not in the medical institution but is on call.

6: Procedures undertaken by study participants for cusum analysis*

Procedure

Success

Failure


Arterial line

76 (84%)

15 (16%)

Central venous catheter

34 (94%)

2 (6%)

Spinal

106 (90%)

12 (10%)

Epidural

125 (86%)

20 (14%)


Ninety of these procedures were also undertaken by the supervisors: 22 arterial line, 9 central venous catheter, 17 spinal and 42 epidural.

  • Paul D Bent1
  • Stephen N Bolsin2
  • Bernie J Creati3
  • Andrew J Patrick4
  • Mark E Colson5

  • Department of Anaesthesia, Pain Management and Peri-operative Medicine, Barwon Health, Geelong, VIC.

Correspondence: paulbent@bigpond.net.au

Acknowledgements: 

Mark Runge and Steve Prior, from Sync International Pty Ltd, assisted with software design and production.

Competing interests:

None identified. Funding for this project was secured from Palm Corporation Australia, the Australian and New Zealand College of Anaesthetists (ANZCA), United Medical Protection and the Victorian Medical Indemnity Association. The sponsors had no role in study design, data collection, analysis and interpretation, the writing of the article or the decision to submit the final manuscript for publication.

  • 1. Bolsin SN. The Bristol cardiac disaster. BMJ 1998; 317: 1579-1580.
  • 2. Bolsin SN. Professional misconduct: the Bristol Case. Med J Aust 1998; 169: 369-372.
  • 3. De Leval M, Francois K, Bull C, et al. Analysis of a cluster of surgical failures: application of cusum to a series of neonatal arterial switch operations. J Thorac Cardiovasc Surg 1994; 107: 914-923.
  • 4. Greaves JD, Grant J. Watching anaesthetists work: using the professional judgement of consultants to assess the developing clinical competence of trainees. Br J Anaesth 2000; 84: 525-533.
  • 5. Morriss W. How many anaesthetics did you give last year? Using a logbook to document personal training experience. ANZCA Bulletin 2001; 10: 25-29.
  • 6. Larkin M. Can handheld computers improve the quality of care? Lancet 2001; 358: 1438.
  • 7. Wilcox RA, La Tella RR. The personal digital assistant: a new medical instrument for the exchange of clinical information at the point of care. Med J Aust 2001; 175: 659-662. <MJA full text>
  • 8. Kestin I. A statistical approach to measuring the competence of anaesthetic trainees at practical procedures. Br J Anaesth 1995; 75: 805-809.
  • 9. Bolsin S, Colson M. Methodology matters; CUSUM. Int J Qual Health Care 2000; 12: 433-438.
  • 10. Davis MH, Dent JA. Comparison of student learning in the outpatient setting and ward round. Med Educ 1994; 28: 218-212.
  • 11. Australian and New Zealand College of Anaesthetists: Professional Document TE3: Supervision of clinical experience for trainees in anaesthesia. Melbourne: ANZCA, 1999.
  • 12. Australian and New Zealand College of Anaesthetists: Objectives of training in anaesthesia. 3rd edition. Melbourne: ANZCA, 2000.
  • 13. Short TG, O'Regan A, Lew J, Oh TE. Critical incident reporting in an anaesthetic department quality assurance programme. Anaesthesia 1993; 48: 3-7.
  • 14. Findlay GP, Spittal MJ, Radcliffe JJ. The recognition of critical incidents: quantification of monitor effectiveness. Anaesthesia 1998; 53: 595-598.
  • 15. Banks IC, Tackley RM. A standard set of terms for critical incident recording. Br J Anaesth 1994; 73: 703-708.

Author

remove_circle_outline Delete Author
add_circle_outline Add Author

Comment
Do you have any competing interests to declare? *

I/we agree to assign copyright to the Medical Journal of Australia and agree to the Conditions of publication *
I/we agree to the Terms of use of the Medical Journal of Australia *
Email me when people comment on this article

Responses are now closed for this article.