Connect
MJA
MJA

The Threats to Australian Patient Safety (TAPS) study: incidence of reported errors in general practice

Meredith A B Makeham, Michael R Kidd, Deborah C Saltman, Michael Mira, Charles Bridges-Webb, Chris Cooper and Simone Stromer
Med J Aust 2006; 185 (2): 95-98.
Published online: 17 July 2006

Abstract

Objective: To determine the incidence of errors anonymously reported by general practitioners in NSW.

Design: The Threats to Australian Patient Safety (TAPS) study used anonymous reporting of errors by GPs via a secure web-based questionnaire for 12 months from October 2003.

Setting: General practices in NSW from three groupings: major urban centres (RRMA 1), large regional areas (RRMA 2–3), and rural and remote areas (RRMA 4–7).

Participants: 84 GPs from a stratified random sample of the population of 4666 NSW GPs — 41 (49%) from RRMA 1, 22 (26%) from RRMA 2–3, and 21 (25%) from RRMA 4–7. Participants were representative of the GP source population of 4666 doctors in NSW (Medicare items billed, participant age and sex).

Main outcome measures: Total number of error reports and incidence of reported errors per Medicare patient encounter item and per patient seen per year.

Results: 84 GPs submitted 418 error reports, claimed 490 864 Medicare patient encounter items, and saw 166 569 individual patients over 12 months. The incidence of reported error per Medicare patient encounter item per year was 0.078% (95% CI, 0.076%–0.080%). The incidence of reported errors per patient seen per year was 0.240% (95% CI, 0.235%–0.245%). No significant difference was seen in error reporting frequency between RRMA groupings.

Conclusions: This is the first study describing the incidence of GP-reported errors in a representative sample. When an anonymous reporting system is provided, about one error is reported for every 1000 Medicare items related to patient encounters billed, and about two errors are reported for every 1000 individual patients seen by a GP.

The importance of better understanding error and safety in primary care is widely accepted, with recent calls to promote efforts and improve resources in this area of research.1 A few studies have described errors in general practice using the incident monitoring technique,2-10 but none have quantified the problem.

A study of family medicine physicians in Cincinatti, USA, showed errors were identified in almost a quarter of clinical encounters.2 An error rate of 75.6 per 1000 general practice appointments (7.6%) was reported in one UK study.3 Reviewing incident reports from an outpatient primary care setting, a US study reported a prevalence of adverse events of 3.7 per 100 000 clinic visits.11

The Threats to Australian Patient Safety (TAPS) study is the first to estimate the incidence of errors reported by general practitioners in the community from a representative sample, and describes a new method for reporting errors using a secure web-based tool.

Methods
Ethical approval

Approval was obtained from the University of Sydney Human Research Ethics Committee. The investigating committee was granted qualified privilege by the NSW Minister for Health, providing confidentiality of meeting minutes and protection from their use in legal actions.

Definition of error

We adopted the definition of “error” used in previous work.5,6,12,13 Errors may have been attributable to the reporter’s actions or other unwanted occurrences:

“Errors are events in your practice that make you conclude: ‘That was a threat to patient wellbeing and should not happen. I don’t want it to happen again.’ Such an event affects or could affect the quality of care you give your patients. Errors may be large or small, administrative or clinical, or actions taken or not taken. Errors may or may not have discernible effects. Errors in this study are anything that you identify as something wrong, to be avoided in the future.”

Sampling method

The General Practice Programs Branch of the Commonwealth Department of Health and Ageing (GP Branch) provided a stratified random sample of 320 names from the population of 4666 vocationally registered GPs in NSW whose Medicare billings in the previous quarter indicated that they were engaged in full-time work.14

A sample size of 80 was calculated from pilot study results of 134 reports made from up to 23 GPs over 4 months, which had a 25% participation rate.5 Allowing 1 week of absence each and estimating 132 consultations per week,15 we hypothesised a reported error rate of 0.27% per consultation. To estimate this with a 95% confidence interval of 0.25%–0.29%, we required 205 465 patient encounters. We anticipated that 80 GP participants would have about 480 000 patient encounters, with the excess allowing for any effect of clustered samples.

The sample was drawn from Rural, Remote and Metropolitan Area (RRMA) groupings of RRMA 1, RRMA 2–3, and RRMA 4–7.14 The GP Branch stratified the sample by gender and age, dichotomised around 45 years. One hundred and sixty (50%) of the sample were drawn from RRMA 1, 80 (25%) from RRMA 2–3 and 80 (25%) from RRMA 4–7 regions. The proportions of the source population in these regional groupings were 68%, 16% and 16%, respectively. The smaller groups were over-sampled to ensure adequate counts of Medicare items to allow future subgroup analysis.

A research assistant made initial contact with the GPs by telephone. If interest was expressed, an information pack was sent, followed by further telephone contact. Reasons for non-participation were recorded. All GPs who commenced the study were visited by a research assistant. Informed consent was obtained, the definition of “error” was explained, and a demonstration of the website to be used for reporting was given.

Anonymous online error reporting

A secure website and reporting process were developed to house the error questionnaire, shown in Box 1, which was piloted in 2001.5 Data transmission was protected by 128-bit encryption and the website was hosted by a secure server behind a firewall. The website allowed participants to contact investigators and view linked medical resources. Participants were asked about any website problems in monthly contact via email, telephone or facsimile.

A self-chosen Personal Identification Number (PIN) and the RRMA grouping were required to submit a report. Participants submitted a blank report before the start of the study for website testing. The participant’s PINs were unknown to investigators, which protected reporter anonymity.

Comparing participants to other GPs

Total numbers of GPs, their sex and average age were obtained from the GP Branch by RRMA groupings, so that the study participants could be compared with the source population. We also compared the number of Medicare items billed relating to a patient encounter over the study duration.

Calculating the number of reported errors

Numbers of errors reported per PIN, the numbers of PINs obtained per RRMA group, and total numbers of errors from the group were obtained from website data. At the end of the study, participants were asked whether they had sent any reports electronically. To protect anonymity, we did not ask the number of reports submitted.

Calculating the incidence of reported error

Total numbers of errors from participants, counts of the number of Medicare items billed that related to a patient encounter and the number of individual patients that were seen during the study were used in incidence calculations. Patients who were seen on more than one occasion during the year by a single participant were only counted once per GP.

Data analysis

The percentage of female participants was compared with the source population using Fisher’s exact test. The average participant age and number of Medicare items relating to patient encounters were compared using unpaired Student’s t tests. A non-parametric method adjusting for clustering was used to compare the median number of reports per PIN. Incidence calculations included sampling weights for the three RRMA groupings to adjust for the relative over-sampling of RRMA 2–3 and RRMA 4–7 GPs. All statistical analysis was performed using Stata 8.0 software (StataCorp, College Station, Tex, USA).

Results
Participants

From the initial sample of 320, we were unable to contact 52 GPs (16.3%) who had moved away, retired, or were completely unknown to the practice. A further 43 GPs (13.4%) did not return researchers’ calls or gave no specific reason for declining. Of those remaining, 91 (40.4%) were too busy or not interested, 20 (8.9%) had difficulties with computer use or access, 18 (8.0%) were retiring or taking extended leave during the study, eight (3.6%) did not participate in research, three (1.3%) had concerns about privacy, and one (0.4%) had concerns about bad publicity. There were 84 GPs (26.3% of the original sample) who agreed to commence the study, with 41 from RRMA 1, 22 from RRMA 2–3, and 21 from RRMA 4–7 regions. One RRMA 1 GP and one RRMA 4–7 GP left the study. As reporting was anonymous, their results were included in further calculations.

Comparing participants with other GPs

The participants’ average age, percentage of female GPs, and average number of Medicare patient encounter items were assessed. There was no difference in comparison to the source population within each RRMA grouping, or for all combined, using any of these measures at the 5% significance level.

Number of error reports submitted

There were 418 TAPS reports from 85 PINs describing an error during the study. The mean number of error reports per month was 34.8 (range, 15–60). At feedback interview, 79 of the 82 completing participants had submitted a report.

Box 2 shows RRMA grouping comparisons of the number of reports submitted, PINs submitted, GPs who reported at interview sending a report, average number of reports per GP, and the median number of reports per PIN over the study duration.

Frequency of error reporting

The distribution of reporting frequencies among RRMA groupings was assessed by looking at the number of reports per PIN, shown in Box 3. This varied from one to 25, with a median of three for all participants combined. There was no significant difference between RRMA groupings in the median number of reports submitted per PIN (P = 0.40) (for this analysis, values that were equal to the median were evenly split between the above and below groups).

Incidence of reported error

The total numbers of Medicare items relating to a patient encounter and total numbers of individual patients seen by RRMA grouping are shown in Box 4. Each report within an RRMA grouping was matched with the group’s average number of Medicare items and average number of patients seen, and assigned a sampling weight to adjust for the larger representation of RRMA 2–3 and RRMA 4–7 GPs in the sample design.

The resulting calculations gave an incidence of anonymously reported errors per Medicare patient encounter item per year of 0.078% (95% CI, 0.076%–0.080%), and an incidence of reported errors per patient seen per year of 0.240% (95% CI, 0.235%–0.245%).

Discussion

Original research on errors and adverse events in primary care consists of relatively limited numbers of studies, focusing on qualitative descriptions.16 The TAPS study provides the first calculated incidence of reported error from a representative random sample of GPs. Despite the small sample size and participation rate of 26%, these findings are likely to be generalisable to GPs in NSW, as we detected no statistically significant differences between the age, sex, and Medicare billings when comparing participants to the source population of 4666 GPs. Comparative data were not available from the GP Branch for the non-participants due to consent issues.

We conclude from our incidence findings that, when an anonymous and simple reporting system is provided, about one error is reported for every 1000 Medicare items related to patient encounters billed per year, and about two errors are reported for every 1000 individual patients seen by a GP per year.

It has been suggested previously that an error database in general practice should be established.17,18 Our results indicate that using a secure website with an anonymous reporting method is a practical way of collecting error information from GPs, as 94% of doctors who enrolled in the study reported at interview that they sent an error report.

Our findings provide an estimation of the incidence of errors in general practice that would be reported if an appropriate reporting system were in place, but this should not be interpreted as an estimate of the total number of errors that participants noticed, or of the underlying number of errors occurring in the GP community. It is very difficult to assess the proportion of errors that would go unreported even when a reporting system such as TAPS is available. A GP may not be aware that an error has occurred. GPs have been found to under-report adverse drug events,19 and so other patient safety threats may also be under-reported.

Anonymity was of major importance in our methodological design, to encourage reporting. There has previously been a reluctance of health professionals to address the problem of errors due to feelings of guilt and a desire to avoid disapproval from colleagues.20 However, using a PIN that was unknown to investigators led to some difficulties in interpreting reports per GP. Researchers were unable to assist if a PIN was forgotten, and there was no way of knowing how many GPs changed PINs during the study. Seventy-nine GPs gave feedback that they submitted an error report, but 85 PINs were used for error reports, which suggests that on at least six occasions GPs changed their PINs. Our estimate of report numbers per participant, which is based on PINs, could therefore be an underestimate.

The number of individual Medicare items billed relating to a patient encounter is the best approximation that we could make of separate consultations, but should not be interpreted directly as such. A GP may claim more than one Medicare item from a single encounter, which is not revealed in our data. Consultations may also have occurred which were not billed, charged to the patient privately, or charged to the Department of Veterans’ Affairs, which keeps separate data from the GP Branch.

To our knowledge, there are no studies directly comparable with our incidence results, which are lower than hypothesised. The authors of a US study that found an error rate close to 25% state that their results were not generalisable due to limitations of the study.2 Only a small group of 15 doctors in seven practices made reports, completing a review form after every consultation during three half-day sessions. The UK finding of a 7.6% error rate was based on a 2-week collection of errors from 10 practices in a single city.3 However, a major difference is that this result was based upon a collection of reports from a large cross-section of staff working in general practice, with 163 people able to submit reports, and a high proportion of participants who had administrative rather than clinical roles.

Our method of prospective self-reporting appears to have returned a higher rate of error detection than that in a US study using retrospective record review in eight primary health care clinics.11 Again, direct comparison is difficult, as medical adverse events were counted in this work, rather than errors in the sense that we defined.

In considering the adoption of a method of error reporting for primary care, retrospective record review lacks potential for use as an active learning tool. A prospective electronic reporting tool could be enhanced with the addition of feedback to the user and links to educational activities, plus have the scope to allow many other stakeholders in primary care settings to contribute error information.

The TAPS study improves our current understanding of the incidence of reporting threats to patient safety in general practice, and its method provides a way for GPs to discuss errors in a non-threatening environment.

1 Error questionnaire used in the Threats to Australian Patient Safety (TAPS) study

Question

Answer format


a) What is the RRMA area of your practice?

RRMA 1, 2–3 or 4–7

b) Is the problem related to a specific patient?

Yes/No

c) How well do you know the patient?

5-point Likert scale

d) What is the patient’s age?

Free text

e) What is the patient’s sex?

Male/Female

f) Is the patient from a non-English speaking background?

Yes/No

g) Is the patient of Aboriginal or Torres Strait Islander descent?

Yes/No

h) What happened? Please consider what, where and who was involved.

Free text

i) What was the result? Please think about actual and potential consequences.

Free text

j) What may have contributed to this error? Please consider any special circumstances.

Free text

k) Where did the error happen? Choose all that apply from Office or surgery, Nursing home, Hospital, Patient’s home, Telephone contact, Emergency Room, Laboratory, Pharmacy, Radiology

Check a box or boxes: nine choices

l) To your knowledge, was any patient harmed by this error?

Yes/No

m) If yes, how would you rate the seriousness of this harm?

5-point Likert scale

n) How often does this error occur in your practice? First time, Seldom 1–2 per year, Sometimes 3–11 per year, Frequently >1/month

Check a box: four choices

o) What could have prevented this error? Please consider what could be done to prevent similar episodes from happening in the future.

Free text

p) Other comments?

Free text


RRMA = Rural, Remote and Metropolitan Area.

2 Summary statistics describing TAPS reports submitted by RRMA group over the study period

RRMA

group

Number of participants

TAPS reports received

Unique PINs submitted

Number of GPs who sent a report *

Average reports per GP

Median reports per unique PIN


1

41

199

46

38

5.2

3

2–3

22

121

18

21

5.8

6

4–7

21

98

21

20

4.9

4

Total

84

418

85

79

5.3

3


*According to feedback provided at interview. TAPS = Threats to Australian Patient Safety. RRMA = Rural, Remote and Metropolitan Area. PIN = personal identification number.

3 Frequency of error reporting per unique PIN by RRMA grouping over the study period


RRMA = Rural, Remote and Metropolitan Area. PIN = personal identification number.

4 The number of Medicare items relating to a patient encounter and individual patients seen by participants during the study

RRMA 1

RRMA 2–3

RRMA 4–7

All RRMA


Medicare Items

260 150

127 610

103 104

490 864

Patients seen

89 042

45 176

32 351

166 569

  • Meredith A B Makeham1
  • Michael R Kidd1
  • Deborah C Saltman1
  • Michael Mira1
  • Charles Bridges-Webb2
  • Chris Cooper1
  • Simone Stromer1

  • 1 Discipline of General Practice, The University of Sydney, Sydney, NSW.
  • 2 RACGP NSW Projects, Research and Evaluation Unit, The Royal Australian College of General Practitioners, Sydney, NSW.

Correspondence: makeham@ozemail.com.au

Acknowledgements: 

Meredith Makeham is a National Health and Medical Research Council scholarship holder. We gratefully acknowledge the contribution of the 84 NSW GPs who provided the data, and the funding from the National Health and Medical Research Council and the Primary Health Care Research, Evaluation and Development program. We also thank Mr Tony Lawrence and Ms Angela Mikalauskas of the General Practice Branch, Commonwealth Department of Health and Ageing, for assistance with sampling and Medicare data provision, and Ms Geraldine Card for her valuable contribution in managing the study and collating results.

Competing interests:

None declared.

  • 1. Jacobson L, Elwyn G, Robling M, Jones RT. Error and safety in primary care: no clear boundaries. Fam Pract 2003; 20: 237-241.
  • 2. Elder NC, Vonder Meulen M, Cassedy A. The identification of medical errors by family physicians during outpatient visits. Ann Fam Med 2004; 2: 125-129.
  • 3. Rubin G, George A, Chinn DJ, Richardson C. Errors in general practice: development of an error classification and pilot study of a method for detecting errors. Qual Saf Health Care 2003; 12: 443-447.
  • 4. Dovey SM, Phillips RL, Green LA, Fryer GE. Types of medical errors commonly reported by family physicians. Am Fam Physician 2003; 67: 697.
  • 5. Makeham MA, Dovey SM, County M, Kidd MR. An international taxonomy for errors in general practice: a pilot study. Med J Aust 2002; 177: 68-72.
  • 6. Dovey SM, Meyers DS, Phillips RL Jr, et al. A preliminary taxonomy of medical errors in family practice. Qual Saf Health Care 2002; 11: 233-238.
  • 7. Steven ID, Malpass A, Moller J, et al. Towards safer drug use in general practice. J Qual Clin Pract 1999; 19: 47-50.
  • 8. Bhasale A. The wrong diagnosis: identifying causes of potentially adverse events in general practice using incident monitoring. Fam Pract 1998; 15: 308-318.
  • 9. Bhasale AL, Miller GC, Reid SE, Britt HC. Analysing potential harm in Australian general practice: an incident-monitoring study. Med J Aust 1998; 169: 73-76.
  • 10. Britt H, Miller GC, Steven ID, et al. Collecting data on potentially harmful events: a method for monitoring incidents in general practice. Fam Pract 1997; 14: 101-106.
  • 11. Fischer G, Fetters MD, Munro AP, Goldman EB. Adverse events in primary care identified from a risk-management database. J Fam Pract 1997; 45: 40-46.
  • 12. Woolf SH, Kuzel AJ, Dovey SM, Phillips RL Jr. A string of mistakes: the importance of cascade analysis in describing, counting, and preventing medical errors. Ann Fam Med 2004; 2: 317-326.
  • 13. Tilyard M, Dovey S, Hall K. Avoiding and fixing medical errors in general practice: prevention strategies reported in the Linnaeus Collaboration’s Primary Care International Study of Medical Errors. N Z Med J 2005; 118: U1264.
  • 14. Budget and Performance Branch, Primary Care Division, Department of Health and Ageing. General Practice in Australia: 2004. 1st ed. Canberra: Commonwealth of Australia, 2005.
  • 15. General Practice Branch, Department of Health and Aged Care. General Practice in Australia: 2000. 1st ed. Canberra: Commonwealth of Australia, 2000.
  • 16. Elder NC, Dovey SM. Classification of medical errors and preventable adverse events in primary care: a synthesis of the literature. J Fam Pract 2002; 51: 927-932. [Erratum appears in J Fam Pract 2002; 51: 1079.]
  • 17. Sheikh A, Hurwitz B. Setting up a database of medical error in general practice: conceptual and methodological considerations. Br J Gen Pract 2001; 51: 57-60.
  • 18. Runciman WB. Lessons from the Australian Patient Safety Foundation: setting up a national patient safety surveillance system — is this the right model? Qual Saf Health Care 2002; 11: 246-251.
  • 19. Moride Y, Haramburu F, Requejo AA, Begaud B. Under-reporting of adverse drug reactions in general practice. Br J Clin Pharmacol 1997; 43: 177-181.
  • 20. Kidd MR, Veale B. How safe is Australian general practice and how can it be made safer [editorial]? Med J Aust 1998; 169: 67-68.

Author

remove_circle_outline Delete Author
add_circle_outline Add Author

Comment
Do you have any competing interests to declare? *

I/we agree to assign copyright to the Medical Journal of Australia and agree to the Conditions of publication *
I/we agree to the Terms of use of the Medical Journal of Australia *
Email me when people comment on this article

Responses are now closed for this article.