Design, setting and patients: A retrospective observational study conducted during a 6-month period in 2009 after the introduction of FirstNet and a corresponding 6-month control period in 2008 when the Emergency Department Information System (EDIS) was operational. Data from all patients presenting to the ED during each period were extracted from each system and analysed for changes in key performance indicators (KPIs). Potential confounding variables, including total number of ED presentations, ED occupancy, ED staffing levels and triage category distribution, were also collected.
Main outcome measures: Waiting time for all patients; waiting time, treatment time and total time for patients discharged from the ED; proportion of patients who did not wait to be seen by a doctor (DNW rate); and proportion of ambulance offload waiting times longer than 30 minutes.
Results: We found a reduction in performance with respect to ED KPIs after implementation of the FirstNet system. There were increases in the waiting time for all patients (median, 40 min v 78 min), and the waiting time (median, 49 min v 87 min), treatment time (median, 128 min v 147 min) and total time (median, 214 min v 280 min) for patients discharged from the ED. There were increases in the DNW rate (8.3% v 15.6%) and the proportion of ambulance offload times longer than 30 minutes (10.5% v 13.3%). All differences were statistically significant (P < 0.05).
Electronic medical record (EMR) systems have the potential to improve quality of health care, streamline workflow and increase efficiency in the health care system.1,2 However, potential problems have also been identified, such as the cost of implementing and maintaining EMR systems, and the skills and training needed for using them.1,2
Despite the proposed benefits and pitfalls of EMR systems, there is limited research on their effectiveness.2-4 Specifically, there is a lack of data regarding their effect on definable clinical end points, such as patient mortality.5 Locally developed systems, designed around local procedures and conditions and implemented from the ground up, have shown benefit.6-8 Specific computer-assisted decision support has also been shown to be of benefit.7-9 A clinical information system must be fit for purpose for any gain to be made.10
FirstNet (Cerner), an EMR system, has been introduced in emergency departments (EDs) around New South Wales since 2007. This system replaced the Emergency Department Information System (EDIS; Healthcare Group, CSC), which was previously used in most NSW EDs and is still in use in some. Despite limited literature indicating that FirstNet has decreased performance in EDs in Australia,10,11 and reports of problems with Cerner programs overseas,12-15 there were no objective quantitative data to substantiate these concerns.
FirstNet was introduced to the Nepean Hospital ED in March 2009. All medical documentation became electronic, including doctors’ examination notes, nursing progress notes, patient observations, and electronic ordering of pathology and radiology investigations. Our study’s aim was to determine whether the implementation of FirstNet was associated with an improvement or decline in key performance indicators (KPIs) of patient flow in the Nepean Hospital ED.
We conducted a retrospective observational study comparing ED performance before and after FirstNet was introduced on 24 March 2009. Nepean Hospital is a 453-bed tertiary referral and teaching hospital serving the western suburbs of Sydney, NSW. The ED sees about 54 000 patients each year, with a 20% paediatric caseload and a 36% admission rate.
The control group consisted of patients presenting during the 6-month period from July to December 2008, when EDIS (version 9) was in use. During this period, only patient triage and tracking was electronic, with all other ED functions relying on paper documentation. The study group consisted of patients presenting during the 6-month period from July to December 2009. This period was considered appropriate because the new FirstNet EMR system had then been operational for more than 3 months. The version of FirstNet in use at Nepean Hospital was the standardised state-based build slated to be introduced across all hospitals in NSW.
Outcome measures were: waiting time for all patients; waiting time, treatment time and total time for discharged patients (see definitions in Box 1); proportion of patients who did not wait to be seen by a doctor (DNW rate); proportion of ambulances with offload waiting times longer than 30 minutes; and mean number of patients seen per non-specialist doctor per shift.
We also collected data on potential confounding variables: number of presentations to the ED; number of presentations by Australasian Triage Scale category; mean daily ED occupancy; and number of shifts worked per week by ED specialists and non-specialists. For this analysis, mean daily ED occupancy (Box 1) was used as a measure of overcrowding. Data relating to access block were not collected.
The numbers of shifts worked by medical staff were retrieved from ED rosters. The number of patients seen by each doctor per shift was extracted from EDIS but could not be extracted from FirstNet. Therefore, a mean number of patients seen per doctor per shift was calculated using the FirstNet data on presentations and staffing levels.
The data were imported into SPSS version 19 (IBM). The distribution of waiting time, treatment time and total time was non-normal, and these results are reported as medians with interquartile ranges and compared using the Mann–Whitney U test. All proportional data were compared using the Pearson χ2 test. The mean number of patients seen per non-specialist doctor per shift and ED occupancy were compared using the independent t test. We used statistical modelling to attempt to control for the effect of potential confounding variables (see Appendix).
There were 25 620 presentations in the 2008 control period and 26 128 in the 2009 study period. Triage category distribution was similar in each period (Box 2).
There were too few patients in triage category 1 for meaningful analysis of this subgroup. We also found that data on disposition status (did not wait v discharged v admitted) for most patients presenting in July and August 2009 were missing in FirstNet. Therefore, only ambulance offload data could be analysed for the full 6 months. All other comparative measures used data from September to December only of the control and study periods (16 976 and 17 281 presentations, respectively).
There was a statistically significant increase in waiting time for all patients (Box 3) and in waiting time, treatment time and total time for discharged patients (Box 4), both overall and for most triage categories. The waiting time for triage category 2, for all patients and discharged patients, was shorter after the introduction of FirstNet, but treatment time and total time increased for discharged patients in this category. Treatment time for triage category 4 patients discharged from the ED was also shorter after the introduction of FirstNet, but there was no improvement in waiting time or total time for these patients. There was also a reduction in treatment time and total time for category 5 patients discharged from the ED.
After the introduction of FirstNet, there was a statistically significant increase in the DNW rate and the proportion of ambulances with offload waiting times longer than 30 minutes, and a decrease in the mean number of patients seen per non-specialist doctor per shift (Box 5). The daily medical officer roster was unchanged between the control and study periods. However, there was a significant decrease in the mean number of shifts actually worked by both specialists and non-specialists, due to staff vacancies and sick leave. Mean ED occupancy was similar in the two study periods.
After adjusting for the effect of total presentations, specialist and non-specialist staffing, and ED occupancy, the mean total time for discharged patients was 41 minutes longer (95% CI, 29–53; P < 0.001) in 2009 than 2008. Similarly, the DNW rate remained 6.0 percentage points higher (95% CI, 4.8–7.0; P < 0.001) in 2009 than 2008.
Overall, there was a significant increase in the waiting time for all patients, and the waiting time, treatment time and total time for discharged patients after the introduction of Cerner FirstNet in the Nepean Hospital ED compared with the control period.
ED efficiency is affected by many factors both within and outside the ED, including the time taken for inpatient team reviews and “acceptance” of an admission, waiting for a bed on the ward, and completion of clerical and clinical documentation. In a retrospective observational study, it is not possible to directly control for these confounding variables. However, as discharged patients are the group whose care is least affected by these confounding factors, studying this group attempts to isolate actual changes in ED efficiency from the effects of these factors.
For all patients as well as discharged patients, we found that waiting time for those in triage category 2 was significantly shorter in the study period than the control period. Changes to ED workflow in response to poor performance in standard KPIs is a continuous process, and it may be that there was some change in work practice to try to improve the category 2 performance. However, the treatment time and total time for discharged category 2 patients were still significantly longer in the study group. Thus, any benefit from the decreased waiting time in this category was not sustained through to total time in the ED. For triage category 4 patients, treatment time was reduced in the study period, but again this was not borne out in a reduction in total time for these patients.
In contrast, waiting time for all patients in triage category 5 was increased in the study period. However, treatment time and total time for discharged patients in this group were shorter with FirstNet. Category 5 patients are largely seen in the fast-track area of the ED. They do not occupy a bed for the period of their assessment and treatment and thus are generally not affected by ED overcrowding or access block for ward admissions. It could be that there is some benefit from FirstNet for these patients.
The increases in the DNW rate and the proportion of ambulances waiting more than 30 minutes to offload, and the decrease in the mean number of patients seen per non-specialist doctor per shift indicate overall deterioration in the performance of the ED after implementation of FirstNet.17,18 This could potentially increase ED overcrowding because even a small decrease in treatment rate is cumulative, as it causes further increases in the number of patients waiting ahead of new arrivals.17 Worldwide, studies in different centres have found an association between overcrowding and reduced access to care, decreased quality measures and poor outcomes,17 including increasing mortality as overcrowding in EDs increases.19-21
We used ED occupancy as a marker of ED overcrowding, to assess any confounding effect it may have on the efficiency of care for discharged patients. There was no significant difference in ED occupancy in the two study periods, despite the significant change in ED performance. This is likely due to the large increase in total time for discharged patients being offset by the large increase in the proportion of patients who did not wait.
We chose to sample 6 months of presentations to limit the effect of potential confounding variables such as seasonal variations, staffing changes and chance variability in caseload and casemix between the two periods, and we matched the periods of the year for which we collected data. Junior medical officers change terms every 10 weeks to 3 months, and it is possible that staff relatively new to the ED with little experience in using FirstNet could have affected the results. However, the term changes are similar from year to year, and we would expect the findings to have been controlled by examining the same 6-month period of each year. As patient presentations per triage category were similar in the control and study periods, it is unlikely that differences in patient distribution across categories were responsible for any of the findings of this study.
Staffing levels were lower in the study period than the control period and this could have affected ED performance, as fewer doctors may lead to longer waiting times.22 However, the deterioration in total time performance and DNW rate remained after adjusting for the measured confounders. Specialists in our ED do not have a distinct patient load; they have a supervisory role to ensure quality of care and patient flow. For this reason, we only measured the mean number of patients seen per non-specialist doctor per shift.
Another possible confounder is a learning effect of the new EMR system. FirstNet was operational for more than 3 months before the study period, and all medical staff had undergone the prescribed training before its introduction. As medical and nursing staff may rotate through the ED for variable periods of as little as 10 weeks, if the learning effect is greater than 3 months, then they will have spent a large part of their time in the ED learning the system without ever becoming adept at using it.
Although we performed statistical modelling to attempt to control for the effect of the measured confounding variables, this process is limited, as the functional relationships assumed by our modelling may not match the non-linear nature of the data. Also, it is acknowledged that statistically adjusting for confounders has its own limitations and cannot account for unknown and unmeasured factors.
An important post-hoc finding was the poor quality of data for the first 2 months of the study period, especially with respect to patient disposition. For this reason, we could only use 4 months of data from the control and study periods. In addition, there were missing data points for some time measurements.
Since the second half of 2009, when the study data were collected from FirstNet, there have been upgrades to the software. However, these have been largely cosmetic, such as the design of the discharge summary and the appearance of the tracking screen, or addressing issues of patient safety, such as introducing standard checks when ordering radiology investigations and highlighting the patient row when accessing a patient’s record. It is not clear whether these changes would lead to significant improvements in ED performance.
Discharged patients: patients who were assessed by an emergency department (ED) medical officer, including receiving any investigations and treatment, and discharged home from the ED. Excludes patients who did not wait for medical review.
ED occupancy: number of patients in the ED, including patients yet to be seen by a doctor, patients undergoing assessment and treatment in the ED, and admitted patients boarded in the ED.16 Mean daily ED occupancy was calculated based on hourly census calculations (number of patients present in the ED plus number of presentations to the ED minus number departing the ED each hour) averaged over the 24-hour period.
2 Emergency department presentations per triage category, July – December, 2008 and 2009
3 Comparison of waiting time for all patients, September – December, 2008 and 2009
4 Comparison of waiting time, treatment time and total time for discharged patients,* September – December, 2008 and 2009
5 Comparison of did not wait rate, ambulance offload > 30 min, number of patients seen per non-specialist per shift, staffing levels and emergency department (ED) occupancy, 2008 and 2009*
Received 15 March 2012, accepted 31 January 2013
- 1. Wang SJ, Middleton B, Prosser LA, et al. A cost-benefit analysis of electronic medical record in primary care. Am J Med 2003; 114: 397-403.
- 2. Walker JM. Electronic medical records and health care transformation. Health Aff (Millwood) 2005; 24: 1118-1120.
- 3. Koppel R, Localio AR, Cohen A, Strom BL. Neither panacea nor black box: responding to three Journal of Biomedical Informatics papers on computerized physician order entry systems. J Biomed Inform 2005; 38: 267-269.
- 4. Steward M. Electronic medical records: privacy, confidentiality, liability. J Leg Med 2005; 26: 491-506.
- 5. Dinh M, Chu M. Evolution of health information management and information technology in emergency medicine. Emerg Med Australas 2006; 18: 289-294.
- 6. Tierney WM, Miller ME, Overhage JM, McDonald CJ. Physician inpatient order writing on microcomputer workstations. Effects on resource utilization. JAMA 1993; 269: 379-383.
- 7. McDonald CJ, Overhage JM, Tierney WM, et al. The Regenstrief Medical Record System: a quarter century experience. Int J Med Inform 1999; 54: 225-253.
- 8. Gardner RM, Pryor TA, Warner HR. The HELP hospital information system: update 1998. Int J Med Inform 1999; 54: 169-182.
- 9. Pestotnik SL, Classen DC, Evans RS, Burke JP. Implementing antibiotic practice guidelines through computer-assisted decision support: clinical and financial outcomes. Ann Intern Med 1996; 124: 884-890.
- 10. Patrick J. A study of a health enterprise information system. Sydney: School of Information Technologies, University of Sydney, 2011. (Technical report TR673.)
- 11. Li J, Callen J, Georgiou A, Westbrook J. The challenges of using technology in a busy emergency department: a case study of a next generation ED management information system. In: Sintchenko V, Croll P, editors. HIC 2009: Proceedings; Frontiers of Health Informatics – Redefining Healthcare, National Convention Centre, Canberra, 19–21 August 2009. Melbourne: Health Informatics Society of Australia, 2009: 121-125.
- 12. Todd R. Winchester switches off Cerner in ED. EHealth Insider 2012; Sep 10. http://www.ehi.co. uk/news/EHI/8048/winchester-switches-off-cerner-in-ed (accessed Oct 2012).
- 13. London hospital trust loses £7.2m in upgrade debacle. ComputerWorldUK 2008; Nov 6. http://www.computerworlduk.com/news/it-business/11850/london-hospital-trust-loses-72m-in-upgrade-debacle/ (accessed Oct 2012).
- 14. King L. First London NHS care records rollout causes ‘chaos’ – report. ComputerWorldUK 2008; Aug 6. http://www.computerworlduk.com/news/applications/10389/first-london-nhs-care-records-rollout-causes-chaos--report/ (accessed Oct 2012).
- 15. ‘Heroic’ staff can’t hide flaws. Computer Weekly 2007; Apr 10. http://www.computerweekly.com/Articles/2007/04/10/222891/heroic-staff-cant-hide-flaws.htm (accessed Oct 2012).
- 16. Australasian College for Emergency Medicine. ACEM policy on standard terminology. Melbourne: ACEM, 2009. http://www.acem. org.au/media/policies_and_guidelines/P02_-_Standard_Terminology_16.04.09.pdf (accessed Aug 2012).
- 17. Richardson DB, Mountain D. Myths versus facts in emergency department overcrowding and hospital access block. Med J Aust 2009; 190: 369-374. <MJA full text>
- 18. Weiss SJ, Ernst AA, Derlet R, et al. Relationship between the National ED Overcrowding Scale and the number of patients who leave without being seen in an academic ED. Am J Emerg Med 2005; 23: 288-294.
- 19. Chalfin DB, Trzeciak S, Likourezos A, et al; DELAY-ED study group. Impact of delayed transfer of critically ill patients from the emergency department to the intensive care unit. Crit Care Med 2007; 35: 1477-1483.
- 20. Sprivulis PC, Da Silva JA, Jacobs IG, et al. The association between hospital overcrowding and mortality among patients admitted via Western Australian emergency departments. Med J Aust 2006; 184: 208-212. <MJA full text>
- 21. Richardson DB. Increase in patient mortality at 10 days associated with emergency department overcrowding. Med J Aust 2006; 184: 213-216. <MJA full text>
- 22. Jeanmonod R, Brook C, Winther M, et al. Resident productivity as a function of emergency department volume, shift time of day, and cumulative time in the emergency department. Am J Emerg Med 2009; 27: 313-319.
Publication of your online response is subject to the Medical Journal of Australia's editorial discretion. You will be notified by email within five working days should your response be accepted.