MJA
MJA InSight
MJA Careers
Systematic reviews

Computerised reminders and feedback in medication management: a systematic review of randomised controlled trials

John W Bennett and Paul P Glasziou
Med J Aust 2003; 178 (5): 217-222.
Abstract

Objective: To systematically review randomised controlled trials (RCTs) of computer-generated medication reminders or feedback directed to healthcare providers or patients.

Data sources: Extensive computerised and manual literature searches identified 76 English-language reports of RCTs reported before 1 January 2002. Searches were conducted between June 1998 and April 2002.

Study selection: 26 papers making 29 comparisons (two papers reported on multiple interventions) of computer-supported medication management to a control group.

Data extraction: The quality of the RCTs was systematically assessed and scored independently by two reviewers. Rates of compliance with (potential) reminders for the control and intervention groups were extracted.

Data synthesis: Heterogeneity of studies prevented a meta-analysis. Where possible, rates were calculated using the intention-to-treat principle. The comparisons were grouped into five areas. Reminders to providers in outpatient settings: six of 12 comparisons demonstrated positive effects (relative rates [RRs: intervention rates/control rates], 1.0 to 42.0). Provider feedback in outpatient settings: five of seven comparisons showed improved clinician behaviour (RRs, 1.0 to 2.5). Combined reminders and feedback in outpatient settings: the single comparison found no improvement. Reminders to providers in inpatient settings: three of five comparisons showed improvements (RRs, 1.0 to 2.1). Patient-directed reminders: two of four comparisons showed improvements in patient compliance.

Conclusion: Reminders are more effective than feedback in modifying physician behaviour related to medication management. Patient-directed reminders can improve medication adherence.

Drug therapy often does not provide a benefit, for several reasons: many patients are not prescribed the appropriate medicine or dosage; doctors may not properly monitor for efficacy or adverse events; patients may misunderstand even simple instructions;1 and patients taking long term medication often fail to adhere to the prescribed regimen.2

Medication error plays a role in morbidity and death.3,4 In one study, more than half the iatrogenic adverse events at a teaching hospital were related to medication.5 A meta-analysis of drug-related hospital admissions found 5.1% (95% CI, 4.4%–5.8%) of all hospital admissions were related to adverse drug reactions.6

Computerised decision support systems (DSSs) can help reduce these problems. Reminder systems have improved both preventive practices7,8 and compliance with clinical guidelines.9 Feedback from peer comparison results in a small effect on rates of clinical procedures (eg, prescribing rates, laboratory tests).10

We developed a categorisation of DSSs based on reviews of randomised controlled trials (RCTs) of their clinical effectiveness.11,12 Reminders and feedback differ in terms of their timing, information content and intended scope of effect. Reminders are delivered at the time of or before decision-making and are specific to a single patient; they prompt healthcare providers or patients to act (eg, order tests, pick up a repeat prescription). Feedback aggregates information from multiple patients (and sometimes providers) with the intention of altering future decisions.

We aimed to systematically review the benefits of computerised systems that support medication management through reminders or feedback to healthcare providers, or reminders to patients.

Methods
Identification of studies

One of us (J W B) searched the complete MEDLINE database to identify all RCTs of computerised DSSs published in English from 1 January 1966 to 31 December 2001. Searches were conducted between June 1998 and April 2002 and used combinations of medical subject headings (MeSH) terms (randomized controlled trial, reminder systems, drug therapy, medical informatics [exploded]) and free text (random*, medication* OR drug*, adheren* OR complian*). The Cochrane Controlled Trials Register (1996–2001), CINAHL (1982–2001), Current Contents (1997–2001) and COMPENDEX (1987–2001) were also searched using similar techniques. The reference lists of retrieved articles were scanned for references to further trials. The contents pages of the major informatics conference proceedings MEDINFO (1980–1992) and SCAMC/AMIA Annual Fall Symposium (1984–1991) were searched manually.

Study selection

When an article seemed relevant based on the title and abstract, the full article was reviewed by both of us. We included any system that used computers to assist in identifying patients and generating reminders or feedback. We also included studies that tried to influence medication use but measured non-clinical outcomes (eg, rate of generic prescribing or costs). Such studies provide evidence of the capacity of DSSs to alter the behaviour of healthcare providers. We excluded systems that calculate drug doses, for which there is a Cochrane review.13

Analysis

Studies were critically appraised using an instrument developed specifically for evaluating the quality of RCTs of reminder or feedback DSSs. Each RCT was assessed on randomisation method (3 points), parity of baseline data (2), objectivity of primary outcome measure (4), loss to follow-up (4), clarity of inclusion criteria (1), unit of allocation (1), ease of implementation of reminder system (1), and statistical analysis (1), to give a score out of 17. Although there is little empirical evidence to support our choices,14 the scale contains the elements present in many similar instruments.15,16

Data extraction and synthesis

We independently assessed quality and eligibility, and resolved disagreements by consensus. Where possible, data were extracted using an intention-to-treat principle. We report the rate of compliance with (potential) reminders for the control and intervention groups, or, where this is not possible, we indicate how the values were derived.

The studies and their outcome measures were too heterogeneous (eg, system, outcome measures, settings, participants) for meta-analysis. We consider them in five categories (numbers of comparisons in parentheses): reminders to providers in outpatient settings (12);17-28 feedback to providers in outpatient settings (7);20,29-34 combined reminders and feedback to providers in outpatient settings (1);20 reminders to providers in inpatient settings (5);35-39 and reminders to patients (4).40-42

Results

From 76 potentially relevant trials, we agreed on including 26. The quality scores of these 26 RCTs ranged from 9 to 17 (mean, 13.6; κ = 0.54, indicating fair agreement between raters' scores). The methodological quality was high, with 25 (96%) either describing a valid method of randomisation or stating that random allocation was used. Eighteen studies (69%) presented adequate baseline data or made appropriate adjustments. Twenty-four studies (92%) were judged to have objective outcome measures or appropriately blinded subjective outcome measures. Follow-up was adequate (greater than 95%) in 18 studies (69%). The inclusion criteria were clear in 23 of the studies (88%). The unit of allocation was judged to be appropriate for the type of intervention in 24 (92%). The reminder system was considered to be easy to implement elsewhere in 18 (69%). Statistical analysis was judged appropriate in 17 studies (65%). The flaw in the remainder was that allocation was by healthcare provider or group (eg, general practice, resident team), but the analysis was by patient, which will underestimate the standard error,, although the effect size is unbiased.

Box 1 to Box 5 present the extracted data. Two studies20,41 provided more than one comparison. A L'Abbé plot43 (Box 6) compares effect sizes for RCTs containing sufficient data. This shows significant heterogeneity of studies.

Twenty-two trials evaluated the effects of a medication management DSS on either provider (19) or patient adherence to medication (3).40-42 Three studies assessed cost as their main outcome measure.30,31,35 No studies were able to demonstrate an effect on patient outcomes.

Reminders to providers in outpatient settings

DSSs generally improved medication management in this setting, with relative rates from 1.0 to 42 (Box 1). Poor user interface24 and requiring healthcare providers to enter redundant data22,25 were cited as reasons for poor system utilisation.

Provider feedback systems in outpatient settings

Physician feedback systems in outpatient settings (Box 2) generally had smaller effects on clinician behaviour than reminder systems. Relative rates were from 1.0 to 2.5. One group29 influenced rates of median weighted generic prescribing in a family medicine clinic by 35% through monthly feedback of prescription patterns. There were mixed results from feedback aimed at containing prescribing costs. One group decreased the average charge per prescription,30 but another group was unable to reduce prescribing costs.31 Feedback was ineffective in reducing polypharmacy,32 in increasing the use of aspirin in coronary artery disease,33 and in increasing the use of antidepressants in depression.34

Combined reminders and feedback in outpatient settings

Only one study directly compared feedback and reminders (Box 3).20 Feedback and reminders (10.2%) compared with reminders alone (10.4%) did not increase compliance with preventive medication recommendations. This is consistent with results from the previous two sections showing that reminders are more effective than feedback.

Reminders to providers in inpatient settings

There were five studies in this category (Box 4).35-39 In one, medical staff using computerised workstations to order investigations and medications were offered reminders aimed at decreasing costs; this resulted in a 15% reduction in drug costs per admission.35 Extending upon their work in outpatients, Overhage et al38 more than doubled the ordering of additional tests or treatment to monitor or ameliorate effects of the primary treatments or tests. In the final study in this category, provision of electronic guidelines decreased vancomycin prescribing.39

Patient-directed reminders

The three RCTs of patient-directed reminders in outpatient settings used computer-generated reminders (timetable, mailed and phone) to improve medication compliance (Box 5). Two studies40,41 conducted entirely in outpatient settings used compliance with refills (ie, repeat prescriptions) as their outcome measurement. Overall, the effect of these interventions was disappointing. One study group showed a 7% increase (not statistically significant) in refill compliance.41 The third study commenced while the patients were medical inpatients in a British district hospital. It demonstrated a 25% increase in the mean compliance scores (measured by pill counts) for patients in the timetable group, but the follow-up period was only 10 days.42

Discussion

Computerised DSSs providing reminders and feedback to healthcare providers and patients can make modest improvements in medication management. They have successfully changed the class of medication prescribed, increased generic prescribing, improved activities related to medication management (eg, diagnostic testing), and enhanced patient adherence to medication regimens. It appears that reminders are more effective than feedback. However, other factors, such as the study population, the nature of "usual care", or the outcome measures (eg, compliance measures), may explain these differences. Mugford et al44 found information was most effective if presented close to the time of decision making. Our results support this, as, compared with feedback, reminders are presented closer to the time of decision-making.

The current evidence should encourage wider use of DSS for medication management, based on careful consideration of local factors. One academic medical centre produced eight studies with some of the larger effect sizes.17-20,35-38 Whether commercial systems in non-academic settings can produce the same magnitude of effect is an open question. US policy makers are calling for mandatory use of computerised physician order entry as a means to increasing use of DSSs for medication management.45

Four studies17-20 of reminders in outpatient settings were conducted by the same group using a sophisticated electronic health record (EHR) that has operated since 1973.17 Very few EHRs have such richness of patient data and completeness of history. Clinician-authored rules detected critical changes in clinical variables (eg, serum potassium for patients on diuretics) that might need correction and would trigger a recommendation from the DSS.17-20 A DSS coupled to an EHR removes reliance upon duplicate data entry by healthcare providers,22,25 but there are many barriers to the implementation of EHRs.46

Some authors20,26 commented on how little improvement was achieved, despite the healthcare providers agreeing with the importance of the actions requested by the reminders. Further research should focus on what features of DSSs — such as multiple rather than single options, "help" and explanation functions, or speed — might enhance the effects seen. For example, by requiring a response to computer-generated reminders Litzelman et al47 improved compliance with preventive care protocols by internal medicine residents.

Conclusion

Using computers to improve medication use is worthwhile, but care is needed in choosing the most appropriate means of delivering messages. Reminders and feedback can improve various behaviours related to medication management. However, their implementation requires consideration of factors that are likely to bring success.

1: Provider reminders in outpatient settings

First author (year; country)

Setting; participants

Nature of reminders

Control complied/ no. of potential reminders (rate)

Intervention complied/no. of actual reminders (rate)

Relative rate


McDonald17
(1976; US)

Diabetes clinic (primary care); 257 patients

Follow and assess medication use to assure investigations were undertaken at proper intervals and that clinicians reacted to results

54/470 (12%)

175/500 (35%)

3.0

McDonald18
(1980; US)

General medicine clinic; 26 physicians, 5 nurse practitioners

Reminders regarding "usage and follow-up of medications"

236/1158 (20%)

956/2533 (38%)

1.9

McDonald19
(1984; US)

General medicine clinic; 126 physicians, 4 nurse clinicians

Monitoring of serum K+ or K+ supplements for diuretics; digitalis trial if evidence of CCF on chest x-ray or echocardiogram; monitoring of liver enzyme levels for hepatotoxic drugs; antacid prophylaxis for aspirin, NSAIDs or corticosteroids

(54%)

(67%)

1.3

We were unable to extract exact values. We averaged the percentage responses of the four medication-related actions, assuming even distribution of values.

Tierney20
(1986; US)

General medicine clinic; 135 internal medicine house staff

Antacids, antidepressants, aspirin, β-blockers, calcium supplements, digitalis, long-acting nitrates, metronidazole. Specific reminders at patient visit or monthly feedback reports

(8%)

(10%)

1.3

2×2 factorial design of feedback and reminders. House staff were their own controls.

Rossi21
(1997; US)

General medicine clinic; 59 physicians, 12 nurse practitioners

Reminder to switch from calcium-channel blockers to other antihypertensive agents

1/373 (0.3%)

39/346 (11%)

42.0

Hetlevik22
(1999; Norway)

29 Health centres; 53 GPs, 2239 patients

Computerised clinical guidelines for hypertension

No usable data for medications, and no effect on medication management.

1.0

Demakis23
(2000; US)

12 Ambulatory care clinics; 275 resident physicians

Standards-of-care reminders for atrial fibrillation, myocardial infarction and gastrointestinal bleeding (+ other preventive care reminders). Computerised and paper reminders

621/1001 (62%)

657/1059 (62%)

1.0

Montgomery24
(2000; UK)

27 General practices; 74 GPs; 11 practice nurses

Computer-generated cardiovascular risk assessment

No usable data, and no difference in overall prescribing rates between groups.

1.0

McCowan25
(2001; UK)

17 General practices; 477 patients

Computerised asthma management clinical guidelines

No usable data for medications, and no difference in maintenance prescribing.

1.0

Christakis26
(2001; US)

Paediatric clinic; 36 physicians, 2 nurse practitioners

Point-of-care evidence-based messages on otitis media antibiotic prescribing

(10%)

(44%)

4.2

Used a primary study outcome of change in frequency of antibiotics prescribed for <10 days (before v after).

Frances27
(2001; US)

3 General medicine clinics; 66 physicians

Combination of computer-generated and written reminders for aspirin, β-blockers and cholesterol lowering agents in coronary artery disease

(30%)

(29%)

1.0

These are averages for the 3 medication groups.

Rollman28 (2001; US)

Primary medical care practice; 227 patients; 16 internists

Computer-generated patient-specific advisory messages on management of depression

No usable data and no difference in overall prescribing rates between groups.

1.0

2: Provider feedback

First author (year; country)

Setting; participants

Nature of feedback

Control complied/ no. of potential reminders (rate)

Intervention complied/no. of actual reminders (rate)

Relative rate


Gehlbach29
(1984; US)

Family medicine residency; 32 physicians

Monthly feedback reports regarding brand versus generic prescribing

(23%)

(58%)

2.5

The article contained no raw data to derive rates. Reported "median weighted rates of generic prescribing".

Hershey30 (1986; US)

General medicine clinic; 48 residents

Feedback regarding average charges per prescription

US$8.79

US$8.22

0.9

The article contained no raw data to derive rates. We used their "resident's average charge per prescription".

Tierney20
(1986; US)

General medicine clinic; 135 internal medicine house staff

Antacids, antidepressants, aspirin, β-blockers, calcium supplements, digitalis, long-acting nitrates, metronidazole. Specific reminders at patient visit or monthly feedback reports

(8%)

(9%)

1.2

2×2 factorial design of feedback and reminders. House staff were their own controls.

Steele31
(1989; US)

General medicine clinic; 20 physicians (residents and fellows)

Peer-comparison feedback of prescribing costs (+ another arm with educational intervention)

No usable data, and no difference in prescribing costs between peer-comparison feedback and controls.

1.0

Meyer32
(1991; US)

General medicine clinic; 292 patients

Two types of feedback: letter identifying patients on 10 or more long term medications; similar letter including personalised review of medications

No usable data, and no difference reported between either intervention group and control in reducing polypharmacy.

1.0

McCartney33
(1997; UK)

28 General practices

Baseline prescribing of prophylactic aspirin in coronary artery disease. Facilitated support through medical audit advisory groups

610/1220 (50%)

1004/1725 (58%)

1.2

Simon34
(2000; US)

5 Primary medical care clinics; 613 patients

Two detailed reports about depression management and treatment recommendations

(18%)

(21%)

1.2

Data for adequate antidepressant use at least 90 days after initial prescription.

3: Combined reminders and feedback

First author (year; country)

Setting; participants

Nature of reminders and feedback

Control complied/ no. of potential reminders (rate)

Intervention complied/no. of actual reminders (rate)

Relative rate


Tierney20
(1986; US)

General medicine clinic; 135 internal medicine house staff

Antacids, antidepressants, aspirin, β-blockers, calcium supplements, digitalis, long-acting nitrates, metronidazole. Specific reminders at patient visit or monthly feedback reports

(10%)

(10%)

1.0

2×2 factorial design of feedback and reminders. House staff were their own controls.

4: Inpatient reminders

First author (year; country)

Setting; participants

Nature of reminders

Control complied/ no. of potential reminders (rate)

Intervention complied/no. of actual reminders (rate)

Relative rate


Tierney35
(1993; US)

General medicine; 68 resident teams (5219 patients)

All inpatient orders (including medications) on computer, linked to a comprehensive electronic medical record

US$1181

US$1001

0.85

The article contained no raw data we could use to derive rates. We used "total drug charges per admission".

Overhage36
(1996; US)

General medicine; 68 physicians (1622 patients)

Aspirin, oestrogen, calcium, ACE inhibitor, heparin prophylaxis, β-blocker, and 18 other preventive care measures

71/794 (9%)

63/811 (8%)

0.9

Smith37
(1996; US)

General medicine; 12 medicine ward teams (348 patients)

Computer-generated list of all active outpatient prescriptions at discharge and the capacity to alter discharge prescriptions.

(3%)

(3%)

1.0

The article contained no raw data to derive rates. We used their "change in active prescriptions from admission to discharge".

1.0

Overhage38 (1997; US)

General medicine service; 89 physicians (2181 patients)

Reminders of corollary orders to prevent errors of omission. Computer suggested orders to detect or ameliorate adverse reactions to the trigger orders (76 drugs)

(22%)

(46%)

2.1

We were unable to extract exact values. We used percentages as reported in the article.

Shojania39 (1998; US)

Tertiary care teaching hospital; 396 physicians (1798 patients)

Electronic guideline for vancomycin use

17

11

0.7

The article contained no raw data to derive rates We used their "Total numbers of vancomycin orders per prescriber".

5: Patient-directed reminders

First author (year; country)

Setting; participants

Nature of reminders

Control complied/ no. of potential reminders (rate)

Intervention complied/no. of actual reminders (rate)

Relative rate


Baird40 (1984; US)

Urban outpatient pharmacy; 324 patients

Mailed reminders for prescription refills

32/163 (20%)

29/161 (18%)

0.9

Simkins41 (1986; US)

Primary care and speciality clinics; 207 patients

Postcard refill reminders for cardiovascular medications due in 2 days

(58%)

(65%)

1.1

The article contained no raw data. We used their "mean compliant events per group".

Simkins41 (1986; US)

Primary care and speciality clinics; 208 patients

Telephone refill reminders for cardiovascular medications due in 2 days

(58%)

(42%)

0.7

We used the "mean compliant events per group", but with an intention-to-treat analysis, assuming the worst case. The authors' original analysis had excluded 35 of 104 patients who could not be contacted by phone.

Raynor42 (1993; UK)

3 Medical wards, district hospital; 197 patients

Medication timetable for patients on discharge

60/96 (62%)

83/95 (87%)

1.4

6: L'Abbé plot of the effect of computerised reminders or feedback in 13 randomised controlled trials

* Three subsets: reminders, feedback, and combined. † Two subsets: phone and mail patient reminders.

Received 
29 May 2002
accepted 
13 Dec 2002
John W Bennett, MB BS FRACGP, Adjunct Senior Lecturer in Medical Informatics1
Paul P Glasziou, MB BS PhD, Professor of Evidence-based Medicine2
1 University Health Service, University of Queensland, Brisbane, QLD.
2 School of Population Health, University of Queensland Medical School, Herston, QLD.
Acknowledgements: 

We thank the anonymous referees for their most thoughtful and constructive comments.

Competing Interests: 

None identified.

Reference Text: 
Hanchak NA, Patel MB, Berlin JA, Strom BL. Patient misunderstanding of dosing instructions. J Gen Intern Med 1996; 11: 325-328.
Reference Order: 
1
PubMed ID: 
8803737
Reference Text: 
Rich M, Gray D, Beckham V, et al. Effect of a multidisciplinary intervention on medication compliance in elderly patients with congestive heart failure. Am J Med 1996; 101: 270-276.
Reference Order: 
2
PubMed ID: 
8873488
Reference Text: 
Leape LL. Error in medicine. JAMA 1994; 272: 1851-1857.
Reference Order: 
3
PubMed ID: 
7503827
Reference Text: 
Kohn LT, Corrigan JM, Donaldson MS. To err is human: building a safer health system. Washington, DC: National Academy Press, 1999.
Reference Order: 
4
PubMed ID: 
Reference Text: 
Steel K, Gertman PM, Crescenzi C, Anderson J. Iatrogenic illness on a general medical service at a university hospital. N Engl J Med 1981; 304: 638-642.
Reference Order: 
5
PubMed ID: 
7453741
Reference Text: 
Einarson TR. Drug-related hospital admissions. Ann Pharmacother 1993; 27: 832-840.
Reference Order: 
6
PubMed ID: 
8364259
Reference Text: 
Austin SM, Balas EA, Mitchell JA, Ewigman BG. Effect of physician reminders on preventive care: meta-analysis of randomized clinical trials. In: Ozbolt JG, editor. Proceedings for 18th Annual Symposium on Computer Applications in Medical Care; 1994 November 5–9; Washington, DC. Philadelphia: Hanley & Belfus, 1994: 121-124.
Reference Order: 
7
PubMed ID: 
Reference Text: 
Shea S, DuMouchel W, Bahsmonde L. A meta-analysis of 16 randomized controlled trials to evaluate computer-based clinical reminder systems for preventive care in the ambulatory setting. J Am Med Inform Assoc 1996; 3: 399-409.
Reference Order: 
8
PubMed ID: 
8930856
Reference Text: 
Shiffman RN, Liaw Y, Brandt CA, Corb GJ. Computer-based guideline implementation systems: a systematic review of functionality and effectiveness. J Am Med Inform Assoc 1999; 6: 104-114.
Reference Order: 
9
PubMed ID: 
10094063
Reference Text: 
Balas EA, Boren SA, Brown GD, et al. Effect of physician profiling on utilization. Meta-analysis of randomized clinical trials. J Gen Intern Med 1996; 11: 584-590.
Reference Order: 
10
PubMed ID: 
8945689
Reference Text: 
Bennett JW, Glasziou PP. A review of the usefulness of computerised systems in clinical medicine. In: Royal Australian College of General Practitioners 7th Computer Conference; 1993 June 3–5. Melbourne: RACGP, 1993; 127-130.
Reference Order: 
11
PubMed ID: 
Reference Text: 
Bennett JW, Glasziou PP. A review of the usefulness of advice and therapy systems in clinical medicine. In: Hovenga, EJS, Whymark GK, editors. Proceedings of the Inaugural Health Informatics Conference; Brisbane, Australia; 1993 August 2–3. Melbourne: HISA, 1993; 228-233.
Reference Order: 
12
PubMed ID: 
Reference Text: 
Walton RT, Harvey EL, Dovey S, Freemantle N. Computerised advice on drug dosage to improve prescribing practice. In: Bero L, Grilli R, Grimshaw J, Oxman A, editors. Cochrane Collaboration on Effective Professional Practice Module of The Cochrane Database of Systematic Reviews [updated March 2001]. Available in The Cochrane Library. The Cochrane Collaboration; Issue 2. Oxford: Update Software, 2001.
Reference Order: 
13
PubMed ID: 
Reference Text: 
Moher D, Jadad AR, Tugwell P. Assessing the quality of randomized controlled trials. Current issues and future directions. Int J Technol Assess Health Care 1996; 12: 195-208.
Reference Order: 
14
PubMed ID: 
8707495
Reference Text: 
Guyatt GH, Sackett DL, Cook DJ, for the Evidence-Based Medicine Working Group. Users' guides to the medical literature. II. How to use an article about therapy or prevention. A. Are the results of the study valid? JAMA 1993; 270: 2598-2601.
Reference Order: 
15
PubMed ID: 
8230645
Reference Text: 
Sackett DL, Richardson WS, Rosenberg W, Haynes RB. Evidence-based medicine: how to practice and teach EBM. 1st ed. New York: Churchill Livingstone, 1997.
Reference Order: 
16
PubMed ID: 
Reference Text: 
McDonald CJ. Use of computer to detect and respond to clinical events: its effect on clinician behavior. Ann Intern Med 1976; 84: 162-167.
Reference Order: 
17
PubMed ID: 
1252043
Reference Text: 
McDonald CJ, Wilson GA, McCabe GP. Physician response to computer reminders. JAMA 1980; 244: 1579-1581.
Reference Order: 
18
PubMed ID: 
7420656
Reference Text: 
McDonald CJ, Hui SL, Smith DM, et al. Reminders to physicians from an introspective computer medical record. A two-year randomized trial. Ann Intern Med 1984; 100: 130-138.
Reference Order: 
19
PubMed ID: 
6691639
Reference Text: 
Tierney WM, Hui SL, McDonald CJ. Delayed feedback of physician performance versus immediate reminders to perform preventive care. Effects on physician compliance. Med Care 1986; 24: 659-666.
Reference Order: 
20
PubMed ID: 
3736141
Reference Text: 
Rossi RA, Every NR. A computerized intervention to decrease the use of calcium channel blockers in hypertension. J Gen Intern Med 1997; 12: 672-678.
Reference Order: 
21
PubMed ID: 
9383135
Reference Text: 
Hetlevik I, Holmen J, Kruger O. Implementing clinical guidelines in the treatment of hypertension in general practice. Evaluation of patient outcome related to implementation of a computer-based clinical decision support system. Scand J Prim Health Care 1999; 17: 35-40.
Reference Order: 
22
PubMed ID: 
10229991
Reference Text: 
Demakis JG, Beauchamp C, Cull WL, et al. Improving residents' compliance with standards of ambulatory care: results from the VA Cooperative Study on Computerized Reminders. JAMA 2000; 284: 1411-1416.
Reference Order: 
23
PubMed ID: 
10989404
Reference Text: 
Montgomery AA, Fahey T, Peters TJ, et al. Evaluation of computer based clinical decision support system and risk chart of management of hypertension in primary care: randomised controlled trial. BMJ 2000; 320: 686-690.
Reference Order: 
24
PubMed ID: 
10710578
Reference Text: 
McCowan C, Neville RG, Ricketts IW, et al. Lessons from a randomized controlled trial designed to evaluate computer decision support software to improve the management of asthma. Med Inform Internet Med 2001; 26: 191-201.
Reference Order: 
25
PubMed ID: 
11706929
Reference Text: 
Christakis DA, Zimmerman FJ, Wright JA, et al. A randomized controlled trial of point-of-care evidence to improve the antibiotic prescribing practices for otitis media in children. Pediatrics 2001; 107: E15-E18.
Reference Order: 
26
PubMed ID: 
11158489
Reference Text: 
Frances CD, Alperin P, Adler JS, Grady D. Does a fixed physician reminder system improve the care of patients with coronary artery disease? A randomized controlled trial. West J Med 2001; 175: 165-166.
Reference Order: 
27
PubMed ID: 
11527840
Reference Text: 
Rollman BL, Hanusa BH, Gilbert T, et al. The electronic medical record. Arch Intern Med 2001; 161: 189-197.
Reference Order: 
28
PubMed ID: 
11176732
Reference Text: 
Gehlbach SH, Wilkinson WE, Hammond WE, et al. Improving drug prescribing in a primary care practice. Med Care 1984; 22: 193-201.
Reference Order: 
29
PubMed ID: 
6700282
Reference Text: 
Hershey CO, Porter DK, Breslau D, Cohen DI. Influence of simple computerized feedback on prescription charges in an ambulatory clinic. A randomized clinical trial. Med Care 1986; 24: 472-481.
Reference Order: 
30
PubMed ID: 
3520177
Reference Text: 
Steele MA, Bess DT, Franse VL, Graber SE. Cost effectiveness of two interventions for reducing outpatient prescribing costs. Drug Intell Clin Pharm 1989; 23: 497-500.
Reference Order: 
31
PubMed ID: 
Reference Text: 
Meyer TJ, Van Kooten D, Marsh S, Prochazka AV. Reduction of polypharmacy by feedback to clinicians. J Gen Intern Med 1991; 6: 133-135.
Reference Order: 
32
PubMed ID: 
2023020
Reference Text: 
McCartney P, Macdowall W, Thorogood M. A randomised controlled trial of feedback to general practitioners of their prophylactic aspirin prescribing. BMJ 1997; 315: 35-36.
Reference Order: 
33
PubMed ID: 
9233326
Reference Text: 
Simon GE, VonKorff M, Rutter C, Wagner E. Randomised trial of monitoring, feedback, and management of care by telephone to improve treatment of depression in primary care. BMJ 2000; 320: 550-554.
Reference Order: 
34
PubMed ID: 
10688563
Reference Text: 
Tierney WM, Miller ME, Overhage JM, McDonald CJ. Physician inpatient order writing on microcomputer workstations. Effects on resource utilization. JAMA 1993; 269: 379-383.
Reference Order: 
35
PubMed ID: 
8418345
Reference Text: 
Overhage JM, Tierney WM, McDonald CJ. Computer reminders to implement preventive care guidelines for hospitalized patients. Arch Intern Med 1996; 156: 1551-1556.
Reference Order: 
36
PubMed ID: 
8687263
Reference Text: 
Smith DM, Cox MR, Brizendine EJ, et al. An intervention on discharge polypharmacy. J Am Geriatr Soc 1996; 44: 416-419.
Reference Order: 
37
PubMed ID: 
8636588
Reference Text: 
Overhage JM, Tierney WM, Zhou XH, McDonald CJ. A randomized trial of "corollary orders" to prevent errors of omission. J Am Med Inform Assoc 1997; 4: 364-375.
Reference Order: 
38
PubMed ID: 
9292842
Reference Text: 
Shojania KG, Yokoe D, Platt R, et al. Reducing vancomycin use utilizing a computer guideline: results of a randomized controlled trial. J Am Med Inform Assoc 1998; 5: 554-562.
Reference Order: 
39
PubMed ID: 
9824802
Reference Text: 
Baird TK, Broekemeier RL, Anderson MW. Effectiveness of a computer-supported refill reminder system. Am J Hosp Pharm 1984; 41: 2395-2397.
Reference Order: 
40
PubMed ID: 
6507445
Reference Text: 
Simkins CV, Wenzioff NJ. Evaluation of a computerized reminder system in the enhancement of patient medication refill compliance. Drug Intell Clin Pharm 1986; 20: 799-802.
Reference Order: 
41
PubMed ID: 
3769774
Reference Text: 
Raynor DK, Booth TG, Blenkinsopp A. Effects of computer generated reminder charts on patients' compliance with drug regimens. BMJ 1993; 306: 1158-1161.
Reference Order: 
42
PubMed ID: 
8499815
Reference Text: 
L'Abbé KA, Detsky AS, O'Rourke K. Meta-analysis in clinical research. Ann Intern Med 1987; 107: 224-233.
Reference Order: 
43
PubMed ID: 
3300460
Reference Text: 
Mugford M, Banfield P, O'Hanlon M. Effects of feedback of information on clinical practice: a review. BMJ 1991; 303: 398-402.
Reference Order: 
44
PubMed ID: 
1912809
Reference Text: 
Overhage JM, Middleton B, Miller RA, et al. Does national regulatory mandate of provider order entry portend greater benefit than risk for health care delivery? J Am Med Inform Assoc 2002; 9: 199-208.
Reference Order: 
45
PubMed ID: 
11971880
Reference Text: 
McDonald CJ. The barriers to electronic medical record systems and how to overcome them. J Am Med Inform Assoc 1997; 4: 213-221.
Reference Order: 
46
PubMed ID: 
9147340
Reference Text: 
Litzelman DK, Dittus RS, Miller ME, Tierney WM. Requiring physicians to respond to computerized reminders improves their compliance with preventive care protocols. J Gen Intern Med 1993; 8: 311-317.
Reference Order: 
47
PubMed ID: 
8320575