MJA
MJA InSight
MJA Careers
Supplement

Practitioner-supported delivery of internet-based cognitive behaviour therapy: evaluation of the feasibility of conducting a cluster randomised trial

Ian B Hickie, Tracey A Davenport, Georgina M Luscombe, Michael Moore, Kathleen M Griffiths and Helen Christensen
Med J Aust 2010; 192 (11): 31.
Abstract

Objective: To determine the feasibility of conducting a cluster randomised trial in Australia of the effectiveness of general practitioner-supported delivery of internet-based cognitive behaviour therapy (CBT) and enhanced psychological care.

Design, setting and participants: Cluster randomised trial involving patients attending general practices in Australia. Participating practices were randomly allocated to interventions. The study was conducted between January 2004 and January 2007.

Interventions: Enhanced GP care was delivered by doctors who had completed specific mental health training; the experimental condition consisted of enhanced GP care plus MoodGYM, an internet-based CBT intervention.

Main outcome measures: Demographic and behavioural characteristics of patients, and demographic and practice characteristics of GPs; time to resolution of psychological symptoms for patients involved in the longitudinal phase of the trial.

Results: 1571 patients attending 90 GPs from 84 general practices were identified as potentially suitable for recruitment. These patients had a mean age of 35 years, 76% were female, 84% had access to the internet for personal use, and 22% reported high or very high levels of psychological distress on the Kessler Psychological Distress Scale. The 90 GPs had a mean age of 49 years, 53% were female and 25% had completed formal mental health training. Of the 1571 screened patients, 340 reported high levels of psychological distress, but only 140 of these could be further assessed for eligibility in the trial. Of these 140, 83 patients with depression (attending 10 GPs in eight general practices) proceeded to randomisation. For these patients, the experimental intervention (enhanced GP care plus MoodGYM) tended to result in prompt and more sustained resolution of depressive symptoms.

Conclusion: Our capacity to conduct a definitive trial was limited by available resources. Preliminary data suggest that primary care patients with depression may derive additional benefits from an internet-based CBT program delivered in conjunction with enhanced psychological care from GPs.

Although major depression is the leading cause of health-related disability in Australia, access to appropriate forms of psychological treatment remains low.1 Cost-effective mental health interventions can now be delivered via the internet to broad sectors of society, including people who have limited access to clinical services due to geographic, economic or social barriers.2,3 MoodGYM, an Australian innovation, is an internet-based cognitive behaviour therapy (CBT) intervention designed to treat and prevent depression and anxiety.4 MoodGYM has been shown to reduce symptoms of depression and anxiety in people in the community and in school settings.5-7

We have previously demonstrated the lack of access to skilled psychological treatments in primary care settings.8 There is an urgent need to evaluate the potential effectiveness of internet-based therapies in clinical settings, including primary care. About 85% of adult Australians visit a general practitioner in any year, and GPs are most likely to be the first point of professional contact for most individuals with mental health problems.9 Although there is evidence that use of the internet to treat mental health problems is acceptable to both patients and doctors,10 internet-based programs have not been widely used to deliver mental health interventions in conjunction with traditional general practice. Since the introduction in 2001 of the federal government’s Better Outcomes in Mental Health Care (BOiMHC) initiative, greater emphasis has been placed on delivery of more sophisticated psychological assessment and active management by Australian GPs.11 Although the introduction of Medicare-based support for psychological services in late 2006 also increased access to professionally delivered mental health services, most people still rely largely on their GP to manage their psychological difficulties.12

We sought to evaluate the feasibility of conducting a cluster randomised trial of the effectiveness of practitioner-supported delivery of MoodGYM as an adjunct to enhanced psychological care delivered by GPs.

Methods

We recruited patients attending GPs for common forms of psychological distress to a cluster randomised trial comparing the effects of two forms of enhanced care for depressive syndromes. The enhanced GP care was delivered by doctors who had completed mental health training for managing depressive disorders in patients presenting to primary care (Level 1 BOiMHC training). The experimental intervention consisted of enhanced GP care plus practitioner-supported use of an effective internet-based treatment system (MoodGYM). The study was conducted between January 2004 and January 2007.

Level 1 BOiMHC training and trial familiarisation

GPs were trained using the introductory component of the SPHERE training program (run by Educational Health Solutions and sponsored by Pfizer Pharmaceuticals), coordinated through the Central Sydney Division of General Practice. The introductory component (“Depression and anxiety: an introductory training program for general practitioners”) provides GPs with the knowledge and skills required to effectively identify, treat and manage depression, anxiety and somatic distress in their practice.13,14 This highly interactive and discussion-based training program includes three seminars that focus on: psychological distress in general practice; diagnosis of specific psychological disorders; and management of depression and anxiety. More than 9000 GPs have had access to some aspect of the SPHERE training program over the past decade.

This training was followed by a 2-hour trial familiarisation meeting that covered trial design, contents of enhanced GP care, use of MoodGYM, patient management during the trial including the use of antidepressants, and methods for conducting the research trial within general practice. Each practice was asked to recruit 15 patients.

Cluster randomisation of general practices

General practices were identified for participation in the study through the Central Sydney Division of General Practice, the national SPHERE database and advertising in relevant trade journals. Of 84 general practices (90 GPs) who participated in the initial assessment, 34 were randomly assigned to offer enhanced GP care plus MoodGYM or enhanced GP care alone. The cluster randomised trial design is shown in Box 1.

Patient screening

A research nurse was present within each practice on specific days and offered screening questionnaires to all patients before their appointment with the GP. In some practices, the practice manager was able to assist. Only those patients who scored positive for psychological distress (using an arbitrary cut-off score of at least 20 on the Kessler Psychological Distress Scale [K10],15 to capture all patients with high to very high levels of distress) and self-reported that they met other trial criteria were offered a follow-up appointment, and had their records flagged to alert the GP to their potential participation in the trial (Box 2). GPs then made a final determination of whether these patients were eligible for inclusion. The trial criteria included: age between 16 and 50 years (the upper age limit was used to eliminate potential confounding from patients presenting with age-related cognitive and behavioural decline); English as a first language; depressive symptoms not due to grief, and duration of less than 2 years; access to the internet; not currently enrolled in any other research project or receiving psychological or pharmacological care; and not having a major medical illness, psychomotor agitation, neurological impairment, head injury, lifetime alcohol or drug dependence or misuse, or a mixed depressive episode.

Interventions

Once patients were recruited to the trial, they received either enhanced GP care plus MoodGYM or enhanced GP care alone (Box 2). Both interventions were delivered over about 8 weeks, with baseline and end-of-treatment assessments conducted by the research nurse. Follow-ups were also undertaken 6 months and 12 months after the intervention.

Data collection

Data were collected from all GPs who initially enrolled in the trial and all patients who were screened. At enrolment, GPs reported individual demographics and practice characteristics. Patients who were screened for entry to the trial completed a survey that recorded demographic information and measured their current level of psychological distress with the 12-item Somatic and Psychological Health Report (SPHERE-12)16 and the K10. The K10 cut-off scores used in the 2000 Health and Wellbeing Survey and the 2001 National Health Survey to estimate the prevalence of levels of psychological distress were applied.17 Patients’ self-reported disability was also measured with the Brief Disability Questionnaire (BDQ).18

For patients subsequently recruited into the trial, the SPHERE-12, K10 and BDQ measures were repeated before, immediately after, and 6 and 12 months after treatment.

Statistical analysis

GP and patient demographics and general practice characteristics were analysed using simple descriptive statistics, χ2 analysis and independent samples t tests. SPHERE-12 and K10 scores over the course of the study were examined using Kaplan–Meier curves, which estimated the time to resolution of psychological symptoms for each intervention. To test for any clinically meaningful differences between interventions over time, Cohen’s d effect size was used. Cohen defined d as the difference between the means, divided by the standard deviation, of either group (effect sizes: small, d = 0.2–0.3; medium, d = 0.4–0.5; and large, d = 0.8 to infinity).19 Paired samples t tests were used to assess the relationship between BDQ “days out of role” before treatment and immediately after treatment, and between-condition differences were assessed using independent samples t tests.

Intention-to-treat analysis (where patients had at least one study evaluation) was used to handle drop-out data; the last observation was carried forward over the remaining study timepoints.

Data were analysed using SPSS 17.0 for Windows (SPSS Inc, Chicago, Ill, USA).

Ethics approval

This project received approval from the University of Sydney Human Research Ethics Committee and the Sydney South West Area Health Service Ethics Review Committee (Protocol #X02-0347), as well as the Australian National University Human Research Ethics Committee (Protocol #2003/56). All participants provided informed consent to participate in the trial.

Results
Sample characteristics

Characteristics of the 90 GPs who enrolled in the trial are shown in Box 3. In addition to the SPHERE training, a quarter (22/88) indicated they had received other mental health training (eg, psychiatry training, a university Masters or Diploma course in psychological medicine).

Demographic data of the 1571 patients screened for potential eligibility for the trial are shown in Box 4. Of these patients, 84% had access to the internet for personal use, and over a fifth reported significant psychological distress (25% scored as Type 1 [substantial psychological and somatic symptoms] on the SPHERE-12, and 22% scored ≥ 22 [high to very high levels of psychological distress] on the K10). A further third of patients reported some significant psychological symptoms, with just under half reporting no symptoms.

Recruited patient characteristics

Of the 140 patients identified at screening as potentially eligible, 83 were subsequently identified by their GPs as eligible and willing to participate in the trial. The mean age of these participants was 33.7 years (range, 19–50 years), and 58 (70%) were female. Enhanced GP care alone was delivered to 54 participants (65%), and enhanced GP care plus MoodGYM to 29 (35%). Between the enhanced GP care alone and enhanced GP care plus MoodGYM groups, no significant differences were found for sex (female: 76% v 67%; χ2 = 0.70, P = 0.40) or age (mean age: 34.13 years v 32.85 years; t = 0.63, P = 0.53). These data were collected from 10 GPs in eight general practices. Each condition included four general practices (and five GPs) and did not differ significantly in terms of practice or doctor characteristics.

Resolution of symptoms

Over the four study timepoints, full SPHERE-12 and K10 data were available for 56 of the 83 recruited patients (Box 5). Of the 27 patients who dropped out, 15 were recruited to the enhanced GP care alone condition (28% of that group), and 12 to the enhanced GP care plus MoodGYM condition (41% of that group). Participants who dropped out did not differ significantly from the rest of the participants in terms of sex or age (recruited patients v drop-out patients: female, 75% v 69%, χ2 = 0.29, P = 0.59; mean age, 38.8 years v 36.0 years, t = 0.55, P = 0.58).

Box 6 and Box 7 illustrate the resolution of psychological symptoms over time, by condition, using survival functions. For most participants, symptoms resolved immediately after treatment, as assessed by both the SPHERE-12 and K10. More patients experienced a resolution of symptoms with enhanced GP care plus MoodGYM compared with enhanced GP care alone. This pattern was continued at 6 and 12 months after treatment.

For the SPHERE-12, a clinically meaningful difference of medium size was found between enhanced GP care plus MoodGYM and enhanced GP care alone immediately after treatment (mean [SD], 0.12 [0.33] v 0.28 [0.46]; d = 0.40). A small clinically meaningful difference was found between enhanced GP care plus MoodGYM and enhanced GP care alone at 6 months after treatment for both the SPHERE-12 (mean [SD], 0.06 [0.24] v 0.15 [0.37]; d = 0.29) and the K10 (mean [SD], 0.06 [0.24] v 0.15 [0.37]; d = 0.29).

Disability

An analysis of BDQ “days out of role” before treatment and immediately after treatment showed a trend towards reduction for the enhanced GP care plus MoodGYM condition (mean [SD], 4.33 [5.94] v 0.58 [1.20]; t = 2.21, P < 0.10) and a significant reduction for the enhanced GP care alone condition (mean [SD], 8.89 [8.69] v 2.93 [3.22]; t = 3.67, P < 0.05). Both before treatment and immediately after treatment, no significant differences were found between the two conditions.

Discussion

This evaluation of the feasibility of conducting a cluster randomised trial for an adjunctive internet therapy for depression in primary care settings has highlighted major logistical issues. Only 34 of 84 potential general practices proceeded to randomisation. Consequently, only 140 potential subjects of the pool of 1571 patients screened by self-report were assessed for eligibility for the trial. After exclusions, 83 of the 140 patients (59%) eventually entered the trial, indicating that about 200 subjects may have been recruited if all 84 practices (with a total of 340 potential subjects) had proceeded to randomisation.

This study shows the difficulties involved in mounting such effectiveness studies in Australian general practice. Our prior calculations had indicated that we needed to recruit and randomise at least 1460 subjects to be likely to detect a significant difference between the two interventions. Although we had no difficulty in identifying potential patients, general practices, or GPs with a significant interest in conducting the study, it appears that a combination of resource constraints (reflecting the limited funds available for implementing such studies) and practical difficulties associated with conducting interventional research in clinical settings limited our capacity to mount a large enough study to adequately test the hypothesis that adjunctive internet therapy increases the effect of enhanced GP care for depression.

While we did proceed to analysis of those who entered and completed the study, we view these data principally as providing early insight into the possible effectiveness of the intervention when delivered in real-world settings by GPs. The data provide preliminary evidence of the advantage of this style of care, in the short and longer terms, over other forms of enhanced psychological care provided by GPs. By integrating the psychological interventions contained within MoodGYM into standard care, GPs may be able to provide effective non-pharmacological interventions to patients with depression. This style of intervention is provided at no additional cost to patients or GPs, is available to anyone with internet access and can be provided at times and locations convenient to the patient.

Given these considerations, and the direction of benefit shown in this preliminary study, we believe there is a substantive case for mounting a larger and more definitive study. However, we would not repursue this course without first being able to address two key feasibility issues. First, many person-hours are required to assess patients in general practice, even by self-report. The logistics of conducting such assessments before subsequent enrolment by the practitioner are substantial. Consequently, the costs of conducting these trials are large, and they require much greater investment than that available for this feasibility study.

Second, and much more challenging, is moving from engaging general practices in cross-sectional audits of practice to participation in complex treatment trials — even when this has been made simpler by randomising practices rather than patients (and where all patients therefore receive active and credible interventions). Even though this issue has been discussed for many years, our experience has been that conducting longitudinal or interventional mental health studies in Australian general practice remains extremely challenging. We suspect that without specific and targeted support from national health service development and evaluation agencies, such research will remain neglected.20,21

Despite these barriers, we remain committed to testing internet-based adjuncts to clinical care in real-world settings. Use of the internet provides an experience of psychological therapy between the face-to-face contacts with the practitioner. This active engagement of the subject in self-directed therapy supported by a health professional combines the recognised advantages of the internet (low cost, high availability and convenience, no limitations to volume of use, high-fidelity delivery of the structured and evidence-based content of the MoodGYM CBT program) and the non-specific treatment benefits of contact with a GP with recognised psychological assessment and management skills. The suggestion of differences between this combined therapy and practitioner-delivered therapy alone is consistent with other studies of the longer-term benefits of structured forms of CBT.3,22

Our study had some limitations. The number of participants eventually recruited to the study from the potential pool of primary care patients was relatively small; and the number of participants who dropped out from the longer-term follow-up phases of the study was higher than anticipated. As discussed, these outcomes largely reflect the difficulties of undertaking research in general practice. However, they may also reflect the need for improvements in the delivery of internet-based interventions in the general practice setting. As developers of these systems, we need to engage more strongly with the needs of practitioners and the systems that support the delivery of their practices.

Australians express a strong preference for non-pharmacological therapies as first-line treatments for common mental disorders.1 Given the current geographical, financial and other service-related difficulties that many people experience when trying to access psychological care through primary care settings,12,23 the development of internet-based therapies that may reduce such barriers is a key health-service development strategy. This preliminary study suggests that MoodGYM, when integrated with psychological care provided by a GP, may have advantages over enhanced GP care alone. This remains to be formally tested in a trial with sufficient power to detect meaningful clinical differences in long-term outcomes.

1 Flow diagram of general practices through each stage of the cluster randomised trial

2 Flow diagram of patients through each stage of the cluster randomised trial


* Screening questionnaires were offered to all patients before seeing the general practitioner. † Only patients who scored positive for psychological distress and met other trial criteria were further assessed for eligibility.

3 Characteristics of the 90 enrolled general practitioners

Characteristic

No. (%)*


Age in years, mean ± SD (range)

49.1 ± 10.5  (27–85)

Female

46 (53%)

Years in primary care medicine, mean ± SD (range)

19.4 ± 9.5  (1–51)

Patients seen in 1 week, mean ± SD (range)

113 ± 55  (10–300)

Practice type

Solo

29 (33%)

Partnership/group

47 (53%)

Medical centre

12 (14%)

Full-time practice

53 (60%)

Formal mental health training

22 (25%)


* Unless otherwise indicated. Denominators vary due to missing data.

4 Characteristics of the 1571 patients screened in general practice

Characteristic

No. (%)*


Age in years, mean ± SD (range)

35.2 ± 8.5 (16–50)

Female

1191 (76%)

English as a first language

1035 (87%)

Internet access for personal use

1267 (84%)

SPHERE-12

No symptoms

528 (43%)

Type 3 (somatic symptoms only)

262 (21%)

Type 2 (psychological symptoms only)

124 (10%)

Type 1 (both psychological and somatic symptoms)

312 (25%)

K10 score (level of distress)

10–15 (low or none)

764 (49%)

16–21 (moderate)

466 (30%)

22–29 (high)

222 (14%)

30–50 (very high)

118 (8%)


SPHERE-12 = 12-item Somatic and Psychological Health Report. K10 = Kessler Psychological Distress Scale. * Unless otherwise indicated. Denominators vary due to missing data.

5 Self-reported psychological symptoms over time for recruited patients*

All participants
(n = 56)


Enhanced GP care alone (n = 39)


Enhanced GP care plus MoodGYM (n = 17)


SPHERE-12

K10

SPHERE-12

K10

SPHERE-12

K10


Before treatment

53  (95%)

55  (98%)

37  (95%)

38  (97%)

16  (94%)

17  (100%)

Immediately after treatment

13  (23%)

17  (30%)

11  (28%)

12  (31%)

2  (12%)

5  (29%)

6 months after treatment

7  (13%)

7  (13%)

6  (15%)

6  (15%)

1  (6%)

1  (6%)

12 months after treatment

5  (9%)

5  (9%)

4  (10%)

5  (13%)

1  (6%)

0


GP = general practitioner. SPHERE-12 = 12-item Somatic and Psychological Health Report. K10 = Kessler Psychological Distress Scale. * Data analysed using the intention-to-treat method. Types 1–3 (somatic or psychological symptoms or both). Patients scoring 16 or more (moderate to very high levels of distress).

6 Resolution over time of psychological symptoms on the SPHERE-12, by intervention*


SPHERE-12 = 12-item Somatic and Psychological Health Report. * Data analysed using the intention-to-treat method.

7 Resolution over time of psychological symptoms on the K10, by intervention*


K10 = Kessler Psychological Distress Scale. * Data analysed using the intention-to-treat method.

Received 
19 Jul 2009
accepted 
1 Mar 2010
Ian B Hickie, AM, MD, FRANZCP, FASSA, Executive Director1
Tracey A Davenport, BA(Hons), eMBA, Research Psychologist1
Georgina M Luscombe, BSc(Hons), PhD, Adjunct Lecturer1
Michael Moore, MB BS, FRACGP, GradDipPH, Chief Executive Officer2
Kathleen M Griffiths, PhD, Professor and e-hub Director3
Helen Christensen, PhD, FASSA, Director3
1 Brain & Mind Research Institute, University of Sydney, Sydney, NSW.
2 Central Sydney Division of General Practice, Sydney, NSW.
3 Centre for Mental Health Research, Australian National University, Canberra, ACT.
Correspondence: 
Acknowledgements: 

The MoodGYM GP trial (chief investigators: Helen Christensen, Kathleen Griffiths, Ian Hickie and Michael Moore) was funded by the Australian Government Department of Health and Ageing. The research was also supported by National Health and Medical Research Council Principal Research Fellowships 148947 and 366781. We are grateful to Zoe Richards, Wendy Rees and Andrea Fogarty for their research assistance with the study.

Competing Interests: 

Helen Christensen and Kathleen Griffiths are the developers of the MoodGYM website.

Reference Text: 
Australian Bureau of Statistics. National Survey of Mental Health and Wellbeing: summary of results, 2007. Canberra: ABS, 2008. (ABS Cat. No. 4326.0.) http://www.abs.gov.au/AUSSTATS/abs@.nsf/mf/4326.0 (accessed Jun 2009).
Reference Order: 
1
PubMed ID: 
Reference Text: 
Christensen H. Computerised therapy for psychiatric disorders. Lancet 2007; 370: 112-113.
Reference Order: 
2
PubMed ID: 
17630022
Reference Text: 
Perini S, Titov N, Andrews G. Clinician-assisted Internet-based treatment is effective for depression: randomized controlled trial. Aust N Z J Psychiatry 2009; 43: 571-578.
Reference Order: 
3
PubMed ID: 
19440890
Reference Text: 
MoodGYM [website]. http://moodgym.anu. edu.au (accessed Jun 2009).
Reference Order: 
4
PubMed ID: 
Reference Text: 
Christensen H, Griffiths KM, Korten A. Web-based cognitive behavior therapy: analysis of site usage and changes in depression and anxiety scores. J Med Internet Res 2002; 4: e3.
Reference Order: 
5
PubMed ID: 
11956035
Reference Text: 
Christensen H, Griffiths KM, Jorm AF. Delivering interventions for depression by using the internet: randomised controlled trial. BMJ 2004; 328: 265.
Reference Order: 
6
PubMed ID: 
14742346
Reference Text: 
O’Kearney R, Kang K, Christensen H, Griffiths K. A controlled trial of a school-based Internet program for reducing depressive symptoms in adolescent girls. Depress Anxiety 2009; 26: 65-72.
Reference Order: 
7
PubMed ID: 
18828141
Reference Text: 
Hickie IB, Davenport TA, Naismith SL, et al. Treatment of common mental disorders in Australian general practice. Med J Aust 2001; 175 (2 Suppl): S25-S30.
Reference Order: 
8
PubMed ID: 
11556432
Reference Text: 
Burgess PM, Pirkis JE, Slade TN, et al. Service use for mental health problems: findings from the 2007 National Survey of Mental Health and Wellbeing. Aust N Z J Psychiatry 2009; 43: 615-623.
Reference Order: 
9
PubMed ID: 
19530018
Reference Text: 
Titov N, Andrews G, Johnston L, et al. Shyness programme: longer term benefits, cost-effectiveness, and acceptability. Aust N Z J Psychiatry 2009; 43: 36-44.
Reference Order: 
10
PubMed ID: 
19085526
Reference Text: 
Hickie IB, Pirkis JE, Blashki GA, et al. General practitioners’ response to depression and anxiety in the Australian community: a preliminary analysis. Med J Aust 2004; 181 (7 Suppl): S15-S20.
Reference Order: 
11
PubMed ID: 
15462637
Reference Text: 
Hickie IB, McGorry PD. Increased access to evidence-based primary mental health care: will the implementation match the rhetoric? Med J Aust 2007; 187: 100-103.
Reference Order: 
12
PubMed ID: 
17635093
Reference Text: 
Hickie I, Scott E, Ricci C, et al. Treating depression and anxiety in general practice: training manual. Sydney: Educational Health Solutions, 1998.
Reference Order: 
13
PubMed ID: 
Reference Text: 
Naismith SL, Hickie IB, Scott EM, Davenport TA. Effects of mental health training and clinical audit on general practitioners’ management of common mental disorders. Med J Aust 2001; 175 (2 Suppl): S42-S47.
Reference Order: 
14
PubMed ID: 
11556436
Reference Text: 
Kessler RC, Andrews G, Colpe LJ, et al. Short screening scales to monitor population prevalences and trends in non-specific psychological distress. Psychol Med 2002; 32: 959-976.
Reference Order: 
15
PubMed ID: 
12214795
Reference Text: 
Hickie IB, Davenport TA, Hadzi-Pavlovic D, et al. Development of a simple screening tool for common mental disorders in general practice. Med J Aust 2001; 175 (2 Suppl): S10-S17.
Reference Order: 
16
PubMed ID: 
11556430
Reference Text: 
Australian Bureau of Statistics. Information paper: use of the Kessler Psychological Distress Scale in ABS health surveys, Australia, 2001. Canberra: ABS, 2003. (ABS Cat. No. 4817.0.55.001.) http://www.abs.gov.au/ausstats/abs@.nsf/mf/4817.0.55.001 (accessed Jun 2009).
Reference Order: 
17
PubMed ID: 
Reference Text: 
Von Korff M, Ustun TB, Ormel J, et al. Self-report disability in an international primary care study of psychological illness. J Clin Epidemiol 1996; 49: 297-303.
Reference Order: 
18
PubMed ID: 
8676177
Reference Text: 
Cohen J. Statistical power analysis for the behavioral sciences. 2nd ed. Hillsdale, NJ: Lawrence Erlbaum Associates, 1988.
Reference Order: 
19
PubMed ID: 
Reference Text: 
Anderson WP, Papadakis EM. Research to improve health practice and policy. Med J Aust 2009; 191: 646-647.
Reference Order: 
20
PubMed ID: 
20028292
Reference Text: 
Mitchell RJ, McClure RJ, Olivier J, Watson WL. Rational allocation of Australia’s research dollars: does the distribution of NHMRC funding by National Health Priority Area reflect actual disease burden? Med J Aust 2009; 191: 648-652.
Reference Order: 
21
PubMed ID: 
20028294
Reference Text: 
Perini S, Titov N, Andrews G. The Climate Sadness program: an open trial of Internet-based treatment for depression. E-J Appl Psychol 2008; 4 (2): 18-24.
Reference Order: 
22
PubMed ID: 
Reference Text: 
Rosenberg S, Hickie IB, Mendoza J. National mental health reform: less talk, more action. Med J Aust 2009; 190: 193-195.
Reference Order: 
23
PubMed ID: 
19220184