In evaluating evidence for clinical care, study designs are graded according to their potential to eliminate bias,1 and the most robust evidence is considered to come from randomised controlled trials (RCTs).2,3 However, the reliance on study design as the main criterion for credibility of evidence has its critics,4 as does this view of what constitutes the “best” evidence.5,6 In public health in particular, there is debate about the primacy of the RCT for evaluating interventions and about the tendency to downgrade the contribution of observational studies.7,8

More recently, this debate has moved to emerging research areas, such as palliative care. This discipline urgently requires a wider evidence base, but acquiring this evidence presents particular problems.

In palliative care research, methodological difficulties arise because of the complex physical, psychological, existential and spiritual problems faced by patients, families and professionals.9-13 These difficulties include patient recruitment, gate-keeping by professionals (ie, reluctance to enrol patients in research studies), small sample sizes, high attrition rates, rapidly changing clinical situations and limited survival times.10,12,13

Palliative care research often focuses on the effectiveness of services for populations, rather than the effect of treatments on individual patients.9 Trials of palliative care services are almost entirely pragmatic (ie, they compare a new service with current best practice).13 The difficulties in identifying, recruiting and retaining patients mean that study populations often comprise those who are best able to cope and least ill. As palliative care is by its nature holistic and often tailored towards the needs of individual patients (pain relief and improved quality of life), it may be difficult to define the intervention precisely and uniformly.

Palliative care is also characterised by a multidisciplinary approach. It can be difficult, and possibly also inappropriate, to isolate an individual intervention from a multidisciplinary approach. In addition, treatments that involve various components, changes in services, and surgical or radiological interventions are harder to deliver in a blinded manner to all concerned. Because treatment packages are the mainstay of palliative care research, the ideal type of RCT is seriously compromised.13

It is also important to reflect on the outcomes that we wish to assess. In general, the outcomes of RCTs are to reduce mortality and morbidity and improve survival.14,15 However, extending life is not the central aim of palliative care services, and duration of survival may therefore be irrelevant. Instead, symptom management and health-related quality of life are important outcomes. The timing of measurements is also crucial for trials, yet timing in palliative care is problematic because of the short time between eligibility and death.13

Furthermore, RCTs have been considered inappropriate or unethical in palliative care.9 They are seldom acceptable to patients and their families, who may not wish to risk reducing the quality of life in their remaining days in a trial with a non-intervention arm. Deliberate withholding of support services from the control group has been deemed unethical,16 and it is difficult for researchers to easily gain control “within ethically defensible limits”.13 For example, the Cambridge Hospital at Home study compared 186 patients randomised to receive up to 2 weeks of 24-hour nursing care when nearing death, with 43 control patients on an intention-to-treat basis.17 Problems included the limited power of the study to show differences, service resource constraint of 2 weeks, doctors not wanting to withdraw a desirable service before a patient’s death, 39% of the intervention group dying before receiving the intervention, and the control group receiving an alternative good nursing service. These problems made it difficult to show the worth of the intervention.

It is difficult to grade published studies in palliative care using the traditional taxonomies for levels of evidence. Our recent literature review during the preparation of evidence-based guidelines for palliative care in aged care18 revealed numerous problems; many publications fell into evidence levels III (non-randomised comparative studies) and IV (case series),15 and many of the studies could not have been ethically conducted as RCTs. Consequently, to ensure a consistent, defensible approach to evaluating the available studies, we adapted traditional taxonomies in accord with recommendations of the National Health and Medical Research Council (NHMRC).19 We scored studies for quality of methods used to minimise bias, strength and relevance and, based on these scores, defined two new levels of evidence — qualitative evidence and consensus opinion of experts in the field (Box). Although some may consider these levels of evidence less rigorous, we believe that, given the limitations of the study designs, they are the most appropriate criteria for assessing evidence to guide palliative care practice.

There have been calls in both public health and palliative care for study designs to incorporate the social, economic and political factors that usually influence the effectiveness of the intervention.4,5,14,20,21 The NHMRC has recognised that clinical practice guidelines may improve health more readily for the relatively health-advantaged than for the relatively disadvantaged, potentially increasing health inequalities.22 In response, the NHMRC has developed a framework for incorporating evidence about socioeconomic position and health into these guidelines.14

The tendency for evidence classified as “best” (based on study design) to have been gathered on simple interventions and from groups that are easy to reach in a population raises issues about its relevance and transferability to other groups. Assessing evidence on multiple dimensions would better allow these issues to be taken into account For example, it has been suggested that evidence on the effectiveness of public health interventions should be assessed on three dimensions, similar to those we devised for palliative care interventions, namely: strength of the evidence, which is determined by a combination of study design (level), methodological quality and statistical precision; magnitude of the measured effects; and relevance of the measured effects to the context in which the intervention is to be implemented.4 A pragmatic approach is recommended when considering the importance of study design relative to the other dimensions.4 Study design should not be seen as synonymous with quality of evidence, as it is only one aspect.

There are many useful observational designs, including, in particular, prospective open-label studies.23,24 These have a more realistic methodology for palliative care research, with each patient acting as as his or her own control, and data compared before and after the intervention. For example, the efficacy of ketamine as an analgesic was investigated with a prospective, multicentre, unblinded, open-label audit: 39 patients received a 3–5 day continuous subcutaneous infusion of ketamine, in addition to their existing analgesic regimen.23 Patients who achieved a 50% or greater reduction in mean pain scores were designated responders. The responder rate was 67%. A second trial on 43 patients in eight centres found a responder rate of 51%.24 The authors concluded that such data can be used to inform practice, if input and output data are rigorously recorded, and patients act as their own controls.23,24

Quality improvement methods are emerging as a way of obtaining evidence in palliative care. These methods involve stating an aim, measuring success, and testing possible improvements, for example through a PDSA (“Plan, Do, Study, and Act on new insights”) cycle. These cycles can generate deep understanding of complex systems and make sustainable improvements rapidly.25

Although RCTs have their place whenever possible,10,26 the above alternative designs may offer more feasible research protocols that can be successfully implemented in palliative care. If studies are to be fairly and accurately graded for the development of evidence-based guidelines, a second look at this taxonomy is warranted.