Surgery and the randomised controlled trial: past, present and future

Michael J Solomon and Robin S McLeod

One must attend in medical practice not primarily to plausible theories, but to experience combined with reason -- Hippocrates¹

MJA 1998; 169: 380-383
For editorial comment, see Maddern
 

→ Other articles have cited this article

Synopsis - Introduction - How do surgeons assess treatment effectiveness? - Who is performing surgical RCTs and where do they publish? - How good are surgical randomised controlled trials? - Are there problems initiating surgical RCTs? - Where to now? - Conclusion - References - Authors' details
Make a comment - Register to be notified of new articles by e-mail - Current contents list - More articles on Surgery - ©MJA1998

Synopsis	Randomised controlled trials (RCTs), with their prospective definition of methods and outcome measures, double-blind assessment of outcomes and unbiased selection of subjects and controls, provide the best possible evidence for deciding the value of a medical or surgical intervention. Few surgical studies are designed as RCTs, and those that are should be of a higher quality. The lack of good surgical RCTs may be a result of surgeons lacking the necessary training, expertise and desire to perform RCTs, inadequate funding from granting agencies, difficulties in securing patient consent or a lack of sufficient patient numbers. If an RCT is not feasible for a particular study, then alternative research designs, such as prospective matched-pair trials, may need to be better developed and used. If RCTs can be performed, other strategies to increase the number and quality of RCTs may be needed: Education of surgeons in clinical research methods Improved funding of surgical RCTs Compulsory evaluation of new techniques and technology before their general adoption is permitted.
Introduction	Recent decades have brought an increasing awareness of the need for stronger clinical research methods. Statistical and epidemiological principles have been adapted to clinical research, and there has been recognition of a hierarchy of research designs, their individual strengths and weaknesses, and of the clinical situations to which each is suited. Understanding of the attributes and advantages of the controlled clinical trial has grown. In the medical literature, this has led to an increase in the proportion of studies based on comparison of two groups, particularly since the 1980s.2-5 In contrast, the surgical literature has shown no increase in clinical trials or studies with adequate control groups.6-8 Surgeons have been criticised for a lack of adequate scientific assessment of new and old techniques and technology. In response, it has been argued that the problems of surgery lie not with the quality of clinical research but with doctor-patient communication, and that more science and more clinical trials cannot heal the deep rifts in communication between surgeons and their patients.9 Given the general acceptance of the randomised controlled trial (RCT) as the "gold standard" in clinical research, why is it not used more often to research surgical techniques and interventions? Is it the most appropriate research design for clinical research in surgery? What are the barriers to, or arguments against it?   How do surgeons assess treatment effectiveness? RCTs comprise only 3%-9% of clinical study designs across all areas of surgery, despite the rapid expansion of new technology in surgery and the linear improvement in adoption of RCTs in other areas of medicine (see Box).5,6,10-12 Several publications have assessed clinical study designs adopted in general surgery. Pollock determined the relative frequency of RCTs in the British Journal of Surgery: in 1965 there were none; in 1975, 7%; and in 1985, 9%.12 A more recent assessment of all clinical studies, published between 1980 and 1990 in three major international surgical journals, has demonstrated little change in the frequency of different clinical study designs.6 Retrospective studies involving only one cohort of patients make up well over 80% of clinical studies in the surgical literature sampled in the past 15 years, with case reports comprising a third of all the clinical studies.6 The frequency of case reports decreased over the 1980s, but they were replaced, not by clinical studies with stronger research designs, but by cross-sectional surveys and case series.6 Surgical specialties show similar patterns. In the 1970s, 45% of abstracts accepted at the annual joint meetings of the Society for Vascular Surgery and the International Society for Cardiovascular Surgery were cross-sectional surveys.10 The proportion of cross-sectional surveys decreased in the 1980s to 30%, while RCTs increased, but from only 1% in the 1970s to 5% in the 1980s.11 Only 5% of articles in the Journal of Neurosurgery between 1973 and 1977 were controlled clinical trials.11   Who is performing surgical RCTs and where do they publish? Given that the surgical literature contains few RCTs in surgery compared with the plethora of case studies, are surgical RCTs published elsewhere, or under a different guise? A search of all RCTs published in general surgery showed that only a third had a surgeon as the principal author, only a quarter were published in surgical journals (the rest being published in medical journals), and (most importantly) less than a quarter actually assessed surgical operations. Most trials looked at medical therapies in surgical patients (eg, adjuvant chemotherapy, or prophylaxis against postoperative infection or deep venous thrombosis).13   How good are surgical randomised controlled trials? The general surgical literature shows an increase in knowledge of trial design during the past decade, with increases in sample sizes, multicentre studies and duration of follow-up, although there has been no increase in the number of prospective trials or cohort studies.6 There are, nevertheless, problems with the standard of the RCTs performed; in particular, the false negative conclusions (Type II errors) that are perceived to be the primary result of so many RCTs in surgery.14 Contrary to this, Gilbert et al found that about 50% of surgical RCTs find in favour of the new therapy, although the authors did not specify the number of RCTs that involved surgical operations, whether surgeons performed the RCTs, or the standard of the RCTs.15 Three recent reviews of the strengths and weaknesses of surgical RCTs used qualitative scoring systems to assess the studies in terms of their science rather than their clinical applicability.6,13,16 In more than 90% of trials reviewed, basic issues such as sample size calculation were ignored, so inadequacies in the power of the study to make valid conclusions were not considered. More than a third of trials did not specify exclusion criteria, and an equal proportion made no attempt to eliminate bias by measuring outcomes objectively. In 20% of surgical RCTs, the conclusions were not justified by the data. The most recent review focused on RCTs in laparoscopic surgery and found major errors in sample size assessments and in determining objective outcomes.16 Using a 10-point qualitative score to compare the standard of RCTs, nonrandomised controlled trials and retrospective cohort designs in general surgery since 1980, there was found to be some improvement in all types of comparative studies, including RCTs, but the differences did not reach significance.6 Using the more complex scoring system devised by Chalmers et al for meta-analysis of RCTs, it is possible to compare trials from different specialties in medicine and surgery. Chalmers' group has shown the standard of RCT in medicine increased in a linear fashion through the 1980s.5,17 In surgery, the only study of RCTs using Chalmers' system was performed in general surgery for the year 1990, and the mean quality score for these surgical RCTs was similar to that in other medical disciplines.13 Nevertheless, the quality was shown to be lower if a surgeon was the principal author, and the quality varied depending on whether the trial was published in a medical or surgical journal. The number of centres involved in trials also influenced the quality, and trials involving surgical operations in an arm of the trial were of significantly lower standard than those comparing medical therapies in surgical patients. There were significant differences in the standard of trials across the general surgical specialty subgroups.13   Are there problems initiating surgical RCTs? Despite several successful RCTs of surgical operations in the 1960s, the fall of the surgical RCT has perhaps been more notable than its rise.12,18-22 Why is the RCT not more widely adopted for assessing the effectiveness of surgical operations? Recruitment problems Even when there is indecision among competent experts faced with competing therapies (equipoise), there may be specific methodological and feasibility problems in surgery that make randomisation difficult.23-26 There seem to be important differences in compliance with randomised allocation of treatment, irrespective of patient preferences, when comparing drugs with surgery.27 Perhaps the most poorly understood variable, however, is patient preference.28-30 Whether a patient elects to enter a clinical trial may reflect a principled view of trials themselves, but may also result from differences in the "magnitude" of competing therapies (eg, surgery v. drugs, or major v. minor surgical procedures).31-32 Doctors, when asked to choose from competing therapies as expert "surrogates" for their patients, show a low and variable acceptance of trials (3%-60%), and their preferences for therapies depend on their specialty training and geographical differences (rural v. urban) in their practices.33,34 RCTs have successfully assessed therapies of unequal magnitude in the past: for instance, coronary artery surgery versus medical therapy for ischaemic heart disease, and lumpectomy versus mastectomy for early stage breast cancer, although poor accrual did threaten the latter trial, and the external validity (generalisability) remains in question for the larger group of non-participant breast cancer patients.35 Another major barrier may be that surgeons perceive randomisation of therapies as creating uncertainty, both for themselves and their patients.35,36 One review assessed all the articles in gastrointestinal surgery published in one year, identified the treatment effectiveness hypotheses discussed in each article and determined whether an RCT could have been performed.28 Less than 40% of the published hypotheses involving surgical operations would have been answered successfully by an RCT, even assuming an ideal setting with unlimited resources and availability of all clinical cases. The two most important barriers were, firstly, that the disease incidence was not common enough to perform an RCT even with 100% accrual and, secondly, that patient preference was highly likely to severely limit accrual due to a dramatic inequality in the magnitude of the competing therapies. These two factors alone would have precluded 47% of all treatment evaluation questions involving surgical operations. Methodological issues would have prevented an RCT in only 2% of cases.28 This is clearly only part of the answer. While this review suggests that, even in an ideal clinical research setting, RCTs are applicable to less than 40% of questions involving surgical treatment options, published surgical RCTs account for only 3%-9% of articles.28 Attitudinal problems It may be that surgeons see little need for RCTs. If the proportion of different study designs in the literature is an accurate guide, surgeons appear to accept the results of weaker clinical studies with no comparative groups, no prospective collection of data and no attempt to minimise selection bias.6,10-12 Perhaps clinical research lacks the same esteem that basic research affords in academic surgery; this would explain the abundance of surgical journals weighted heavily towards basic research and the lack of a surgical journal where high quality surgical clinical trials can be found. Funding problems Few surgeons appear to obtain funding for trials, although how many apply is unknown.12 This may reflect a lack of interest in surgical trials among potential funders (industry, community organisations and government bodies), a lack of training on the part of surgeons, or lack of a controlling body like the United States Food and Drug Administration.6,8,37 Good trial design and quality assurance (internal validity) are clearly major issues in surgical RCTs, and it is possible that the qualitative weaknesses of surgical RCTs (which may have a lower standard of research protocol) influences funding allocations from major granting bodies. If so, this may counter criticisms that the lack of funding for surgical RCTs is because physicians and laboratory researchers are overrepresented on granting bodies. Recent experiences in laparoscopically assisted colorectal cancer surgery have highlighted many difficulties in the assessment of new technology and the role of the RCT. Despite initial enthusiasm and widespread adoption in the early 1990s (without adequate trials), the potential to do harm from port-site recurrence (cancer occurring in the laparoscopic port holes) and the additional expense of this surgical technique mean that it is now largely abandoned (without adequate trials).38 Problems with standardisation of therapy and accreditation of surgeons in laparoscopic resection have been major issues in international trials and in planned trials in Australia and New Zealand.39 Multicentre trials in Australia were supported by the Royal Australasian College of Surgeons and the Commonwealth Department of Human Services and Health in 1995, but, despite good trial design involving the NHMRC Clinical Trials Centre, five major colorectal centres, two years of planning and a clinical question with significant ramifications for costs of treatment, quality of life and survival, federal funding was withdrawn before the first patient entered the trial.40 Lack of funding is obviously a major deterrent to large surgical RCTs.   Where to now? For the questions that can perhaps be answered by RCTs, the number and standard of RCTs, and thus the quality of evaluation of surgical interventions, may be improved by more economic encouragement from funding agencies and stronger control of new technology by governing bodies (eg, specialist societies, surgical colleges), with a requirement that trials be carried out before, rather than after, widespread adoption of new procedures. It is clear that not all clinical questions can be addressed effectively by an RCT. Rather than continue to assert that the RCT is the only method of evaluating treatment effectiveness and blaming surgeons for not adopting this standard, we need to define better the clinical treatment effectiveness questions that can and should be answered by an RCT. This may involve preference trials (pilot studies of patient and clinician compliance with treatment allocations) before RCTs, as the preferences of both patient and surgeon are probably one of the most significant barriers to the surgical RCT. The development of alternative research designs, both prospective and retrospective, with methods modelled on the rigour of RCT, remains a task for clinical researchers and epidemiologists.28,33-35 For example, matched-pair trials (where preferences are incorporated into the trial design) can still include randomisation for the patients and surgeons, with no strong preferences.28 Internal validity can still be strong, with rigid inclusion and exclusion criteria, prospective data collection, objective outcome measures, and stratified analyses performed to take into account these preferences and the lack of randomisation at inception. While the effects of random error cannot be minimised, systematic error can be reduced with good trial design and analysis. The disadvantages of the effect of random error in these trial designs without randomisation may be minimised by better accrual of the total cohort of eligible patients, avoiding selection biases and improving the generalisability of the results (external validity).41-43 Not only can the lessons learned in minimising bias in RCTs be transferred to other prospective designs, but they can help to improve the quality of retrospective research and database analyses. As with controlled trials, such analyses can adopt inclusion and exclusion criteria, with a log of all eligible and non-eligible patients, emphasis on hard outcomes such as survival, sample size determinants before outcome analysis, attempts at blinding the assessor to treatment groups and stratified and matched-pair analyses to control for systematic error.
Conclusion	Whether the lack of good surgical RCTs is because surgeons lack the necessary training, expertise and desire to perform RCTs, inadequate funding from granting agencies, or methodological problems is not entirely clear. If there are problems precluding an RCT in a significant proportion of clinical treatment effectiveness questions, then alternative research designs, such as prospective matched-pair trials, may need to be better developed and used. If RCTs can be performed, other strategies to increase the number and quality of RCTs need to be adopted, including continuing education of surgeons in clinical research methods, compulsory evaluation of new techniques and technology by governing bodies and more funding for clinical research.
References	Jones WHS. Hippocrates (transl). Vol. 1. London: Heinemann, 1923: 313. Fletcher RT, Fletcher SW. Clinical research in general medical journals: a 30 year perspective. N Engl J Med 1979; 301: 180-183. Feinstein AR. Clinical biostatistics. XLIV. A survey of the research architecture used for publications in general medical journals. Clin Pharmacol Ther 1978; 24: 117-125. Kramer MS, Boivin JF. Toward an "unconfounded" classification of epidemiology research design. J Chron Dis 1987; 40: 683-688. Emerson JD, Berdick E, Hoaglin DC, et al. An empirical study of the possible relation of treatment differences to quality scores in controlled randomized clinical trials. Controlled Clin Trials 1990; 11: 339-352. Solomon MJ, McLeod RS. Clinical studies in surgical journals. Have we Improved? Dis Colon Rectum 1993; 36: 43-48. Spodick DH. The surgical mystique and the double standard: controlled trials of medical and surgical therapy for cardiac disease: analysis, hypothesis, proposal. Am Heart J 1973; 85: 579-583. Spodick DH. Numerators without denominators. There is no FDA for the Surgeon. JAMA 1975; 232: 35-38. Little JM. Humane medicine. Cambridge: Cambridge University Press, 1995. Barnes RW. Understanding investigative clinical trials. J Vasc Surg 1989; 9: 609-618. Haines SJ. Randomized clinical trials in the evaluation of surgical innovation. J Neurosurg 1979; 51: 5-11. Pollock AV. The rise and fall of the random controlled trial in surgery. Theor Surg 1989; 4: 163-170. Solomon MJ, McLeod RS, Laxamana A, Devore L. Randomized controlled trials in surgery. Surgery 1994; 115: 707-712. Sacks H, Chalmers TC, Smith H Jr. Randomized versus historical controls for clinical trials. Am J Med 1982; 72: 233-240. Gilbert JP, McPeek B, Mosteller F. Statistics and ethics in surgery and anaesthesia. Science 1977; 198: 684-689. Slim K, Bousquet J, Kwiatkowski F, et al. Analysis of randomized controlled trials in laparoscopic surgery. Br J Surg 1997; 84: 610-614. Chalmers TC, Smith H Jr, Blackburn B, et al. A method for assessing the quality of a randomized control trial. Controlled Clin Trials 1981; 2: 31-49. Goligher JC, Pulvertaft CN, Watkinson G. Controlled trial of vagotomy and gastroenterostomy, vagotomy and antrectomy and subtotal gastrectomy in elective treatment of duodenal ulcer. Br Med J 1964; 1: 455-460. Conn HO, Lindenmuth WW. Prophylactic portacaval anastomosis in cirrhotic patients with esophageal varices: preliminary report of controlled study. N Engl J Med 1962; 266: 743-749. Garceau AJ, Donaldson RM, O'Hara ET, et al. A controlled trial of prophylactic portacaval-shunt surgery. N Engl J Med 1964; 270: 496-500. Oettinger W, Berger HG. Commentary on "the rise and fall of the random controlled trial in surgery". Theor Surg 1989; 4: 170. Russell PS. Commentary on the rise and fall of the random controlled trial in surgery. Theor Surg 1989; 4: 169-170. Fielding LP, Stewart-Brown S, Dudley HAF. Surgeon-related variables and the clinical trial. Lancet 1978; 2: 778-779. Van der Linden W. Pitfalls in randomized surgical trials. Surgery 1980; 87: 258-262. Balch CM, Durant JR, Bartolucci AA. The impact of surgical quality control in multi-institutional group trials involving adjuvant cancer treatments. Ann Surg 1983; 198: 164-167. Neugebauer E, Troidl H, Spangenberger W, et al. Conventional versus laparoscopic cholecystectomy and the randomized controlled trial. Br J Surg 1991; 78: 150-154. Love JW. Drugs and operations; some important differences. JAMA 1975; 232: 37-38. Solomon MJ, McLeod RS. Should we be performing more randomized controlled trials evaluating surgical operations? Surgery 1995; 118: 459-467. Barofsky I, Sugarbaker PH. Determinants of patient non-participation in randomized clinical trials for treatment of sarcomas. Cancer Clin Trials 1979; 2: 237-246. Llewellyn-Thomas HA, McGreal MJ, Thiel EC, et al. Patients' willingness to enter clinical trials: measuring the association with perceived benefit and preference for decision participation. Soc Sci Med 1991; 32: 35-42. Angell M. Patients' preferences in randomized clinical trials. N Engl J Med 1984; 310: 1385-1387. Brewin CR, Bradley L. Patient preferences and randomized clinical trials. Br Med J 1989; 299: 313-315. MacKillop WJ, Ward GK, O'Sullivan B. The use of expert surrogates to evaluate clinical trials in non-small cell lung cancer. Br J Cancer 1986; 54: 661-667. Moore MJ, O'Sullivan B, Tannock IF. How expert physicians would wish to be treated if they had genitourinary cancer. J Clin Oncol 1988; 6: 1736-1745. Taylor KM, Mangolese RG, Soskolne CL. Physicians' reasons for not entering eligible patients in a randomized clinical trial of surgery for breast cancer. N Engl J Med 1984; 310: 1363-1367. Hellman S. Randomized clinical trials and the doctor-patient relationship. An ethical dilemma. Cancer Clin Trials 1979; 2: 189-193. Beahrs OH. Clinical trials from a surgeon's view. Cancer 1990; 65 Suppl: 2383-2384. Solomon MJ, Egan M, Roberts R, et al. Incidence of free colorectal cancer cells on the peritoneal surface. Dis Colon Rectum, 1997; 40: 1294-1298. Bagshaw PF, Allardyce RA. NZ national colon cancer trial. Aust N Z J Surg 1997; 66 Suppl 1: 50: A13. Hewett P. The Australian laparoscopic assisted resection for adenocarcinoma of the colon clinical trial. Aust N Z J Surg 1997; 66 Suppl 1: 49: A13. Hamburg D, in "Clinical investigations in the 1980s. Needs and opportunities" (conference summary). Washington, DC: Washington Institute of Medicine, 1981: 2. (Report No.10M-81-007.) Cross design synthesis: a new strategy for medical effectiveness research. Washington, DC: US Government Accounting Office, 1992. (GAO/PEMD-92-18.) Rabenech L, Visioli CM, Horwitz RI. Problems in the conduct and analysis of randomized clinical trials. Are we getting the right answers to the wrong questions? Arch Intern Med 1992; 152: 507-512. Hill AB (Br Med J 1948; 2: 791). Reprinted in Statistical Methods in Clinical and Preventive Medicine. Oxford University Press, New York, 1962. Hill AB. Medical ethics and controlled trials. Br Med J 1963; 1043-1048. Cochrane AL. Effectiveness and efficacy. Random reflections on health services. London: Nuffield Provincial Hospitals Trust, 1972. Sackett DL, Haynes RB, Tugwell, P. Clinical epidemiology. A basic science for clinical medicine. Boston: Little, Brown & Company, 1985. Chalmers TC, Celano P, Sacks HS, Smith HJR. Bias in treatment assignment in controlled clinical trials. N Engl J Med 1983; 309: 1358-1361. Marsoni S, Torri W, Taiana A, et al. Critical review of the quality and development of randomized clinical trials (RCTs) and their influence on the treatment of advanced ovarian cancer. Ann Oncol 1990; 1: 343-350.

Authors' details

Department of Surgery, University of Toronto, Toronto, Canada.
Robin S McLeod, MD, FRCSC, FACS, Professor of Surgery.

Reprints will not be available from the authors.
Correspondence: Associate Professor Michael J Solomon, RPAH Medical Centre, Suite 419/100 Carillon Avenue, Newtown, NSW 2042.

Other articles have cited this article:

  Adam B Chapman and Julian A Feller. Therapeutic arthroscopy for knee osteoarthritis: time to reconsider? Med J Aust 2003; 179 (4): 179-180. [Editorials] <http://www.mja.com.au/public/issues/179_04_180803/cha10283_fm-1.html>