Publicly available reports of "surgical waiting times" are, at face value, of interest to patients and referring doctors wishing to access surgical care. Such information might be expected to provide a reasonable indication of the absolute time to surgical intervention for an individual patient, and allow reasonable conclusions to be drawn on the relative performance (in terms of waiting times) of surgical services.

Surgical waiting times are a specific example of "healthcare performance indicators" (see Definition). In addition to providing information for users, such indicators are likely to inform the opinions of politicians, journalists, hospital managers and state and federal health departments on the adequacy of our healthcare system and relative hospital or regional performances. They may be used to construct "league tables" of the relative performance of surgical units — individual hospitals, surgical units or surgeons may be deemed to have "good" or even "substandard" performance. Public outcries and political pointscoring are likely to ensue.

Good indicators should be easy to understand and use by the intended audience. Depending on how these data are collected, processed and presented, reported waiting time data might or might not provide useful information to people seeking guidance on time to treatment. Reports of surgical waiting times that use different definitions of "waiting time", or simply report on past performance, are of limited value. Waiting time data presented as the frequency with which a certain proportion of patients receive treatment within a stated time (eg, 75% treated within 4 weeks) may also fail to adequately inform patients or general practitioners as to likely delays. Few existing systems are capable of adjusting for delays before initial surgical consultation (ie, waiting time to get onto the surgical waiting list), let alone factors such as primary illness severity, comorbidity or health insurance status — all of which influence actual waiting times.

In this issue of the Journal (page 253), Cromwell et al report an assessment of the utility of information regarding surgical waiting times available on the World Wide Web.1 Their findings indicate that current Web-derived information has significant shortcomings in data quality. They conclude that waiting time data currently published on the Web are, by and large, unsuitable for informing either clinician referral or patient decision-making.1

This critique should not be misinterpreted as an example of the well-recognised "dot.com" data reliability phenomenon. It is not just Web-based sources of such data that are open to criticism. Analysis of healthcare performance indicator data derived from any existing sources would generate similar critiques, with similar caveats required on interpretation and use.2,3 The appropriate desire to develop performance indicators in healthcare has often seen a race to deliver indicators overwhelm the need for methodological rigour in development and implementation. All too often, too little emphasis is placed on initial identification of who will use the indicator and how and why they will apply the data. The absence of such ab initio clarity of purpose leads to performance indicators that do not meet the needs and expectations of consumers, providers or purchasers of healthcare services.2-5

Surgical waiting times, and many other indicators, generate a natural curiosity to compare or rank relative performance. For comparisons to be made, common indicator definitions must exist and be systematically applied in data generation, with common data collection methodologies and results that are risk-adjusted. As Cromwell et al found, requirements for clear, consistent definitions are frequently not met, rendering comparisons invalid.2-8 It is crucial that people intending to use indicator data for judging comparative performance, or in any potentially punitive fashion, fully understand the strengths and weaknesses of the primary data. Perhaps the greatest error by those who use indicator data is that of assuming the indicator is an objective measure of relative performance based solely upon its apparent face validity. Reported surgical waiting times would then be assumed to be a direct linear measure of access to care. This ignores evidence that clinician decision-making processes and administrative practices have major impacts on reported waiting times. Without adjustment for relative urgency or disease severity (at a minimum), reports of waiting times are of limited utility.

Significant progress has been made in developing and refining healthcare performance measurement locally (by the Australian Council on Healthcare Standards among others) and internationally (by groups such as the Joint Commission on Accreditation of Healthcare Organisations and the Health Care Financing Administration in North America). There is, however, still considerable scope for improving the methodological rigour of both indicator development and application in the field. At present, the reliability and utility of indicator data cannot be assumed.

Most current indicators of healthcare performance should be viewed as tools that prompt additional inquiry, rather than allowing definitive judgements on quality and safety of care. Over time, robust, credible indicators will increasingly become available to reliably inform consumers and allow accountability to purchasers of healthcare services. Nevertheless, given the complexity of healthcare, the predominant enduring benefit from attempts to measure performance in healthcare is likely to be the use of data generated by providers of care to provoke reflection on existing practice and to plan efforts at improving care.