Surgery the quality of information disseminated by Web-based waiting time information services

David A Cromwell, David A Griffiths and Irene A Kreis
Med J Aust 2002; 177 (5): 253-255. || doi: 10.5694/j.1326-5377.2002.tb04758.x
Published online: 2 September 2002


Objectives: To assess Web-based waiting time information services to identify how they aimed to meet the information needs of patients and general practitioners, and to evaluate how well waiting time information was presented.

Design: A cross-sectional survey of six government websites in English-speaking countries with publicly funded hospitals. Sites were evaluated on the clarity of instructions about how their information should be used, and the accuracy of the statistics they contained was assessed indirectly using methodological criteria.

Results: The services were judged to encourage GPs and patients to use the statistics to avoid surgical units with long waiting times, but overall advice was poor. Services did not state whether the statistics predicted expected waiting times, and just one stated that the statistics were only intended as a guide. Statistics were based on different types of data, and derived at different levels of aggregation, raising questions of accuracy. Most sites explained waiting list terms, but provided inadequate advice on the uncertainty associated with making statistical inferences.

Conclusions: GPs and patients should use Web-based waiting time information services cautiously because of a lack of guidance on how to appropriately interpret the presented information.

Various governments worldwide have established websites showing how long patients wait for elective surgery at different hospitals. The general aim of these services is to assist general practitioners and patients make referral decisions, and to improve system-wide performance by removing imbalances in surgeon workloads. When launched, some websites were criticised by the medical profession as being useless (because GPs already know the waiting times of local surgeons), and potentially misleading if doctors with long waiting times are assumed to be the best.1,2 Concerns were also expressed about data quality and the accuracy with which the statistics predicted waiting times.

Although all these criticisms raise valid issues, how accurately patient waiting times can be predicted is of intrinsic importance. The level of accuracy required depends on what waiting time information GPs and patients want. It is not clear how either group interprets its information needs, but three interpretations stand out in relation to the issue of accuracy. GPs and patients can be regarded as wanting information that

  1. Predicts how long a patient might expect to wait for admission to a particular surgical unit;

  2. Identifies units at which a patient will wait different lengths of time; or

  3. Identifies units at which a patient will wait an acceptable time.

For interpretation 1, statistics have to meet an absolute standard of accuracy. For interpretations 2 and 3, their practical value will also depend on, respectively, the difference in waiting times between the various units, and the difference between the estimate and the threshold used to define an acceptable waiting time.

A typical user of a waiting time information service is unlikely to be aware of these issues. If a service is not explicit about how its information should be used, the figures could be used inappropriately.

We reviewed six Web-based waiting time information services to examine how they aimed to meet users' information needs, the potential accuracy of presented statistics, and how they advised users to interpret the presented statistics.


The websites we reviewed were the English and Welsh services,3,4 the British Columbia (BC) (Canada) service,5 and services for New South Wales, Queensland and Western Australia.6-8 These services were chosen because they were in English and provided statistics that enabled the situation at different surgical units to be compared at a surgeon or specialty level. The sites were reviewed on 22 October 2001.

Accuracy of the statistics was assessed indirectly against two potential sources of bias. The first was the type of data used. Statistics are typically derived from data on admitted patients (throughput data) or from data on patients on the waiting list (census data). Throughput data have the advantage of capturing complete waiting times, whereas census data only measure the time up to the census date. Census data can also contain the records of patients who will not subsequently be admitted.9 For these and other reasons10 throughput data statistics are often regarded as the more accurate measure of waiting time.10

The second potential source of bias was the adopted level of data aggregation. Ideally, this should account for factors that cause differences among patient waiting times, such as the level at which a list is managed (eg, surgeon, specialty) and urgency category. High levels of aggregation can hide problems of particular units, making statistics unresponsive. There is also the risk of making an ecologically biased inference about a patient's likely waiting time.11 This may arise for specialty-level statistics if lists are managed by individual surgeons and there are significant differences between surgeons within a specialty.


Box 1 summarises statements on the aim of the service, and about how statistics should be interpreted for each of the six services reviewed. Box 2 summarises the main statistics presented by the six services.

Accuracy of the statistics

There was considerable diversity across the sites in the statistics presented. First, services varied in their use of throughput or census data. From a theoretical perspective, the use of census data statistics is a concern. However, their use may be evidence that throughput data statistics are also affected by substantial levels of bias (eg, when waiting times are long, changes in behaviour will take months to show up).

Second, as waiting lists are often managed by individual surgeons, the aggregation of data at a specialty level by some services may be problematic, for the reasons outlined above. On the other hand, the precision of surgeon-level statistics might be poor if they were derived from few observations.12 This potential danger seems particularly pertinent for two services (NSW and WA) that aggregated data by surgeon and procedure, especially as both used a classification of over 100 types of procedure.

For throughput statistics, this problem of precision can be tackled by increasing the period over which data are aggregated. This might explain some of the differences between services. However, doing this is a trade-off with potential bias due to time-dependent behaviour. The NSW service seems particularly susceptible to this bias given that data are aggregated over 12 months.

Study limitations

The simplicity of this study means that it cannot cover other issues arising from the creation of waiting time information services.13,14 The study also has various limitations. The interpretation of information needs by each service was subjective (although we all agreed on the findings presented here), and the accuracy of the statistics was examined only against methodological criteria. In addition, we reviewed only six services. However, this reflected the small number of services in existence at the time of the review. A search of all government websites in English-speaking countries with publicly funded hospital services produced no other candidates for inclusion in the study.


Our study raises various issues about the quality of waiting time information presented on government websites. That few services gave users instructions on how best to interpret the statistics is a concern because of the various ways in which GPs and patients might use the figures. This could lead users to draw unwarranted conclusions. Statements are needed about how the information should, and should not, be used. Predictive accuracy is a key issue, but there are other issues to consider. For example, we would argue that services should just support interpretation 3, because it encourages change in referral patterns only when there is a problem, and thus should not result in patterns becoming unstable.

The other main concern is the uncertainty about the accuracy of the statistics. This does not mean services cannot be used successfully to avoid units with a backlog of elective patients. If the waiting time of many patients exceeds one year, concerns about accuracy become less critical. However, where average waiting times are less than six months, we suggest the services should be used cautiously until they provide guidance on what might constitute a real difference in performance.

  • David A Cromwell1
  • David A Griffiths2
  • Irene A Kreis3

  • University of Wollongong, Wollongong, NSW.


Competing interests:

None identified.

  • 1. Kent H. Waiting-list web site "inaccurate" and "misleading" BC doctors complain. CMAJ 1999; 161: 181-182.
  • 2. Whelan J. Surfing for quicker surgery. Sydney Morning Herald 2000; May 12: 4 (col 1).
  • 3. NHS Executive. Waiting lists and waiting times data for England. <>. Accessed 22 October 2001, link updated 13 September 2005.
  • 4. Health of Wales Information Service. NHS Wales waiting time information service for GPs and patients. <>. Accessed 22 October 2001.
  • 5. Ministry of Health Government of British Columbia. Surgical waiting list registry. <>. Accessed 22 October 2001.
  • 6. NSW Health. NSW Health Waiting Times Information. <>. Accessed 22 October 2001.
  • 7. Queensland Health. Elective surgery waiting list report as at 1 July 2001. <>. Accessed 22 October 2001.
  • 8. Health Department of Western Australia. Welcome to Central Wait List Bureau. <>. Accessed 22 October 2001.
  • 9. Nicholl J. Comparison of two measures of waiting times. BMJ 1989; 296: 65.
  • 10. Don B, Lee A, Goldacre MJ. Waiting list statistics III. Comparison of two measures of waiting times. BMJ 1987; 295: 1247-1248.
  • 11. Morganstern H. Ecologic studies. In: Rothman KJ, Greenland S, editors. Modern epidemiology. 2nd ed. Philadelphia: Lippincott-Raven, 1998.
  • 12. Altman DG. Practical statistics for medical research. London: Chapman and Hall, 1991.
  • 13. Smith T. Waiting times: monitoring the total postreferral wait. BMJ 1994; 309: 593-596.
  • 14. Worthington DJ. Queuing model for hospital waiting lists. J Operational Res Soc 1987; 38: 413-422.


remove_circle_outline Delete Author
add_circle_outline Add Author

Do you have any competing interests to declare? *

I/we agree to assign copyright to the Medical Journal of Australia and agree to the Conditions of publication *
I/we agree to the Terms of use of the Medical Journal of Australia *
Email me when people comment on this article

Online responses are no longer available. Please refer to our instructions for authors page for more information.