ONLINE and mobile application symptom checkers get the diagnosis right first time in just a third of instances, according to research published online today by the Medical Journal of Australia.
Researchers from Edith Cowan University in Perth tested 36 symptom checkers with 48 medical condition vignettes designed to assess both the diagnostic accuracy and appropriate triage advice.
“The 27 diagnostic symptom checkers listed the correct diagnosis first in 421 of 1170 vignette tests (36%), among the top three results in 606 tests (52%), and among the top 10 results in 681 tests (58%),” the authors, led by Michella Hill, found.
“Symptom checkers (SCs) using artificial intelligence algorithms listed the correct diagnosis first in 46% of tests compared with 32% for other SCs.
“The 19 triage SCs provided correct advice for 338 of 688 vignette tests (49%). Appropriate triage advice was more frequent for emergency care (63%) and urgent care vignette tests (56%) than for non-urgent care (30%) and self-care tests (40%).
“Triage advice, especially for less serious case vignettes, tended to be risk-averse, although there were also notable instances of the opposite; the former can place unnecessary burdens on health care systems, but the latter can be life-threatening,” Hill and colleagues wrote.
“Diagnosis is not a single assessment, but rather a process requiring knowledge, experience, clinical examination and testing, and the passage of time, impossible to replicate in a single online interaction. Overseas SCs also suggest health care services inappropriate for users in Australia, including self-referral to specialist care. As reported by other investigators, some SCs did not report specific information about their authors or editors, or about the validity of their advice.”
Hill and colleagues did say there were some useful features of online symptom checkers.
“Epidemiological data can be tracked on sites such as WebMD, providing timely data to health professionals. Users can educate themselves about their own health, potentially improving patient–physician relationships. SCs can also direct people to appropriate care, and some tools are even directly linked with health care services” they wrote.
In Australia, the Therapeutic Goods Administration oversees the regulation of medical software products and devices, but cannot regulate symptom checkers because they are not classified as medical devices.
“The medical terminology used in SCs and its effect on comprehensibility for users and their adherence to the advice provided should be investigated, as should the real life performance of SCs and their impact on health outcomes,” Hill and colleagues concluded.
“To diminish the burden on health care systems, particularly emergency care, it is vital that online tools promptly direct people to appropriate care. Accordingly, online programs should be backed by quality sources, and provide diagnostic or triage advice that is as accurate as possible.”
All MJA media releases are open access and can be found at: https://www.mja.com.au/journal/media
- Cate Swannell
- Medical Journal of Australia