Machine learning in clinical practice: prospects and pitfalls

Machine learning has huge potential to enhance clinical decision making, but there are still many limitations

Machine learning (ML), a subdiscipline of artificial intelligence, encompasses a family of computerised (machine) methods that identify (learn) patterns in large (training) datasets not detectable to humans (Box 1). Identified patterns are then encoded in a computer model or algorithm which is then tested and validated on new data. Three basic ML types exist (Box 2), with supervised and reinforcement learning being used most frequently.

Algorithms can take various forms — deep neural networks (deep learning systems) currently dominate.1 These networks comprise algorithms that cluster and classify information in a manner resembling the human brain. Just as neural synapses are strengthened through repeated activity, deep neural networks iteratively strengthen their functions through mathematical means, adjusting the weights of inputs as they move through layers of intermediate nodes (neurons) towards a desired output.

Recent renewed interest in ML is driven by the availability of massive digitised datasets from genomics, biobanks, medical images, administrative datasets, electronic health records and wearable biosensors; advances in computer processing speed, storage and internet‐mediated computing power; and increasing commercial investment in ML.2 Here, we introduce ML applications with potential clinical utility while considering current limitations. More exhaustive reviews of ML methodology and health care uses can be found elsewhere.3,4

Emerging machine learning applications

Clinical applications of artificial intelligence systems built using ML are being used to diagnose conditions, estimate risk, select optimal treatments and determine drug dosing. Some applications can perform as well as, if not faster or more accurately than, clinicians, and do so more consistently, devoid of human error from fatigue. ML applications are currently most advanced for diagnoses based on imaging data and risk prediction.

Diagnostic applications

Retinal cameras incorporating ML algorithms can confer high diagnostic accuracy in general practice in the early diagnosis of referable diabetic retinopathy, possible glaucoma and age‐related macular degeneration. In one study, a deep learning system was trained on a total of 494 661 retinal images to detect these conditions in community and clinic‐based multi‐ethnic populations with diabetes. The deep learning system was then tested on 71 896 images from 14 880 patients, and its results were validated by trained eye specialists. The deep learning system diagnosed referable diabetic retinopathy, possible glaucoma and age‐related macular degeneration with sensitivity/specificity of 90.5%/91.6%, 96.4%/87.2%, and 93.2%/88.7%, respectively.5 Similarly, another deep learning system demonstrated sensitivity, specificity and discrimination (area under the receiver operator characteristic curve [AUC]) for diagnosing referable diabetic retinopathy of 87.0–90.3%, 98.1–98.5% and 0.990–0.991 (0.80–1.00 representing good to perfect discrimination), respectively.6 A successful trial in primary care led the United States Food and Drug Administration to list a commercial retinal camera system as its first ML‐based medical application.7

In dermatology, early and accurate diagnosis of suspicious skin lesions could be aided by smartphone cameras with ML‐based apps, allowing individuals to conveniently perform their own regular skin checks and transmit results to specialists who may not be easily accessible. Trained on 129 450 dermoscopy images of 2032 different skin conditions, a deep learning system was tested against 21 dermatologists using 1700 biopsy‐proven examples of keratinocyte carcinomas, benign seborrhoeic keratoses, melanomas and benign nevi as reference.8 The deep learning system in all cases matched and, in some cases, out‐diagnosed the dermatologists.

Accurate diagnosis of breast cancer metastases in axillary lymph node biopsies is vital in determining treatment. In a simulation challenge, 32 ML algorithms were trained on data from whole‐slide images with and without metastases, as determined by immunohistochemical staining.9 When applied to an independent test set of 129 whole‐slide images (49 with and 80 without metastases) and performance compared with that of 11 pathologists examining the same slides under a time constraint‐simulating routine pathology workflow, the best performing algorithm outperformed the pathologists (AUC, 0.994 v mean, 0.810 [range, 0.738–0.884]).9

Echocardiography requires considerable expert operator time in measurement and interpretation, limiting its use in primary care and rural settings — limitations which ML may overcome. An automated deep learning system was trained on 14 035 echocardiographs to quantify cardiac chamber volumes, left ventricular mass and ejection fraction, and detect hypertrophic cardiomyopathy, cardiac amyloid and pulmonary arterial hypertension.10 When applied to 8666 echocardiograms performed by ultrasonographers during routine workflows using commercial software, automated measurements were comparable or superior across 11 internal consistency metrics, and hypertrophic cardiomyopathy, cardiac amyloid and pulmonary arterial hypertension were identified with AUCs of 0.93, 0.87 and 0.85, respectively.10

Risk prediction

Current prediction tools may lack generalisability by virtue of a limited set of preselected variables judged to be clinically relevant. In contrast, ML can utilise many more variables available through electronic health records and may better predict patient trajectories across diverse populations. These outputs could inform models of care, resource allocation and targeting of care to high risk patients.

Electronic health record data from 194 470 admissions across two US hospitals were used to train ML algorithms to predict in‐hospital mortality, long length of stay, discharge diagnoses 24 hours after admission, and 30‐day unplanned readmission risk at discharge.11 When tested on another 21 751 admissions across both hospitals and compared with traditional prediction tools, algorithms more accurately predicted in‐hospital mortality (AUC, 0.93–0.95 v 0.85–0.86), long length of stay (AUC, 0.85–0.86 v 0.74–0.77) and readmission risk (AUC, 0.76–0.77 v 0.68–0.70), and classified all discharge diagnosis codes with weighted AUCs of 0.86 and 0.87.11

Regarding specific conditions, identifying high risk septic patients may guide resuscitation and treatment efforts. In a retrospective study, an ML algorithm trained on 4222 admissions using over 500 clinical variables was compared with five commonly used regression‐based prediction tools in a test set of 1056 admissions. The algorithm outperformed all other tools with an AUC of 0.86 versus a range of 0.69–0.76.12 Predicting in‐hospital survival among patients with major head trauma can help decide indications for neurosurgery and rehabilitation. Using data from 7769 patients with computed tomography head scans showing brain damage, five ML algorithms trained on 11 input variables had performance on 100 novel patients compared with that of four traditional prediction models and the opinions of ten neurosurgeons. Algorithms were more accurate, more sensitive, as specific and more discriminating than both traditional models and clinicians, with a mean AUC of 0.86 versus 0.77 and 0.74, respectively.13

Other ML applications are in development (Box 3).14,15,16,17,18 Australian researchers are actively engaging in ML,19,20 with the potential to do so at scale because every citizen has a unique Medicare identifier, and large digitised data repositories are evolving with state‐wide digital hospital systems and My Health Records, all protected by strong privacy legislation.

Limitations and challenges

Dependence on data quality

Data that are incomplete, incorrect (including wrong diagnoses), poorly described or labelled, inadequately structured (semantically or temporally), obsolescent, unrepresentative of diseases or populations of interest, or of low volume will introduce error in ML training. Errors can reflect random omissions or misclassifications, or more importantly, systematic biases in data collection, for example, regarding race, ethnicity, language, socio‐economic status and sexual preference.

Poorly constructed algorithms can hinder decision making

Inner workings of ML algorithms can be opaque and uninterpretable to clinicians, and poorly constructed or improperly used algorithms can impair decision making. For example, an ML model predicting survival of post‐menopausal women performed worse than conventional Framingham scores, partly because it lacked training information on key blood investigations.21 Another ML model falsely classified asthmatic patients with pneumonia as being low risk and eligible for early hospital discharge, because training data did not capture life‐saving admissions of many patients to intensive care units.22 Algorithms have been developed to predict optimal treatments for various cancers, but after 4 years, none have demonstrated superior performance to oncologists, with systems still struggling with simply identifying cancer types.23

Data relating to clinical features irrelevant to outcomes of interest (eg, eye colour and cholesterol levels in predicting antibiotic response in appendicitis) can render algorithms excessively complex and diminish predictive accuracy. Clinical experts working alongside data experts must identify the most pertinent features.

Need for reference standards

For algorithms developed using supervised learning, a robust reference standard must be included in both training and testing datasets for each diagnosis (gold standard testing or assessment protocols) or outcome (appropriately specified, verifiable event or outcome criteria).

Insensitivity to context and timing of events

In defining predictor variables or associations between variables, algorithms may be insensitive to contextual factors, such as local clinician preferences, care standards or admission policies. Temporal variations in variables or sequences of clinical decisions are other confounders. For example, severely ill septic patients may appropriately receive fluids earlier than healthier patients, yet be more likely to die. ML correctly associates earlier fluid administration with higher mortality, which may be misinterpreted as contributing to death. Similarly, ML can perpetuate prior patterns of poor decisions and errors recorded in electronic health records or coded data on which ML is trained. These examples underscore the need for expert clinician interpretation of ML outputs.

Lack of defined evidence standards

Standards for assessing safety and utility of ML applications are currently not well defined. Recent United Kingdom guidance24 recommends randomised trials for ML algorithms that directly affect patient care, and these are starting to appear.25 Replication studies which test validity, reproducibility and generalisability are also needed.26

Lack of impact studies

The value of ML is gauged by increased diagnostic accuracy and therapeutic effectiveness, decreased time on routine tasks, faster turnaround of investigation results, reduced costs of care, and better patient outcomes. Clinical impact studies and cost–benefit analyses of ML in routine care are mostly lacking. Implementing ML algorithms in routine clinical workflows requires making interfaces accessible within clinical information systems, avoiding both alert fatigue and unquestioning acceptance of ML predictions, determining liability if patient harm ensues, and ensuring privacy of patient and practitioner data.

Conclusion

While ML will likely disrupt clinical practice over coming decades, particularly imaging‐based disciplines, it requires judicious application. ML can provide better, more patient‐specific information, affording clinicians greater capacity to make the most appropriate clinical decisions in partnership with their patients.

Box 1 – Relationship between artificial intelligence, machine learning and deep learning

---

Artificial intelligence describes any computer method that mimics human reasoning capabilities, including pattern recognition, abstract reasoning and planning. Machine learning is one of several subdomains of artificial intelligence and comprises a family of methods including neural networks, information theory and probabilistic approaches to learn new knowledge from past examples. Deep learning is one of many approaches to machine learning and involves training multilayered neural networks on large datasets to learn patterns to perform tasks such as speech and image recognition. ◆