Clinical utility and safety

Clinical utility means that patients are better off when clinicians use the AI in real practice: better diagnoses, better treatment decisions, fewer harms or more efficient care. It is the most important question in evaluation and the hardest one to answer.

Why clinical utility differs from accuracy¶

A model can score well on a held-out test set and still fail to help, because:

• it changes no decisions that matter (clinicians already get the easy cases right);

• clinicians ignore or override it because it is hard to use;

• clinicians follow it when it is wrong (automation bias), introducing new errors;

• it improves outcomes for some patients and worsens them for others;

• it performs worse in the clinic than on the test set.

Retrospective and external validation show how well the model predicts. Only a prospective evaluation in clinical practice can show that patients benefit.

The clinical evaluation plan¶

Your clinical evaluation plan should state the primary clinical outcome (for sepsis, for example, time to antibiotics or unplanned ICU admission), the study design, the comparator (usually care without the AI), the patients included, the setting and the duration. For an early live evaluation, follow the DECIDE-AI guideline Vasey et al. (2022). For a randomised trial, use SPIRIT-AI for the protocol Cruz Rivera et al. (2020) and CONSORT-AI for the report Liu et al. (2020).

A study in which the AI’s output affects patient care may be research under the Dutch Medical Research Involving Human Subjects Act (WMO), which requires review by an accredited medical research ethics committee (METC). Ask your METC early whether your design falls under the WMO.

Levels of evidence for clinical AI¶

The table orders study designs from weakest to strongest evidence of benefit.

Steps 1 and 2 are needed but do not show benefit to patients. Claims of clinical utility need evidence from step 4 or higher. Where a randomised trial is not feasible, say why and describe how your design limits bias.

Assessing safety¶

Ask how this AI could harm a patient, and look for evidence of each route:

• adverse events, where an incorrect output led to harm;

• automation bias, where clinicians followed an incorrect output;

• near misses, where harm was narrowly avoided;

• failure modes, meaning systematic errors in particular patient groups, conditions or data situations.

Record and report every safety issue you find. If the tool is a medical device, serious incidents after it is placed on the market must be reported through MDR vigilance European Parliament and Council (2017).

Cost-effectiveness¶

Hospitals, insurers and health technology assessment bodies want to know whether the benefit justifies the cost. Relevant inputs include clinician time saved or added (for example, time spent responding to false alerts), unnecessary tests or treatments avoided, the value of improved outcomes from health economic modelling, and the costs of infrastructure, licences, maintenance and monitoring. Collect these data during your prospective studies, because you cannot reconstruct most of them afterwards.