Write your evaluation plan before the evaluation starts, and before anyone opens the test data. If you choose metrics and analyses after seeing the results, you can, without meaning to, pick the ones that make the model look best. Readers then cannot tell which findings were planned and which were found by searching.
Pre-specification¶
Before you access the test data, write down:
Primary metric: the single measure that decides whether the evaluation succeeded.
Success threshold: the value of the primary metric that counts as success.
Secondary metrics: other measures you will report.
Comparator: what the AI is compared against, such as current practice or an existing clinical score.
Subgroups: the patient groups for which you will report performance separately.
Statistical analysis plan: how you will compute confidence intervals and any statistical tests.
Register the plan where you can, for example on OSF or ClinicalTrials.gov, so that others can compare it with what you report.
Selecting evaluation datasets¶
During development you used a validation set to choose between models and tune their settings. The test set is different: you keep it apart for the whole of development and use it once, for the final estimate. Never evaluate on data the model was trained or tuned on, and check for duplicates, such as the same admission appearing in both training and test data. Where you can, add data from another hospital (external validation) and from different periods, patient groups and recording practices.
Choosing evaluation metrics¶
Choose metrics that match the clinical task and the consequences of each type of error. A missed sepsis case costs more than an unnecessary bedside check, but too many unnecessary alerts lead to alert fatigue.
| Task type | Common metrics |
|---|---|
| Binary classification (disease yes or no) | Sensitivity, specificity, positive and negative predictive value, AUC |
| Multi-class classification | Sensitivity and specificity per class, averaged AUC |
| Segmentation (outlining structures in images) | Dice coefficient, Hausdorff distance |
| Risk prediction and survival | C-statistic, calibration, Brier score |
| Object detection in images | Average precision, intersection over union |
Some of these terms need a short explanation. The AUC (area under the receiver operating characteristic curve) is the probability that the model gives a higher score to a random patient with the condition than to a random patient without it; 0.5 is chance and 1.0 is perfect. The C-statistic is the same idea for risk and survival models. Calibration describes whether predicted risks match what happens: of all patients given a 20% risk, about 20% should develop the condition. The Brier score combines discrimination and calibration in one number. The Dice coefficient measures how much an outline drawn by the model overlaps with one drawn by an expert, from 0 (no overlap) to 1 (identical).
Report fairness metrics alongside these. Statistical parity difference is the rate of positive predictions in one group minus the rate in another. Equal opportunity asks for equal sensitivity (true positive rates) across groups, and equalised odds asks for equal true and false positive rates. The fairness chapter explains how to choose between them.
Reference methods¶
Compare the AI with something meaningful:
current practice without AI;
clinicians doing the same task, if you can measure that;
existing models or scores for the same task.
Beating a clinician on a retrospective dataset under controlled conditions does not show that patients will do better. The clinical utility chapter describes the prospective studies that answer that question.
- Lekadir, K., Frangi, A. F., Porras, A. R., & others. (2025). FUTURE-AI: international consensus guideline for trustworthy and deployable artificial intelligence in healthcare. BMJ, 388, e081554. 10.1136/bmj-2024-081554