A systematic review of 86 deep learning algorithms for radiological diagnosis found that 70 of them performed worse on external data than on the data from their own development site, some of them substantially worse Yu et al. (2022). External validation tells you whether your model’s performance carries over to data it has never seen, from people and places it was not built on. It does not show that patients benefit; that needs a prospective evaluation in practice (see clinical utility and safety).
Why external validation matters¶
Models pick up whatever patterns separate the outcomes in their training data, including patterns specific to one hospital: how often nurses record vital signs, which lab analysers are used, how diagnoses are coded, which patients are admitted. The model then does well on internal test data from the same hospital and worse elsewhere. Statisticians call one form of this covariate shift: the patients or measurements at the new site differ from those in the training data, so the model meets inputs it has seen less often.
Types of external validation¶
| Type | What you test on | What it shows |
|---|---|---|
| Temporal | Data from the same site, collected after the training period | Whether performance holds over time |
| Geographic | Data from another institution in the same country | Whether performance transfers to another hospital |
| Cross-national | Data from another country | Whether performance holds in a different health system and population |
| Cross-device | Data from other equipment, such as other scanners or lab analysers | Whether performance depends on how the data were measured |
Combining several types gives stronger evidence than any one alone.
Factors affecting external validity¶
Document and investigate what may cause performance to differ between sites. Population differences include how common the disease is, age and comorbidity. Measurement differences include equipment, protocols and how often observations are taken. Workflow differs too: who records the data, when, and under what clinical conditions. Finally, labels can differ. If one hospital defines sepsis by its own coding practice and another by a chart review, the model is being judged against two different reference standards.
Multi-site validation design¶
When you design a study across several sites:
include sites that cover the range of settings where the AI will be used;
include smaller or less well-resourced hospitals if the tool is meant for wide use;
analyse each site separately to find site-specific patterns;
report both pooled and per-site performance.
Local clinical validity¶
Before go-live at a new site, test the AI on a representative sample of that site’s own patients, even if it has passed multi-site validation. This local check confirms that performance is acceptable for the local population, identifies local factors (equipment, workflow, case mix) that need adaptation, and gives you the baseline you will compare against when you monitor the tool after deployment. A silent-mode run, where the model scores live patients without showing the result to clinicians, is one practical way to do this.
- Lekadir, K., Frangi, A. F., Porras, A. R., & others. (2025). FUTURE-AI: international consensus guideline for trustworthy and deployable artificial intelligence in healthcare. BMJ, 388, e081554. 10.1136/bmj-2024-081554
- Yu, A. C., Mohajer, B., & Eng, J. (2022). External validation of deep learning algorithms for radiologic diagnosis: a systematic review. Radiology: Artificial Intelligence, 4(3), e210064. 10.1148/ryai.210064