A model that did well at several external sites can still underperform at yours. Local validation checks, before go-live, that the tool works acceptably for your patients, with your equipment and inside your workflow.
Why local validation matters¶
Each site differs in ways a model can notice. The local population may have a different disease prevalence, age profile or mix of conditions. Equipment differs too: other monitors, lab analysers or scanners, calibrated and set up in other ways. Workflow decides when measurements are taken and by whom, and local data systems differ in EHR format, coding conventions and how complete the records are.
If you find a shortfall before go-live, you can adapt the tool or the workflow. If you find it after go-live, patients may already have been harmed and clinicians will trust the tool less.
Conducting local validation¶
Select a sample of cases that represents your patients, including the range of ages, backgrounds and disease severity you expect.
Establish the reference standard (the “ground truth”): the best available judgement of the true outcome, such as expert review, pathology or the clinical outcome.
Run the AI on the sample and record its outputs, scores and any system flags.
Measure performance overall and for relevant subgroups.
Compare the results with the minimum thresholds you set before you started.
If performance falls short, find the cause and decide whether to adapt the tool before go-live.
Local adaptation¶
If local performance is not good enough, there are three main options. You can fine-tune the model, meaning you retrain part of it on local data; this risks overfitting (the model learning quirks of a small local dataset) and can count as a change that needs regulatory review. You can recalibrate it, so that its predicted risks match the rates actually observed in your patients (calibration means that, of the patients given a 20% risk, about 20% have the outcome). Or you can adjust the configuration, for example by choosing a different alert threshold that suits local prevalence and clinical priorities.
Record every adaptation and the reasons for it. A change to the model itself may need a new regulatory assessment (see Regulatory compliance).
- Lekadir, K., Frangi, A. F., Porras, A. R., & others. (2025). FUTURE-AI: international consensus guideline for trustworthy and deployable artificial intelligence in healthcare. BMJ, 388, e081554. 10.1136/bmj-2024-081554