Addressing AI risks

The risk register from the design phase (risk management) lists what could go wrong. During development you build measures against those risks, and you add the new ones that appear once you work with real data and a real model.

Technical risk mitigation¶

Robustness to distribution shift¶

Distribution shift means that the data a model sees in use differs from the data it was trained on: a different hospital, new equipment, a changed protocol or a different patient mix. Models often perform worse after such a shift. Measures that help:

• Domain adaptation: adjust the model with data from the site where it will be used

• Domain generalisation: train on data from several different sites so the model depends less on the features of any one of them

• Data augmentation: simulate the variation you expect in use (see data collection and management)

• Transfer learning: start from a model already trained on large, varied data and adapt it to your task

Adversarial robustness¶

An adversarial attack changes an input in small, deliberate ways to make the model give a wrong answer. In imaging, such changes can be too small for a person to notice. Defences include training on such manipulated examples (adversarial training), checking inputs for values or patterns the model has not seen before, and using ensembles of models, which are harder to mislead than a single model.

Handling missing or corrupted data¶

Clinical data in daily use is often incomplete or wrong: an observation not yet entered, a value in the wrong unit, a device that sends nothing. The system should handle missing inputs without crashing or producing a silent error, flag implausible inputs for human review, and lose accuracy gradually rather than suddenly as input quality drops. Test this explicitly by removing or corrupting inputs in your validation data.

Baseline AI model¶

FUTURE-AI’s practical steps for G3 start with implementing a baseline AI model and identifying its limitations, such as bias or poor generalisation Lekadir et al. (2025). A baseline is a simple reference model, for example an existing clinical rule or score, a logistic regression, or a published model for the same task. It shows how much each later design choice adds, and in the evaluation phase it becomes one of the reference methods you compare against (G4). See evaluation planning.

Security measures¶

Three attacks are specific to AI systems. In model extraction, someone queries the model many times and uses the answers to build a copy. In data poisoning, someone tampers with training data so the model learns harmful behaviour. In input manipulation, someone alters inputs to change the output. Write a threat model: a short document that lists which attacks are realistic for your system, who could carry them out, and which measure addresses each. Combine it with the hospital’s NEN 7510 information security measures.