Bias enters an AI system through choices about data, labels and the question the model answers, long before anyone measures performance. Finding likely sources at the design stage costs less than correcting a trained model.
Figure 1:The Turing Way Community. This illustration is created by Scriberia with The Turing Way community, used under a CC-BY 4.0 licence. DOI: 10.5281/zenodo.3332807.
Types of bias in healthcare AI¶
Historical bias¶
Medical records reflect how care was given in the past. If a condition was under-diagnosed in women, a model trained on those records learns to under-diagnose it too. Because the bias sits in the labels, the model can look accurate when you test it against the same kind of records.
Representation bias¶
Some groups are under-represented in most datasets: ethnic minorities, older patients, patients with disabilities, patients from low-income areas and people with rare variants of a disease. A model sees fewer examples from these groups and often performs worse for them.
Measurement bias¶
Equipment, protocols and staff produce different data. A model trained mostly on high-field MRI scans may do poorly on low-field scans because of the measurement, not the patients. In EHR data, the same applies to how often and how carefully observations are recorded.
Labelling bias¶
Labels reflect the knowledge and habits of the people who make them. If all annotators come from one hospital or one specialty, their labels may not match practice elsewhere.
Proxy bias¶
A model can pick up a stand-in for a sensitive attribute, such as postcode standing in for ethnicity or income, or an image artefact standing in for the type of scanner. It can then discriminate even though the attribute itself was never an input.
Identifying bias sources in your project¶
Work through these questions at design time, with clinicians and patients in the room.
| Area | Questions |
|---|---|
| Data | Which groups are likely to be under-represented? Will differences in equipment, protocols or sites cause systematic differences in performance? Are there trends over time that could cause distribution shift (a change in the data the model sees after training)? |
| Labels | Who annotated the data, and do they reflect the range of clinical practice? How well do annotators agree? Is the reference standard itself shaped by past bias? |
| Problem framing | Do any inputs correlate with sensitive attributes? Could the model use proxies? Is the outcome you predict a fair measure for all groups? (Predicting healthcare use, for example, can reflect access to care rather than need.) |
Attributes to monitor for bias¶
Decide which attributes you need to check performance across groups. Common ones are age, sex and gender, ethnicity, socioeconomic status and disability. Some of these are protected grounds under equal-treatment law. Others are not protected themselves but can act as proxies: geographic location, for instance, can stand in for ethnicity, income or access to care, and is worth recording for that reason.
Record these attributes where you can, so that bias can be measured in evaluation and monitored after deployment.
- European Parliament and Council. (2024). Regulation (EU) 2024/1689 of the European Parliament and of the Council on artificial intelligence. Official Journal of the European Union.