A data strategy says which data you will use, where it comes from, who labels it and how you will make it represent the patients the tool will serve. Gaps found during development are much harder to fix than gaps found on paper.
Figure 1:The Turing Way Community. This illustration is created by Scriberia with The Turing Way community, used under a CC-BY 4.0 licence. DOI: 10.5281/zenodo.3332807.
The representative data problem¶
A model trained on data that leaves out some groups of patients will perform worse for those groups. In healthcare this is common. Clinical datasets over-represent patients from high-income countries, from university hospitals and from majority ethnic groups. Differences in equipment and protocols between sites add technical variation that a model may not handle. Historical records also reflect past clinical practice, including its biases.
This is a fairness problem as well as a technical one: a tool that works well for some patients and poorly for others can widen existing gaps in health outcomes.
Planning for data heterogeneity¶
Write down the sources of variation that apply to your use case. The four groups below come up in most projects.
| Source | Examples |
|---|---|
| Equipment | Manufacturer, model and software version; acquisition protocol (scanner settings, staining, lab assay); maintenance and calibration |
| Operators | Differences between annotators; differences in clinical practice between sites; experience of the staff who record or measure |
| Patients | Age, sex, ethnicity, socioeconomic status; disease subtype, severity and other conditions; regional differences in how common the disease is |
| Context | Emergency versus planned care; urban versus rural; resources available; time of day and staffing |
Multi-site data collection¶
Data from a single site often gives a model that performs worse when tested elsewhere. Where you can, collect data from more than one site, using shared collection protocols and the same annotation guidelines. If you cannot share patient data between sites, federated learning (training a model at each site and combining only the model updates) is one option.
Even if you develop on data from one hospital, plan now for an external evaluation on data from another site. Arranging the data-sharing agreement and the legal basis takes months.
Annotation and labelling¶
What the model learns depends on the labels. Decide who annotates: for clinical tasks, clinicians with the right expertise. Write annotation guidelines that say how to handle ambiguous cases. Measure agreement between annotators and record where they disagree, because high disagreement means uncertain labels. For some tasks, define how disagreements are resolved, for example by consensus or by an expert adjudicator. And accept that some reference standards are themselves uncertain: sepsis, for instance, has no single test, so the label depends on a definition you must choose and justify.
Data minimisation¶
Collect only the data you need for the stated purpose. The GDPR European Parliament and Council (2016) requires this (Art. 5(1)(c)). Extra variables also bring extra work: each one needs cleaning, a legal basis and a check for bias. Ask for each variable which question it answers.
- European Parliament and Council. (2016). Regulation (EU) 2016/679 – General Data Protection Regulation. Official Journal of the European Union.