If a team publishes only its best metric, its most favourable subgroups or its successful sites, readers overestimate how well the model works. Hospitals then buy or deploy tools that fail, and patients bear the cost. Complete reporting lets others judge your results and lets the next team avoid your mistakes.
International reporting guidelines¶
Reporting guidelines are checklists that tell you what a paper or report on a given kind of study must contain. Several exist for healthcare AI:
| Guideline | Use it for |
|---|---|
| TRIPOD+AI Collins et al. (2024) | Studies that develop or validate prediction models, including those using machine learning |
| CLAIM Mongan et al. (2020), updated in 2024 Tejani et al. (2024) | AI studies in medical imaging |
| STARD-AI Sounderajah et al. (2025) | Diagnostic accuracy studies of AI |
| DECIDE-AI Vasey et al. (2022) | Early live clinical evaluation of AI decision support |
| SPIRIT-AI Cruz Rivera et al. (2020) | Protocols for randomised trials of AI interventions |
| CONSORT-AI Liu et al. (2020) | Reports of randomised trials of AI interventions |
Choose by study design. Several can apply at once: an imaging study that develops a prediction model may need both TRIPOD+AI and CLAIM. For the fictional sepsis model from this book’s running case, the development and external validation follow TRIPOD+AI, and the silent-mode and early live evaluation follow DECIDE-AI.
What to report¶
For study design and data, describe the training, validation and test datasets, their sizes (including per subgroup), the collection dates and sites, and the inclusion and exclusion criteria.
For the model, give enough detail on the architecture, training procedure, hyperparameters (settings chosen before training), model selection, software versions and computing environment for someone else to reproduce it.
For performance, report primary and secondary metrics with confidence intervals, calibration plots and measures, performance per pre-specified subgroup, and the comparison with your comparator.
For limitations, name known failure modes, populations in which the model has not been validated, data limitations such as missing attributes or historical bias, and limits on generalisability.
Negative results¶
Journals publish positive results more readily than negative ones, and AI research is no exception. A model that did not work as hoped, or an evaluation that exposed an unexpected failure, is still useful to others working on the same problem. Report negative results alongside positive ones. If a journal will not take them, consider journals that publish negative results or a preprint server.
Study registration¶
Register your evaluation study before you start, where you can:
clinical trials: ClinicalTrials.gov or another registry in the WHO network, such as ISRCTN;
observational studies and model evaluations: OSF or ISRCTN.
Registration makes it possible to tell planned (confirmatory) analyses from exploratory ones, and makes it harder to switch outcomes after seeing the results.
- Lekadir, K., Frangi, A. F., Porras, A. R., & others. (2025). FUTURE-AI: international consensus guideline for trustworthy and deployable artificial intelligence in healthcare. BMJ, 388, e081554. 10.1136/bmj-2024-081554
- Collins, G. S., Dhiman, P., & others. (2024). TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods. BMJ. 10.1136/bmj-2023-078378
- Mongan, J., Moy, L., & Kahn, C. E. (2020). Checklist for Artificial Intelligence in Medical Imaging (CLAIM): a guide for authors and reviewers. Radiology: Artificial Intelligence, 2(2), e200029. 10.1148/ryai.2020200029
- Tejani, A. S., Klontzas, M. E., Gatti, A. A., Mongan, J. T., Moy, L., Park, S. H., & Kahn, C. E. (2024). Checklist for Artificial Intelligence in Medical Imaging (CLAIM): 2024 update. Radiology: Artificial Intelligence, 6(4), e240300. 10.1148/ryai.240300
- Sounderajah, V., Guni, A., Liu, X., Collins, G. S., Karthikesalingam, A., Markar, S. R., & others. (2025). The STARD-AI reporting guideline for diagnostic accuracy studies using artificial intelligence. Nature Medicine, 31(10), 3283–3289. 10.1038/s41591-025-03953-8
- Vasey, B., Nagendran, M., Campbell, B., & others. (2022). Reporting guideline for the early-stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI. Nature Medicine, 28, 924–933. 10.1038/s41591-022-01772-9
- Cruz Rivera, S., Liu, X., Chan, A.-W., Denniston, A. K., & Calvert, M. J. (2020). Guidelines for clinical trial protocols for interventions involving artificial intelligence: the SPIRIT-AI extension. BMJ, 370, m3210. 10.1136/bmj.m3210
- Liu, X., Cruz Rivera, S., Moher, D., Calvert, M. J., & Denniston, A. K. (2020). Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension. BMJ, 370, m3164. 10.1136/bmj.m3164