The chapters of The ELSA Way explain what responsible AI in healthcare asks of you. The Toolbox collects the practical tools that help you do it: workshop methods, serious games, impact assessments, reporting guidelines and software libraries.
You will also meet these tools in the chapters themselves. Wherever a tool helps with a specific step, a Tools for this step box appears in that section. Each tool has its own page with practical details: how long it takes, whether you need a facilitator, what it costs, its licence, and where else in the book it is useful.
Tools by lifecycle phase¶
Design¶
| Tool | What it helps you do | Type | Facilitation | Cost |
|---|---|---|---|---|
| Aanpak Begeleidingsethiek | A facilitated workshop method in which the people affected by a new technology map its effects and values together and agree on concrete actions for the technology, its setting and its users. | Participatory method | Facilitator required | Free |
| AI Impact Assessment | A question set that walks a multidisciplinary team through the purpose, impact, technical design, data and risks of an AI system, and records the decision whether to use it. | Assessment | Facilitator recommended | Free |
| Algoritmekader | A Dutch government website that brings together the legal requirements, recommended measures and supporting tools for responsible use of algorithms and AI, organised by lifecycle phase, role and topic. | Framework | No facilitator needed | Free |
| ALTAI | A self-assessment checklist from the EU High-Level Expert Group on AI that helps you test an AI system against the seven requirements for trustworthy AI and spot where you need extra measures. | Assessment | Facilitator recommended | Free |
| ARX | Free desktop software and Java library for anonymising structured personal data, with built-in checks of re-identification risk and of how much useful information remains. | Software library | No facilitator needed | Free |
| Data Ethics Canvas | A one-page canvas of questions that helps you spot and manage the ethical issues of a project that collects, shares or uses data, at the start and as the project evolves. | Canvas | No facilitator needed | Free |
| Datasheets for Datasets | A set of questions you answer about a dataset, from why it was made to how it is maintained, so others can judge whether it suits their purpose. | Reporting guideline | No facilitator needed | Free |
| DEDA | A poster and handbook that walk a project team through the ethical and practical questions of a data or AI project and help you document the choices you make. | Canvas | Facilitator recommended | Free |
| Ethical Matrix | A simple grid that makes you check, for every group affected by a technology, how it touches their wellbeing, autonomy and fairness, so you can weigh the concerns openly. | Framework | Facilitator recommended | Free |
| GRIPP2 | A reporting checklist that helps you describe how patients and the public were involved in your research, what they did and what difference it made. | Reporting guideline | No facilitator needed | Free |
| IAMA | A structured discussion guide that helps a public organisation decide whether and how to use an algorithm, with a step-by-step check of its effects on fundamental rights. | Assessment | Facilitator recommended | Free |
| Moral Medtech | A cooperative serious game in which patients, clinicians, developers and regulators agree which values a medical AI tool must protect and turn them into design requirements. | Serious game | Facilitator recommended | Ask the developers |
| Value Sensitive Design | A design approach, with ready-made methods and card decks, for identifying the people and human values a technology affects and building those values into the design from the start. | Framework | Facilitator recommended | Free |
Development¶
| Tool | What it helps you do | Type | Facilitation | Cost |
|---|---|---|---|---|
| AI Fairness 360 | An open-source Python and R toolkit for measuring bias in datasets and models and for trying out algorithms that reduce it before, during or after training. | Software library | No facilitator needed | Free |
| AI Impact Assessment | A question set that walks a multidisciplinary team through the purpose, impact, technical design, data and risks of an AI system, and records the decision whether to use it. | Assessment | Facilitator recommended | Free |
| Algoritmekader | A Dutch government website that brings together the legal requirements, recommended measures and supporting tools for responsible use of algorithms and AI, organised by lifecycle phase, role and topic. | Framework | No facilitator needed | Free |
| ALTAI | A self-assessment checklist from the EU High-Level Expert Group on AI that helps you test an AI system against the seven requirements for trustworthy AI and spot where you need extra measures. | Assessment | Facilitator recommended | Free |
| ARX | Free desktop software and Java library for anonymising structured personal data, with built-in checks of re-identification risk and of how much useful information remains. | Software library | No facilitator needed | Free |
| CodeCarbon | An open-source Python package that estimates the electricity use and carbon emissions of your code while it runs, for measuring and reporting the footprint of training and running models. | Software library | No facilitator needed | Free |
| Data Ethics Canvas | A one-page canvas of questions that helps you spot and manage the ethical issues of a project that collects, shares or uses data, at the start and as the project evolves. | Canvas | No facilitator needed | Free |
| Datasheets for Datasets | A set of questions you answer about a dataset, from why it was made to how it is maintained, so others can judge whether it suits their purpose. | Reporting guideline | No facilitator needed | Free |
| DEDA | A poster and handbook that walk a project team through the ethical and practical questions of a data or AI project and help you document the choices you make. | Canvas | Facilitator recommended | Free |
| Fairlearn | An open-source Python library that breaks model performance down by subgroup and offers algorithms to reduce the gaps you find. | Software library | No facilitator needed | Free |
| Flower | An open-source Python framework for federated learning: you train one model across several sites while patient data stays where it is. | Software library | No facilitator needed | Free |
| MLflow | An open-source platform that records what went into each model run and keeps a versioned registry of trained models, which lets you trace any deployed model back to its code, data and settings. | Software library | No facilitator needed | Free |
| Model Cards | A short, structured document that travels with a trained model and tells readers what it is for, how it was tested and where it performs worse. | Reporting guideline | No facilitator needed | Free |
| Moral Medtech | A cooperative serious game in which patients, clinicians, developers and regulators agree which values a medical AI tool must protect and turn them into design requirements. | Serious game | Facilitator recommended | Ask the developers |
| SHAP | A Python library that explains individual predictions by estimating how much each input feature pushed the model’s output up or down, based on Shapley values from game theory. | Software library | No facilitator needed | Free |
| TRIPOD+AI | A 27-item checklist for writing up studies that develop or validate a clinical prediction model, whether it uses regression or machine learning. | Reporting guideline | No facilitator needed | Free |
| Value Sensitive Design | A design approach, with ready-made methods and card decks, for identifying the people and human values a technology affects and building those values into the design from the start. | Framework | Facilitator recommended | Free |
Evaluation¶
| Tool | What it helps you do | Type | Facilitation | Cost |
|---|---|---|---|---|
| AI Fairness 360 | An open-source Python and R toolkit for measuring bias in datasets and models and for trying out algorithms that reduce it before, during or after training. | Software library | No facilitator needed | Free |
| Algoritmekader | A Dutch government website that brings together the legal requirements, recommended measures and supporting tools for responsible use of algorithms and AI, organised by lifecycle phase, role and topic. | Framework | No facilitator needed | Free |
| CONSORT-AI | An extension of the CONSORT checklist that tells you what to report when a randomised trial tests an intervention with an AI component. | Reporting guideline | No facilitator needed | Free |
| DECIDE-AI | A reporting checklist for small, early clinical studies of AI decision support, where you first test a tool with real users before a large trial. | Reporting guideline | No facilitator needed | Free |
| Evidently | An open-source Python library for checking data quality, data drift and model performance, which you can run as one-off reports or as an ongoing monitoring dashboard. | Software library | No facilitator needed | Free |
| Fairlearn | An open-source Python library that breaks model performance down by subgroup and offers algorithms to reduce the gaps you find. | Software library | No facilitator needed | Free |
| GRIPP2 | A reporting checklist that helps you describe how patients and the public were involved in your research, what they did and what difference it made. | Reporting guideline | No facilitator needed | Free |
| Model Cards | A short, structured document that travels with a trained model and tells readers what it is for, how it was tested and where it performs worse. | Reporting guideline | No facilitator needed | Free |
| PROBAST+AI | A structured tool for judging the quality, risk of bias and applicability of studies that develop or test a clinical prediction model, including AI models. | Assessment | No facilitator needed | Free |
| SHAP | A Python library that explains individual predictions by estimating how much each input feature pushed the model’s output up or down, based on Shapley values from game theory. | Software library | No facilitator needed | Free |
| SPIRIT-AI | An extension of the SPIRIT checklist that tells you what to put in a clinical trial protocol when the intervention includes AI. | Reporting guideline | No facilitator needed | Free |
| System Usability Scale | A ten-item questionnaire that produces a single score from 0 to 100 for how usable people find a system. | Questionnaire | No facilitator needed | Free |
| TRIPOD+AI | A 27-item checklist for writing up studies that develop or validate a clinical prediction model, whether it uses regression or machine learning. | Reporting guideline | No facilitator needed | Free |
Deployment¶
| Tool | What it helps you do | Type | Facilitation | Cost |
|---|---|---|---|---|
| Aanpak Begeleidingsethiek | A facilitated workshop method in which the people affected by a new technology map its effects and values together and agree on concrete actions for the technology, its setting and its users. | Participatory method | Facilitator required | Free |
| AI Impact Assessment | A question set that walks a multidisciplinary team through the purpose, impact, technical design, data and risks of an AI system, and records the decision whether to use it. | Assessment | Facilitator recommended | Free |
| Algoritmekader | A Dutch government website that brings together the legal requirements, recommended measures and supporting tools for responsible use of algorithms and AI, organised by lifecycle phase, role and topic. | Framework | No facilitator needed | Free |
| ALTAI | A self-assessment checklist from the EU High-Level Expert Group on AI that helps you test an AI system against the seven requirements for trustworthy AI and spot where you need extra measures. | Assessment | Facilitator recommended | Free |
| CodeCarbon | An open-source Python package that estimates the electricity use and carbon emissions of your code while it runs, for measuring and reporting the footprint of training and running models. | Software library | No facilitator needed | Free |
| Data Ethics Canvas | A one-page canvas of questions that helps you spot and manage the ethical issues of a project that collects, shares or uses data, at the start and as the project evolves. | Canvas | No facilitator needed | Free |
| DECIDE-AI | A reporting checklist for small, early clinical studies of AI decision support, where you first test a tool with real users before a large trial. | Reporting guideline | No facilitator needed | Free |
| Evidently | An open-source Python library for checking data quality, data drift and model performance, which you can run as one-off reports or as an ongoing monitoring dashboard. | Software library | No facilitator needed | Free |
| IAMA | A structured discussion guide that helps a public organisation decide whether and how to use an algorithm, with a step-by-step check of its effects on fundamental rights. | Assessment | Facilitator recommended | Free |
| MLflow | An open-source platform that records what went into each model run and keeps a versioned registry of trained models, which lets you trace any deployed model back to its code, data and settings. | Software library | No facilitator needed | Free |
| Model Cards | A short, structured document that travels with a trained model and tells readers what it is for, how it was tested and where it performs worse. | Reporting guideline | No facilitator needed | Free |
| System Usability Scale | A ten-item questionnaire that produces a single score from 0 to 100 for how usable people find a system. | Questionnaire | No facilitator needed | Free |
Tools by FUTURE-AI principle¶
Each cell lists the tools that support a FUTURE-AI principle in a lifecycle phase.
Tools by type¶
Software library: AI Fairness 360, ARX, CodeCarbon, Evidently, Fairlearn, Flower, MLflow, SHAP
Assessment: AI Impact Assessment, ALTAI, IAMA, PROBAST+AI
Reporting guideline: CONSORT-AI, Datasheets for Datasets, DECIDE-AI, GRIPP2, Model Cards, SPIRIT-AI, TRIPOD+AI
Participatory method: Aanpak Begeleidingsethiek
Serious game: Moral Medtech
Canvas: Data Ethics Canvas, DEDA
Framework: Algoritmekader, Ethical Matrix, Value Sensitive Design
Questionnaire: System Usability Scale
Acknowledgments¶
The idea of a filterable collection of practical tools was inspired by the RAAIT toolbox of the Responsible Applied Artificial InTelligence programme. Tools that RAAIT also lists link to its pages.