SHAP

A Python library that explains individual predictions by estimating how much each input feature pushed the model’s output up or down, based on Shapley values from game theory.

What it is¶

SHAP assigns every feature a contribution to a single prediction so that the contributions add up to the difference between that prediction and the average one. You can use fast exact explainers for tree ensembles, or slower model-agnostic and deep learning explainers for other models. The library also produces plots, such as waterfall, beeswarm and dependence plots, that show both single-case explanations and global patterns across a dataset. Keep in mind that SHAP explains the model’s behaviour, not the causal effect of a feature on the patient.

In healthcare¶

SHAP has a published clinical use by its own authors: Lundberg and colleagues (2018) used it to explain real-time predictions of hypoxaemia risk during surgery to anaesthesiologists. It works best for tabular data such as lab values and vital signs; for medical images, saliency methods are often easier for clinicians to read.

Where to use it in The ELSA Way¶

• Explainability assessment › Types of explanation: SHAP produces the feature-importance type of explanation listed here; test its output with your clinical users to see whether it meets their needs.

• Human-AI interaction design › Designing AI outputs for clinical use: Per-patient SHAP values can supply the supporting evidence shown next to a model’s output, telling clinicians which inputs drove that prediction.

References¶

• Lundberg SM, Lee SI. A Unified Approach to Interpreting Model Predictions. Advances in Neural Information Processing Systems 30 (NIPS 2017). 2017. https://proceedings.neurips.cc/paper/2017/hash/8a20a8621978632d76c43dfd28b67767-Abstract.html

• Lundberg SM, Nair B, Vavilala MS, Horibe M, Eisses MJ, Adams T, et al. Explainable machine-learning predictions for the prevention of hypoxaemia during surgery. Nature Biomedical Engineering. 2018;2(10):749-760. Lundberg et al. (2018)

Related tools¶

• AI Fairness 360: An open-source Python and R toolkit for measuring bias in datasets and models and for trying out algorithms that reduce it before, during or after training.

• Fairlearn: An open-source Python library that breaks model performance down by subgroup and offers algorithms to reduce the gaps you find.

Also listed in: RAAIT toolbox.