The Role of Explainability in Assuring Safety of Machine Learning in Healthcare

Yan Jia; John Alexander McDermid; Tom Lawton; Ibrahim Habli

doi:10.1109/TETC.2022.3171314

The Role of Explainability in Assuring Safety of Machine Learning in Healthcare

Yan Jia, John Alexander McDermid, Tom Lawton, Ibrahim Habli

Research output: Contribution to journal › Article › peer-review

Abstract

Established approaches to assuring safety-critical systems and software are difficult to apply to systems employing ML where there is no clear, pre-defined specification against which to assess validity. This problem is exacerbated by the opaque nature of ML where the learnt model is not amenable to human scrutiny. XAI methods have been proposed to tackle this issue by producing human-interpretable representations of ML models which can help users to gain confidence and build trust in the ML system. However, little work explicitly investigates the role of explainability for safety assurance in the context of ML development. This paper identifies ways in which XAI methods can contribute to safety assurance of ML-based systems. It then uses a concrete ML-based clinical decision support system, concerning weaning of patients from mechanical ventilation, to demonstrate how XAI methods can be employed to produce evidence to support safety assurance. The results are also represented in a safety argument to show where, and in what way, XAI methods can contribute to a safety case. Overall, we conclude that XAI methods have a valuable role in safety assurance of ML-based systems in healthcare but that they are not sufficient in themselves to assure safety.

Original language	English
Number of pages	14
Journal	IEEE Transactions on Emerging Topics in Computational Intelligence
DOIs	https://doi.org/10.1109/TETC.2022.3171314
Publication status	E-pub ahead of print - 5 May 2022

Access to Document

10.1109/TETC.2022.3171314Licence: CC BY-NC-ND

The Role of Explainability in Assuring Safety of Machine Learning in HealthcareFinal published version, 1.89 MBLicence: CC BY-NC-ND

Cite this

@article{fdc24ae2f2d0494385e80b4e7b1dbc00,

title = "The Role of Explainability in Assuring Safety of Machine Learning in Healthcare",

abstract = "Established approaches to assuring safety-critical systems and software are difficult to apply to systems employing ML where there is no clear, pre-defined specification against which to assess validity. This problem is exacerbated by the opaque nature of ML where the learnt model is not amenable to human scrutiny. XAI methods have been proposed to tackle this issue by producing human-interpretable representations of ML models which can help users to gain confidence and build trust in the ML system. However, little work explicitly investigates the role of explainability for safety assurance in the context of ML development. This paper identifies ways in which XAI methods can contribute to safety assurance of ML-based systems. It then uses a concrete ML-based clinical decision support system, concerning weaning of patients from mechanical ventilation, to demonstrate how XAI methods can be employed to produce evidence to support safety assurance. The results are also represented in a safety argument to show where, and in what way, XAI methods can contribute to a safety case. Overall, we conclude that XAI methods have a valuable role in safety assurance of ML-based systems in healthcare but that they are not sufficient in themselves to assure safety.",

author = "Yan Jia and McDermid, {John Alexander} and Tom Lawton and Ibrahim Habli",

year = "2022",

month = may,

day = "5",

doi = "10.1109/TETC.2022.3171314",

language = "English",

journal = "IEEE Transactions on Emerging Topics in Computational Intelligence",

issn = "2471-285X",

publisher = "IEEE",

}

TY - JOUR

T1 - The Role of Explainability in Assuring Safety of Machine Learning in Healthcare

AU - Jia, Yan

AU - McDermid, John Alexander

AU - Lawton, Tom

AU - Habli, Ibrahim

PY - 2022/5/5

Y1 - 2022/5/5

N2 - Established approaches to assuring safety-critical systems and software are difficult to apply to systems employing ML where there is no clear, pre-defined specification against which to assess validity. This problem is exacerbated by the opaque nature of ML where the learnt model is not amenable to human scrutiny. XAI methods have been proposed to tackle this issue by producing human-interpretable representations of ML models which can help users to gain confidence and build trust in the ML system. However, little work explicitly investigates the role of explainability for safety assurance in the context of ML development. This paper identifies ways in which XAI methods can contribute to safety assurance of ML-based systems. It then uses a concrete ML-based clinical decision support system, concerning weaning of patients from mechanical ventilation, to demonstrate how XAI methods can be employed to produce evidence to support safety assurance. The results are also represented in a safety argument to show where, and in what way, XAI methods can contribute to a safety case. Overall, we conclude that XAI methods have a valuable role in safety assurance of ML-based systems in healthcare but that they are not sufficient in themselves to assure safety.

AB - Established approaches to assuring safety-critical systems and software are difficult to apply to systems employing ML where there is no clear, pre-defined specification against which to assess validity. This problem is exacerbated by the opaque nature of ML where the learnt model is not amenable to human scrutiny. XAI methods have been proposed to tackle this issue by producing human-interpretable representations of ML models which can help users to gain confidence and build trust in the ML system. However, little work explicitly investigates the role of explainability for safety assurance in the context of ML development. This paper identifies ways in which XAI methods can contribute to safety assurance of ML-based systems. It then uses a concrete ML-based clinical decision support system, concerning weaning of patients from mechanical ventilation, to demonstrate how XAI methods can be employed to produce evidence to support safety assurance. The results are also represented in a safety argument to show where, and in what way, XAI methods can contribute to a safety case. Overall, we conclude that XAI methods have a valuable role in safety assurance of ML-based systems in healthcare but that they are not sufficient in themselves to assure safety.

U2 - 10.1109/TETC.2022.3171314

DO - 10.1109/TETC.2022.3171314

M3 - Article

SN - 2471-285X

JO - IEEE Transactions on Emerging Topics in Computational Intelligence

JF - IEEE Transactions on Emerging Topics in Computational Intelligence

ER -