Classification of Failures in the Perception of Conversational Agents (CAs) and their Implications on Patient Safety

Haris Aftab; Syed  Hammad Hussain Shah; Ibrahim Habli

doi:10.3233/shti210253

Classification of Failures in the Perception of Conversational Agents (CAs) and their Implications on Patient Safety

Haris Aftab, Syed Hammad Hussain Shah, Ibrahim Habli

Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

The use of Conversational agents (CAs) in healthcare is an emerging field. These CAs seem to be effective in accomplishing administrative tasks, e.g. providing locations of care facilities and scheduling appointments. Modern CAs use machine learning (ML) to recognize, understand and generate a response. Given the criticality of many healthcare settings, ML and other component errors may result in CA failures and may cause adverse effects on patients. Therefore, in-depth assurance is required before the deployment of ML in critical clinical applications, e.g. management of medication dose or medical diagnosis. CA safety issues could arise due to diverse causes, e.g. related to user interactions, environmental factors and ML errors. In this paper, we classify failures of perception (recognition and understanding) of CAs and their sources. We also present a case study of a CA used for calculating insulin dose for gestational diabetes mellitus (GDM) patients. We then correlate identified perception failures of CAs to potential scenarios that might compromise patient safety.

Original language	English
Title of host publication	Public Health and Informatics
Subtitle of host publication	Proceedings of MIE 2021
Editors	John Mantas, Lăcrămioara Stoicu-Tivadar, Catherine Chronaki Chronaki, Arie Hasman, Patrick Weber, Parisis Gallos, Mihaela Crişan-Vida, Emmanouil Zoulias, Oana Sorina Chirila
Publisher	IOS Press
Pages	659-663
Number of pages	5
Volume	281
ISBN (Electronic)	9781643681856
ISBN (Print)	9781643681849
DOIs	https://doi.org/10.3233/shti210253
Publication status	Published - 29 May 2021

Publication series

Name	Studies in Health Technology and Informatics
Publisher	IOS Press
ISSN (Print)	0926-9630
ISSN (Electronic)	1879-8365

Bibliographical note

Keywords

conversational agents
Healthcare
Machine learning
PATIENT SAFETY

Access to Document

10.3233/shti210253Licence: CC BY-NC

Classification of Failures in the Perception of Conversational Agents (CAs) and Their Implications on Patient Safety
© 2021 European Federation for Medical Informatics (EFMI) and IOS Press
Final published version, 376 KBLicence: CC BY-NC

Cite this

Aftab, H., Hammad Hussain Shah, S., & Habli, I. (2021). Classification of Failures in the Perception of Conversational Agents (CAs) and their Implications on Patient Safety. In J. Mantas, L. Stoicu-Tivadar, C. C. Chronaki, A. Hasman, P. Weber, P. Gallos, M. Crişan-Vida, E. Zoulias, & O. Sorina Chirila (Eds.), Public Health and Informatics: Proceedings of MIE 2021 (Vol. 281, pp. 659-663). (Studies in Health Technology and Informatics). IOS Press. https://doi.org/10.3233/shti210253

Aftab, Haris ; Hammad Hussain Shah, Syed ; Habli, Ibrahim. / Classification of Failures in the Perception of Conversational Agents (CAs) and their Implications on Patient Safety. Public Health and Informatics: Proceedings of MIE 2021. editor / John Mantas ; Lăcrămioara Stoicu-Tivadar ; Catherine Chronaki Chronaki ; Arie Hasman ; Patrick Weber ; Parisis Gallos ; Mihaela Crişan-Vida ; Emmanouil Zoulias ; Oana Sorina Chirila. Vol. 281 IOS Press, 2021. pp. 659-663 (Studies in Health Technology and Informatics).

@inproceedings{934f0a1821e945bdace5f221095f2c94,

title = "Classification of Failures in the Perception of Conversational Agents (CAs) and their Implications on Patient Safety",

abstract = "The use of Conversational agents (CAs) in healthcare is an emerging field. These CAs seem to be effective in accomplishing administrative tasks, e.g. providing locations of care facilities and scheduling appointments. Modern CAs use machine learning (ML) to recognize, understand and generate a response. Given the criticality of many healthcare settings, ML and other component errors may result in CA failures and may cause adverse effects on patients. Therefore, in-depth assurance is required before the deployment of ML in critical clinical applications, e.g. management of medication dose or medical diagnosis. CA safety issues could arise due to diverse causes, e.g. related to user interactions, environmental factors and ML errors. In this paper, we classify failures of perception (recognition and understanding) of CAs and their sources. We also present a case study of a CA used for calculating insulin dose for gestational diabetes mellitus (GDM) patients. We then correlate identified perception failures of CAs to potential scenarios that might compromise patient safety.",

keywords = "conversational agents, Healthcare, Machine learning, PATIENT SAFETY",

author = "Haris Aftab and {Hammad Hussain Shah}, Syed and Ibrahim Habli",

note = "{\textcopyright} 2021 European Federation for Medical Informatics (EFMI) and IOS Press",

year = "2021",

month = may,

day = "29",

doi = "10.3233/shti210253",

language = "English",

isbn = "9781643681849",

volume = "281",

series = "Studies in Health Technology and Informatics",

publisher = "IOS Press",

pages = "659--663",

editor = "John Mantas and Stoicu-Tivadar, {L{\u a}cr{\u a}mioara } and Chronaki, {Catherine Chronaki} and Hasman, {Arie } and Weber, {Patrick } and Gallos, {Parisis } and Cri{\c s}an-Vida, {Mihaela } and Zoulias, {Emmanouil } and {Sorina Chirila}, {Oana }",

booktitle = "Public Health and Informatics",

address = "Netherlands",

}

Aftab, H, Hammad Hussain Shah, S & Habli, I 2021, Classification of Failures in the Perception of Conversational Agents (CAs) and their Implications on Patient Safety. in J Mantas, L Stoicu-Tivadar, CC Chronaki, A Hasman, P Weber, P Gallos, M Crişan-Vida, E Zoulias & O Sorina Chirila (eds), Public Health and Informatics: Proceedings of MIE 2021. vol. 281, Studies in Health Technology and Informatics, IOS Press, pp. 659-663. https://doi.org/10.3233/shti210253

Classification of Failures in the Perception of Conversational Agents (CAs) and their Implications on Patient Safety. / Aftab, Haris; Hammad Hussain Shah, Syed ; Habli, Ibrahim.
Public Health and Informatics: Proceedings of MIE 2021. ed. / John Mantas; Lăcrămioara Stoicu-Tivadar; Catherine Chronaki Chronaki; Arie Hasman; Patrick Weber; Parisis Gallos; Mihaela Crişan-Vida; Emmanouil Zoulias; Oana Sorina Chirila. Vol. 281 IOS Press, 2021. p. 659-663 (Studies in Health Technology and Informatics).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Classification of Failures in the Perception of Conversational Agents (CAs) and their Implications on Patient Safety

AU - Aftab, Haris

AU - Hammad Hussain Shah, Syed

AU - Habli, Ibrahim

PY - 2021/5/29

Y1 - 2021/5/29

N2 - The use of Conversational agents (CAs) in healthcare is an emerging field. These CAs seem to be effective in accomplishing administrative tasks, e.g. providing locations of care facilities and scheduling appointments. Modern CAs use machine learning (ML) to recognize, understand and generate a response. Given the criticality of many healthcare settings, ML and other component errors may result in CA failures and may cause adverse effects on patients. Therefore, in-depth assurance is required before the deployment of ML in critical clinical applications, e.g. management of medication dose or medical diagnosis. CA safety issues could arise due to diverse causes, e.g. related to user interactions, environmental factors and ML errors. In this paper, we classify failures of perception (recognition and understanding) of CAs and their sources. We also present a case study of a CA used for calculating insulin dose for gestational diabetes mellitus (GDM) patients. We then correlate identified perception failures of CAs to potential scenarios that might compromise patient safety.

AB - The use of Conversational agents (CAs) in healthcare is an emerging field. These CAs seem to be effective in accomplishing administrative tasks, e.g. providing locations of care facilities and scheduling appointments. Modern CAs use machine learning (ML) to recognize, understand and generate a response. Given the criticality of many healthcare settings, ML and other component errors may result in CA failures and may cause adverse effects on patients. Therefore, in-depth assurance is required before the deployment of ML in critical clinical applications, e.g. management of medication dose or medical diagnosis. CA safety issues could arise due to diverse causes, e.g. related to user interactions, environmental factors and ML errors. In this paper, we classify failures of perception (recognition and understanding) of CAs and their sources. We also present a case study of a CA used for calculating insulin dose for gestational diabetes mellitus (GDM) patients. We then correlate identified perception failures of CAs to potential scenarios that might compromise patient safety.

KW - conversational agents

KW - Healthcare

KW - Machine learning

KW - PATIENT SAFETY

U2 - 10.3233/shti210253

DO - 10.3233/shti210253

M3 - Conference contribution

SN - 9781643681849

VL - 281

T3 - Studies in Health Technology and Informatics

SP - 659

EP - 663

BT - Public Health and Informatics

A2 - Mantas, John

A2 - Stoicu-Tivadar, Lăcrămioara

A2 - Chronaki, Catherine Chronaki

A2 - Hasman, Arie

A2 - Weber, Patrick

A2 - Gallos, Parisis

A2 - Crişan-Vida, Mihaela

A2 - Zoulias, Emmanouil

A2 - Sorina Chirila, Oana

PB - IOS Press

ER -

Aftab H, Hammad Hussain Shah S, Habli I. Classification of Failures in the Perception of Conversational Agents (CAs) and their Implications on Patient Safety. In Mantas J, Stoicu-Tivadar L, Chronaki CC, Hasman A, Weber P, Gallos P, Crişan-Vida M, Zoulias E, Sorina Chirila O, editors, Public Health and Informatics: Proceedings of MIE 2021. Vol. 281. IOS Press. 2021. p. 659-663. (Studies in Health Technology and Informatics). doi: 10.3233/shti210253