Robust Intent Classification using Bayesian LSTM for Clinical Conversational Agents (CAs)

TY - GEN

T1 - Robust Intent Classification using Bayesian LSTM for Clinical Conversational Agents (CAs)

AU - Aftab, Haris

AU - Gautam, Vibhu

AU - Hawkins, Richard David

AU - Alexander, Rob

AU - Habli, Ibrahim

N1 - © 2022 ICST Institute for Computer Sciences, Social Informatics and Telecommunications Engineering. This is an author-produced version of the published paper. Uploaded in accordance with the publisher’s self-archiving policy. Further copying may not be permitted; contact the publisher for details

PY - 2022/6/7

Y1 - 2022/6/7

N2 - Conversational Agents (CAs) are software programs that replicate human conversations using machine learning (ML) and natural language processing (NLP). CAs are currently being utilised for diverse clinical applications such as symptom checking, health monitoring, medical triage and diagnosis. Intent classification (IC) is an essential task of understanding user utterance in CAs which makes use of modern deep learning (DL) methods. Because of the inherent model uncertainty associated with those methods, accuracy alone cannot be relied upon in clinical applications where certain errors may compromise patient safety. In this work, we employ Bayesian Long Short-Term Memory Networks (LSTMs) to calculate model uncertainty for IC, with a specific emphasis on symptom checker CAs. This method provides a certainty measure with IC prediction that can be utilised in assuring safe response from CAs. We evaluated our method on in-distribution (ID) and out-of-distribution (OOD) data and found mean uncer-tainty to be much higher for OOD data. These findings suggest that our method is robust to OOD utterances and can detect non-understanding errors in CAs.

AB - Conversational Agents (CAs) are software programs that replicate human conversations using machine learning (ML) and natural language processing (NLP). CAs are currently being utilised for diverse clinical applications such as symptom checking, health monitoring, medical triage and diagnosis. Intent classification (IC) is an essential task of understanding user utterance in CAs which makes use of modern deep learning (DL) methods. Because of the inherent model uncertainty associated with those methods, accuracy alone cannot be relied upon in clinical applications where certain errors may compromise patient safety. In this work, we employ Bayesian Long Short-Term Memory Networks (LSTMs) to calculate model uncertainty for IC, with a specific emphasis on symptom checker CAs. This method provides a certainty measure with IC prediction that can be utilised in assuring safe response from CAs. We evaluated our method on in-distribution (ID) and out-of-distribution (OOD) data and found mean uncer-tainty to be much higher for OOD data. These findings suggest that our method is robust to OOD utterances and can detect non-understanding errors in CAs.

KW - Conversational Agents (CAs)

KW - Machine Learning

KW - Model Uncertainty

KW - Out-of-Distribution (OOD)

KW - Healthcare

KW - Patient Safety

U2 - 10.1007/978-3-031-06368-8_8

DO - 10.1007/978-3-031-06368-8_8

M3 - Conference contribution

SN - 9783031063671

SP - 106

EP - 118

BT - MobiHealth 2021

PB - Springer

CY - Cham

T2 - 10th EAI International Conference on Wireless Mobile Communication and Healthcare

Y2 - 13 November 2021 through 14 November 2021

ER -