Can you trust your Agent? The Effect of Out-of-Distribution Detection on the Safety of Reinforcement Learning Systems

TY - GEN

T1 - Can you trust your Agent? The Effect of Out-of-Distribution Detection on the Safety of Reinforcement Learning Systems

AU - Haider, Tom

AU - Roscher, Karsten

AU - Herd, Benjamin

AU - Schmoeller Roza, Felippe

AU - Burton, Simon

N1 - Publisher Copyright: © 2024 Copyright held by the owner/author(s).

PY - 2024/4/8

Y1 - 2024/4/8

N2 - Deep Reinforcement Learning (RL) has the potential to revolutionize the automation of complex sequential decision-making problems. Although it has been successfully applied to a wide range of tasks, deployment to real-world settings remains challenging and is often limited. One of the main reasons for this is the lack of safety guarantees for conventional RL algorithms, especially in situations that substantially differ from the learning environment. In such situations, state-of-the-art systems will fail silently, producing action sequences without signalizing any uncertainty regarding the current input. Recent works have suggested Out-of-Distribution (OOD) detection as an additional reliability measure when deploying RL in the real world. How these mechanisms benefit the safety of the entire system, however, is not yet fully understood. In this work, we study how OOD detection contributes to the safety of RL systems by describing the challenges involved with detecting unknown situations. We derive several definitions for unknown events and explore potential avenues for a successful safety argumentation, building on recent work for safety assurance of Machine Learning components. In a series of experiments, we compare different OOD detectors and show how difficult it is to distinguish harmless from potentially unsafe OOD events in practice, and how standard evaluation schemes can lead to deceptive conclusions, depending on which definition of unknown is applied.

AB - Deep Reinforcement Learning (RL) has the potential to revolutionize the automation of complex sequential decision-making problems. Although it has been successfully applied to a wide range of tasks, deployment to real-world settings remains challenging and is often limited. One of the main reasons for this is the lack of safety guarantees for conventional RL algorithms, especially in situations that substantially differ from the learning environment. In such situations, state-of-the-art systems will fail silently, producing action sequences without signalizing any uncertainty regarding the current input. Recent works have suggested Out-of-Distribution (OOD) detection as an additional reliability measure when deploying RL in the real world. How these mechanisms benefit the safety of the entire system, however, is not yet fully understood. In this work, we study how OOD detection contributes to the safety of RL systems by describing the challenges involved with detecting unknown situations. We derive several definitions for unknown events and explore potential avenues for a successful safety argumentation, building on recent work for safety assurance of Machine Learning components. In a series of experiments, we compare different OOD detectors and show how difficult it is to distinguish harmless from potentially unsafe OOD events in practice, and how standard evaluation schemes can lead to deceptive conclusions, depending on which definition of unknown is applied.

KW - AI safety

KW - anomaly detection

KW - OOD detection

KW - reinforcement learning

KW - sequential decision making

UR - http://www.scopus.com/inward/record.url?scp=85197685268&partnerID=8YFLogxK

U2 - 10.1145/3605098.3635931

DO - 10.1145/3605098.3635931

M3 - Conference contribution

AN - SCOPUS:85197685268

T3 - Proceedings of the ACM Symposium on Applied Computing

SP - 1569

EP - 1578

BT - 39th Annual ACM Symposium on Applied Computing, SAC 2024

PB - Association for Computing Machinery, Inc

T2 - 39th Annual ACM Symposium on Applied Computing, SAC 2024

Y2 - 8 April 2024 through 12 April 2024

ER -