Assurance of AI and autonomous technology in complex environments: a human-centered perspective

TY - GEN

T1 - Assurance of AI and autonomous technology in complex environments

T2 - Safety Critical Systems Symposium

AU - Preston, Kate Elizabeth

AU - Sujan, Mark Alexander

AU - Habli, Ibrahim

N1 - This is an author-produced version of the published paper. Uploaded in accordance with the University’s Research Publications and Open Access policy.

PY - 2025/2/6

Y1 - 2025/2/6

N2 - The development of AI and autonomous technologies is increasing rapidly across multiple sectors. The safety assurance of such systems often takes a narrow and technology-driven perspective, focusing on technical criteria such as overall performance metrics. However, these systems operate within broader human, technical, and organisational contexts. For example, an autonomous ship exists with-in a maritime system involving other functions and services including, among others, port and docking services, maintenance services, navigational services etc. Without consideration of these broader contexts, hazardous scenarios can arise, which may result from dysfunctional interactions between the autonomous functions and other elements of the maritime system. Effective safety assurance re-quires, therefore, consideration of the interactions between different elements of the work system, where the AI or autonomous system is just one such element. In this paper we put forward human-centered reflections based on an analysis of 22 demonstrator projects from diverse application domains.

AB - The development of AI and autonomous technologies is increasing rapidly across multiple sectors. The safety assurance of such systems often takes a narrow and technology-driven perspective, focusing on technical criteria such as overall performance metrics. However, these systems operate within broader human, technical, and organisational contexts. For example, an autonomous ship exists with-in a maritime system involving other functions and services including, among others, port and docking services, maintenance services, navigational services etc. Without consideration of these broader contexts, hazardous scenarios can arise, which may result from dysfunctional interactions between the autonomous functions and other elements of the maritime system. Effective safety assurance re-quires, therefore, consideration of the interactions between different elements of the work system, where the AI or autonomous system is just one such element. In this paper we put forward human-centered reflections based on an analysis of 22 demonstrator projects from diverse application domains.

UR - https://scsc.uk/e1099prog

M3 - Conference contribution

BT - Proceedings, Safety Critical Systems Symposium SSS'25

PB - Safety Critical Systems Club

Y2 - 4 February 2025 through 6 February 2025

ER -