Safety assurance of Machine Learning for autonomous systems

TY - JOUR

T1 - Safety assurance of Machine Learning for autonomous systems

AU - Paterson, Colin

AU - Hawkins, Richard David

AU - Picardi, Chiara

AU - Jia, Yan

AU - Calinescu, Radu

AU - Habli, Ibrahim

N1 - This is an author-produced version of the published paper. Uploaded in accordance with the University’s Research Publications and Open Access policy.

PY - 2025/12

Y1 - 2025/12

N2 - Machine Learning (ML) components are increasingly incorporated into systems, with different degrees of autonomy, where model performance is reported as meeting, or exceeding, the capabilities of human experts. This promises to transform products and services, in diverse domains such as healthcare, transport and manufacturing, to better serve underrepresented groups, reduce costs, and increase delivery effectiveness, especially where expert resources are scarce. The greatest potential for transformative impact lies in the development of autonomous systems for safety-critical applications where their acceptance, and subsequent deployment, is reliant on establishing justified confidence in system safety. Creating a compelling safety case for ML is challenging however, particularly since the ML development lifecycle is significantly different to that employed for traditional software systems. Typically ML development involves replacing detailed software specifications with representative data sets from which models of behaviour is learnt. Indeed, ML’s strength lies in tackling problems which are challenging for traditional software development practices. This shift in development practices introduces challenges to established assurance processes which are crucial to developing the compelling safety case required for regulation and societal acceptance. In this paper we introduce the first methodology for the Assurance of Machine Learning for use in Autonomous Systems (AMLAS). The AMLAS process describes how to systematically and attractively integrate safety assurance into the development of ML components and how to generate the evidence base for explicitly justifying the acceptable safety of these components when integrated into autonomous system applications. We describe the use of AMLAS by considering how a safety case may be constructed for an object detector for use in the perception pipeline of an autonomous driving application. We further discuss how AMLAS has been applied in several domains including healthcare, automotive and aerospace as well as supporting policy and industry guidance for defence, healthcare and automotive.

AB - Machine Learning (ML) components are increasingly incorporated into systems, with different degrees of autonomy, where model performance is reported as meeting, or exceeding, the capabilities of human experts. This promises to transform products and services, in diverse domains such as healthcare, transport and manufacturing, to better serve underrepresented groups, reduce costs, and increase delivery effectiveness, especially where expert resources are scarce. The greatest potential for transformative impact lies in the development of autonomous systems for safety-critical applications where their acceptance, and subsequent deployment, is reliant on establishing justified confidence in system safety. Creating a compelling safety case for ML is challenging however, particularly since the ML development lifecycle is significantly different to that employed for traditional software systems. Typically ML development involves replacing detailed software specifications with representative data sets from which models of behaviour is learnt. Indeed, ML’s strength lies in tackling problems which are challenging for traditional software development practices. This shift in development practices introduces challenges to established assurance processes which are crucial to developing the compelling safety case required for regulation and societal acceptance. In this paper we introduce the first methodology for the Assurance of Machine Learning for use in Autonomous Systems (AMLAS). The AMLAS process describes how to systematically and attractively integrate safety assurance into the development of ML components and how to generate the evidence base for explicitly justifying the acceptable safety of these components when integrated into autonomous system applications. We describe the use of AMLAS by considering how a safety case may be constructed for an object detector for use in the perception pipeline of an autonomous driving application. We further discuss how AMLAS has been applied in several domains including healthcare, automotive and aerospace as well as supporting policy and industry guidance for defence, healthcare and automotive.

KW - Machine Learning

KW - Safety

KW - Assurance

KW - Saafety Case

KW - Autonomy

KW - Artificial intelligence

UR - http://www.scopus.com/inward/record.url?scp=105008505755&partnerID=8YFLogxK

U2 - 10.1016/j.ress.2025.111311

DO - 10.1016/j.ress.2025.111311

M3 - Article

AN - SCOPUS:105008505755

SN - 0951-8320

VL - 264

JO - Reliability Engineering and System Safety

JF - Reliability Engineering and System Safety

IS - Part A

M1 - 111311

ER -