Confidence Arguments for Evidence of Performance in Machine Learning for Highly Automated Driving Functions

Simon Burton; Lydia Gauerhof; Richard David Hawkins; Ibrahim Habli; Bibhuti  Sethy

doi:10.1007/978-3-030-26250-1_30

Confidence Arguments for Evidence of Performance in Machine Learning for Highly Automated Driving Functions

Simon Burton, Lydia Gauerhof, Richard David Hawkins, Ibrahim Habli, Bibhuti Sethy

Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Due to their ability to efficiently process unstructured and highly dimensional input data, machine learning algorithms are being applied to perception tasks for highly automated driving functions. The consequences of failures and insu_ciencies in such algorithms are severe and a convincing assurance case that the algorithms meet certain safety requirements is therefore required. However, the task of demonstrating the performance of such algorithms is non-trivial, and as yet, no consensus has formed regarding an appropriate set of verification measures. This paper provides a framework for reasoning about the contribution of performance evidence to the assurance case for machine learning in an automated driving context and applies the evaluation criteria to a pedestrian recognition case study.

Original language	English
Title of host publication	Computer Safety, Reliability, and Security
Subtitle of host publication	SAFECOMP 2019 Workshops, ASSURE, DECSoS, SASSUR, STRIVE, and WAISE, Turku, Finland, September 10, 2019, Proceedings
Publisher	Springer
Pages	365–377
ISBN (Electronic)	978-3-030-26249-5
DOIs	https://doi.org/10.1007/978-3-030-26250-1_30
Publication status	Published - 9 Aug 2019

Publication series

Name	Lecture Notes in Computer Science
Publisher	Springer
Volume	11699

Access to Document

10.1007/978-3-030-26250-1_30

WAISE2019Accepted author manuscript, 3.79 MB

Cite this

Burton, S., Gauerhof, L., Hawkins, R. D., Habli, I., & Sethy, B. (2019). Confidence Arguments for Evidence of Performance in Machine Learning for Highly Automated Driving Functions. In Computer Safety, Reliability, and Security: SAFECOMP 2019 Workshops, ASSURE, DECSoS, SASSUR, STRIVE, and WAISE, Turku, Finland, September 10, 2019, Proceedings (pp. 365–377). (Lecture Notes in Computer Science; Vol. 11699). Springer. https://doi.org/10.1007/978-3-030-26250-1_30

Burton, Simon ; Gauerhof, Lydia ; Hawkins, Richard David et al. / Confidence Arguments for Evidence of Performance in Machine Learning for Highly Automated Driving Functions. Computer Safety, Reliability, and Security: SAFECOMP 2019 Workshops, ASSURE, DECSoS, SASSUR, STRIVE, and WAISE, Turku, Finland, September 10, 2019, Proceedings. Springer, 2019. pp. 365–377 (Lecture Notes in Computer Science).

@inproceedings{dae0399a6b964aca80e716394c8ba172,

title = "Confidence Arguments for Evidence of Performance in Machine Learning for Highly Automated Driving Functions",

abstract = "Due to their ability to efficiently process unstructured and highly dimensional input data, machine learning algorithms are being applied to perception tasks for highly automated driving functions. The consequences of failures and insu_ciencies in such algorithms are severe and a convincing assurance case that the algorithms meet certain safety requirements is therefore required. However, the task of demonstrating the performance of such algorithms is non-trivial, and as yet, no consensus has formed regarding an appropriate set of verification measures. This paper provides a framework for reasoning about the contribution of performance evidence to the assurance case for machine learning in an automated driving context and applies the evaluation criteria to a pedestrian recognition case study.",

author = "Simon Burton and Lydia Gauerhof and Hawkins, {Richard David} and Ibrahim Habli and Bibhuti Sethy",

year = "2019",

month = aug,

day = "9",

doi = "10.1007/978-3-030-26250-1_30",

language = "English",

series = "Lecture Notes in Computer Science",

publisher = "Springer",

pages = "365–377",

booktitle = "Computer Safety, Reliability, and Security",

address = "Germany",

}

Burton, S, Gauerhof, L, Hawkins, RD , Habli, I & Sethy, B 2019, Confidence Arguments for Evidence of Performance in Machine Learning for Highly Automated Driving Functions. in Computer Safety, Reliability, and Security: SAFECOMP 2019 Workshops, ASSURE, DECSoS, SASSUR, STRIVE, and WAISE, Turku, Finland, September 10, 2019, Proceedings. Lecture Notes in Computer Science, vol. 11699, Springer, pp. 365–377. https://doi.org/10.1007/978-3-030-26250-1_30

Confidence Arguments for Evidence of Performance in Machine Learning for Highly Automated Driving Functions. / Burton, Simon; Gauerhof, Lydia; Hawkins, Richard David et al.
Computer Safety, Reliability, and Security: SAFECOMP 2019 Workshops, ASSURE, DECSoS, SASSUR, STRIVE, and WAISE, Turku, Finland, September 10, 2019, Proceedings. Springer, 2019. p. 365–377 (Lecture Notes in Computer Science; Vol. 11699).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Confidence Arguments for Evidence of Performance in Machine Learning for Highly Automated Driving Functions

AU - Burton, Simon

AU - Gauerhof, Lydia

AU - Hawkins, Richard David

AU - Habli, Ibrahim

AU - Sethy, Bibhuti

PY - 2019/8/9

Y1 - 2019/8/9

N2 - Due to their ability to efficiently process unstructured and highly dimensional input data, machine learning algorithms are being applied to perception tasks for highly automated driving functions. The consequences of failures and insu_ciencies in such algorithms are severe and a convincing assurance case that the algorithms meet certain safety requirements is therefore required. However, the task of demonstrating the performance of such algorithms is non-trivial, and as yet, no consensus has formed regarding an appropriate set of verification measures. This paper provides a framework for reasoning about the contribution of performance evidence to the assurance case for machine learning in an automated driving context and applies the evaluation criteria to a pedestrian recognition case study.

AB - Due to their ability to efficiently process unstructured and highly dimensional input data, machine learning algorithms are being applied to perception tasks for highly automated driving functions. The consequences of failures and insu_ciencies in such algorithms are severe and a convincing assurance case that the algorithms meet certain safety requirements is therefore required. However, the task of demonstrating the performance of such algorithms is non-trivial, and as yet, no consensus has formed regarding an appropriate set of verification measures. This paper provides a framework for reasoning about the contribution of performance evidence to the assurance case for machine learning in an automated driving context and applies the evaluation criteria to a pedestrian recognition case study.

U2 - 10.1007/978-3-030-26250-1_30

DO - 10.1007/978-3-030-26250-1_30

M3 - Conference contribution

T3 - Lecture Notes in Computer Science

SP - 365

EP - 377

BT - Computer Safety, Reliability, and Security

PB - Springer

ER -

Burton S, Gauerhof L, Hawkins RD , Habli I, Sethy B. Confidence Arguments for Evidence of Performance in Machine Learning for Highly Automated Driving Functions. In Computer Safety, Reliability, and Security: SAFECOMP 2019 Workshops, ASSURE, DECSoS, SASSUR, STRIVE, and WAISE, Turku, Finland, September 10, 2019, Proceedings. Springer. 2019. p. 365–377. (Lecture Notes in Computer Science). doi: 10.1007/978-3-030-26250-1_30