Pedestrian Models for Autonomous Driving Part II: High-Level Models of Human Behavior

Fanta Camara; Nicola Bellotto; Serhan Cosar; Florian Weber; Dimitris Nathanael; Matthias Althoff; Jingyuan Wu; Johannes Ruenz; Andre Dietrich; Gustav Markkula; Anna Schieben; Fabio Tango; Natasha Merat; Charles Fox

doi:10.1109/TITS.2020.3006767

Pedestrian Models for Autonomous Driving Part II: High-Level Models of Human Behavior

Fanta Camara^*, Nicola Bellotto, Serhan Cosar, Florian Weber, Dimitris Nathanael, Matthias Althoff, Jingyuan Wu, Johannes Ruenz, Andre Dietrich, Gustav Markkula, Anna Schieben, Fabio Tango, Natasha Merat, Charles Fox

^*Corresponding author for this work

Institute for Safe Autonomy

Research output: Contribution to journal › Review article › peer-review

Abstract

Autonomous vehicles (AVs) must share space with pedestrians, both in carriageway cases such as cars at pedestrian crossings and off-carriageway cases such as delivery vehicles navigating through crowds on pedestrianized high-streets. Unlike static obstacles, pedestrians are active agents with complex, interactive motions. Planning AV actions in the presence of pedestrians thus requires modelling of their probable future behavior as well as detecting and tracking them. This narrative review article is Part II of a pair, together surveying the current technology stack involved in this process, organising recent research into a hierarchical taxonomy ranging from low-level image detection to high-level psychological models, from the perspective of an AV designer. This self-contained Part II covers the higher levels of this stack, consisting of models of pedestrian behavior, from prediction of individual pedestrians' likely destinations and paths, to game-theoretic models of interactions between pedestrians and autonomous vehicles. This survey clearly shows that, although there are good models for optimal walking behavior, high-level psychological and social modelling of pedestrian behavior still remains an open research question that requires many conceptual issues to be clarified. Early work has been done on descriptive and qualitative models of behavior, but much work is still needed to translate them into quantitative algorithms for practical AV control.

Original language	English
Article number	9151337
Pages (from-to)	5453-5472
Number of pages	20
Journal	IEEE Transactions on Intelligent Transportation Systems
Volume	22
Issue number	9
Early online date	28 Jul 2020
DOIs	https://doi.org/10.1109/TITS.2020.3006767
Publication status	Published - Sep 2021

Bibliographical note

Funding Information:
Manuscript received October 21, 2019; revised March 26, 2020; accepted April 9, 2020. This work was supported by the EU H2020 interACT under Grant 723395. The Associate Editor for this article was S. A. Birrell. (Corresponding author: Fanta Camara.) Fanta Camara is with the Institute for Transport Studies (ITS), University of Leeds, Leeds LS2 9JT, U.K., and also with the Lincoln Centre for Autonomous Systems, University of Lincoln, Lincoln LN6 7TS, U.K. (e-mail: tsfc@leeds.ac.uk). Nicola Bellotto is with the Lincoln Centre for Autonomous Systems, University of Lincoln, Lincoln LN6 7TS, U.K. Serhan Cosar is with the Institute of Engineering Sciences, De Montfort University, Leicester LE1 9BH, U.K. Florian Weber is with Bayerische Motoren Werke Aktiengesellschaft (BMW), 80809 Munich, Germany. Dimitris Nathanael is with the School of Mechanical Engineering, National Technical University of Athens, 15780 Athens, Greece. Matthias Althoff is with the Department of Computer Science, Technische Universität München, 85748 Garching bei München, Germany. Jingyuan Wu and Johannes Ruenz are with Robert Bosch GmbH, 74232 Abstatt, Germany. André Dietrich is with the Chair of Ergonomics, Technische Universität München (TUM), 85748 Garching bei München, Germany. Gustav Markkula and Natasha Merat are with the Institute for Transport Studies (ITS), University of Leeds, Leeds LS2 9JT, U.K. Anna Schieben is with the German Aerospace Center (DLR), 38108 Brunswick, Germany. Fabio Tango is with the Centro Ricerche Fiat (CRF), 10043 Orbassano, Italy. Charles Fox is with the Institute for Transport Studies (ITS), University of Leeds, Leeds LS2 9JT, U.K., also with the Lincoln Centre for Autonomous Systems, University of Lincoln, Lincoln LN6 7TS, U.K., and also with Ibex Automation Ltd., Sheffield S36 8YW, U.K. This article has supplementary downloadable material available at http://ieeexplore.ieee.org, provided by the authors. Digital Object Identifier 10.1109/TITS.2020.3006767

Publisher Copyright:
© 2000-2011 IEEE.

Keywords

autonomous vehicles
datasets
detection
eHMI
game-theoretic models
microscopic and macroscopic behavior models
pedestrian interaction
pedestrians
Review
sensing
signalling models
survey
tracking
trajectory prediction

Access to Document

10.1109/TITS.2020.3006767

Cite this

Camara, F., Bellotto, N., Cosar, S., Weber, F., Nathanael, D., Althoff, M., Wu, J., Ruenz, J., Dietrich, A., Markkula, G., Schieben, A., Tango, F., Merat, N., & Fox, C. (2021). Pedestrian Models for Autonomous Driving Part II: High-Level Models of Human Behavior. IEEE Transactions on Intelligent Transportation Systems, 22(9), 5453-5472. [9151337]. https://doi.org/10.1109/TITS.2020.3006767

@article{e617e86fa094480baed801fdcaa9352c,

title = "Pedestrian Models for Autonomous Driving Part II: High-Level Models of Human Behavior",

abstract = "Autonomous vehicles (AVs) must share space with pedestrians, both in carriageway cases such as cars at pedestrian crossings and off-carriageway cases such as delivery vehicles navigating through crowds on pedestrianized high-streets. Unlike static obstacles, pedestrians are active agents with complex, interactive motions. Planning AV actions in the presence of pedestrians thus requires modelling of their probable future behavior as well as detecting and tracking them. This narrative review article is Part II of a pair, together surveying the current technology stack involved in this process, organising recent research into a hierarchical taxonomy ranging from low-level image detection to high-level psychological models, from the perspective of an AV designer. This self-contained Part II covers the higher levels of this stack, consisting of models of pedestrian behavior, from prediction of individual pedestrians' likely destinations and paths, to game-theoretic models of interactions between pedestrians and autonomous vehicles. This survey clearly shows that, although there are good models for optimal walking behavior, high-level psychological and social modelling of pedestrian behavior still remains an open research question that requires many conceptual issues to be clarified. Early work has been done on descriptive and qualitative models of behavior, but much work is still needed to translate them into quantitative algorithms for practical AV control.",

keywords = "autonomous vehicles, datasets, detection, eHMI, game-theoretic models, microscopic and macroscopic behavior models, pedestrian interaction, pedestrians, Review, sensing, signalling models, survey, tracking, trajectory prediction",

author = "Fanta Camara and Nicola Bellotto and Serhan Cosar and Florian Weber and Dimitris Nathanael and Matthias Althoff and Jingyuan Wu and Johannes Ruenz and Andre Dietrich and Gustav Markkula and Anna Schieben and Fabio Tango and Natasha Merat and Charles Fox",

note = "Funding Information: Manuscript received October 21, 2019; revised March 26, 2020; accepted April 9, 2020. This work was supported by the EU H2020 interACT under Grant 723395. The Associate Editor for this article was S. A. Birrell. (Corresponding author: Fanta Camara.) Fanta Camara is with the Institute for Transport Studies (ITS), University of Leeds, Leeds LS2 9JT, U.K., and also with the Lincoln Centre for Autonomous Systems, University of Lincoln, Lincoln LN6 7TS, U.K. (e-mail: tsfc@leeds.ac.uk). Nicola Bellotto is with the Lincoln Centre for Autonomous Systems, University of Lincoln, Lincoln LN6 7TS, U.K. Serhan Cosar is with the Institute of Engineering Sciences, De Montfort University, Leicester LE1 9BH, U.K. Florian Weber is with Bayerische Motoren Werke Aktiengesellschaft (BMW), 80809 Munich, Germany. Dimitris Nathanael is with the School of Mechanical Engineering, National Technical University of Athens, 15780 Athens, Greece. Matthias Althoff is with the Department of Computer Science, Technische Universit{\"a}t M{\"u}nchen, 85748 Garching bei M{\"u}nchen, Germany. Jingyuan Wu and Johannes Ruenz are with Robert Bosch GmbH, 74232 Abstatt, Germany. Andr{\'e} Dietrich is with the Chair of Ergonomics, Technische Universit{\"a}t M{\"u}nchen (TUM), 85748 Garching bei M{\"u}nchen, Germany. Gustav Markkula and Natasha Merat are with the Institute for Transport Studies (ITS), University of Leeds, Leeds LS2 9JT, U.K. Anna Schieben is with the German Aerospace Center (DLR), 38108 Brunswick, Germany. Fabio Tango is with the Centro Ricerche Fiat (CRF), 10043 Orbassano, Italy. Charles Fox is with the Institute for Transport Studies (ITS), University of Leeds, Leeds LS2 9JT, U.K., also with the Lincoln Centre for Autonomous Systems, University of Lincoln, Lincoln LN6 7TS, U.K., and also with Ibex Automation Ltd., Sheffield S36 8YW, U.K. This article has supplementary downloadable material available at http://ieeexplore.ieee.org, provided by the authors. Digital Object Identifier 10.1109/TITS.2020.3006767 Publisher Copyright: {\textcopyright} 2000-2011 IEEE.",

year = "2021",

month = sep,

doi = "10.1109/TITS.2020.3006767",

language = "English",

volume = "22",

pages = "5453--5472",

journal = "IEEE Transactions on Intelligent Transportation Systems",

issn = "1524-9050",

publisher = "IEEE",

number = "9",

}

Camara, F, Bellotto, N, Cosar, S, Weber, F, Nathanael, D, Althoff, M, Wu, J, Ruenz, J, Dietrich, A, Markkula, G, Schieben, A, Tango, F, Merat, N & Fox, C 2021, 'Pedestrian Models for Autonomous Driving Part II: High-Level Models of Human Behavior', IEEE Transactions on Intelligent Transportation Systems, vol. 22, no. 9, 9151337, pp. 5453-5472. https://doi.org/10.1109/TITS.2020.3006767

TY - JOUR

T1 - Pedestrian Models for Autonomous Driving Part II

T2 - High-Level Models of Human Behavior

AU - Camara, Fanta

AU - Bellotto, Nicola

AU - Cosar, Serhan

AU - Weber, Florian

AU - Nathanael, Dimitris

AU - Althoff, Matthias

AU - Wu, Jingyuan

AU - Ruenz, Johannes

AU - Dietrich, Andre

AU - Markkula, Gustav

AU - Schieben, Anna

AU - Tango, Fabio

AU - Merat, Natasha

AU - Fox, Charles

N1 - Funding Information: Manuscript received October 21, 2019; revised March 26, 2020; accepted April 9, 2020. This work was supported by the EU H2020 interACT under Grant 723395. The Associate Editor for this article was S. A. Birrell. (Corresponding author: Fanta Camara.) Fanta Camara is with the Institute for Transport Studies (ITS), University of Leeds, Leeds LS2 9JT, U.K., and also with the Lincoln Centre for Autonomous Systems, University of Lincoln, Lincoln LN6 7TS, U.K. (e-mail: tsfc@leeds.ac.uk). Nicola Bellotto is with the Lincoln Centre for Autonomous Systems, University of Lincoln, Lincoln LN6 7TS, U.K. Serhan Cosar is with the Institute of Engineering Sciences, De Montfort University, Leicester LE1 9BH, U.K. Florian Weber is with Bayerische Motoren Werke Aktiengesellschaft (BMW), 80809 Munich, Germany. Dimitris Nathanael is with the School of Mechanical Engineering, National Technical University of Athens, 15780 Athens, Greece. Matthias Althoff is with the Department of Computer Science, Technische Universität München, 85748 Garching bei München, Germany. Jingyuan Wu and Johannes Ruenz are with Robert Bosch GmbH, 74232 Abstatt, Germany. André Dietrich is with the Chair of Ergonomics, Technische Universität München (TUM), 85748 Garching bei München, Germany. Gustav Markkula and Natasha Merat are with the Institute for Transport Studies (ITS), University of Leeds, Leeds LS2 9JT, U.K. Anna Schieben is with the German Aerospace Center (DLR), 38108 Brunswick, Germany. Fabio Tango is with the Centro Ricerche Fiat (CRF), 10043 Orbassano, Italy. Charles Fox is with the Institute for Transport Studies (ITS), University of Leeds, Leeds LS2 9JT, U.K., also with the Lincoln Centre for Autonomous Systems, University of Lincoln, Lincoln LN6 7TS, U.K., and also with Ibex Automation Ltd., Sheffield S36 8YW, U.K. This article has supplementary downloadable material available at http://ieeexplore.ieee.org, provided by the authors. Digital Object Identifier 10.1109/TITS.2020.3006767 Publisher Copyright: © 2000-2011 IEEE.

PY - 2021/9

Y1 - 2021/9

N2 - Autonomous vehicles (AVs) must share space with pedestrians, both in carriageway cases such as cars at pedestrian crossings and off-carriageway cases such as delivery vehicles navigating through crowds on pedestrianized high-streets. Unlike static obstacles, pedestrians are active agents with complex, interactive motions. Planning AV actions in the presence of pedestrians thus requires modelling of their probable future behavior as well as detecting and tracking them. This narrative review article is Part II of a pair, together surveying the current technology stack involved in this process, organising recent research into a hierarchical taxonomy ranging from low-level image detection to high-level psychological models, from the perspective of an AV designer. This self-contained Part II covers the higher levels of this stack, consisting of models of pedestrian behavior, from prediction of individual pedestrians' likely destinations and paths, to game-theoretic models of interactions between pedestrians and autonomous vehicles. This survey clearly shows that, although there are good models for optimal walking behavior, high-level psychological and social modelling of pedestrian behavior still remains an open research question that requires many conceptual issues to be clarified. Early work has been done on descriptive and qualitative models of behavior, but much work is still needed to translate them into quantitative algorithms for practical AV control.

AB - Autonomous vehicles (AVs) must share space with pedestrians, both in carriageway cases such as cars at pedestrian crossings and off-carriageway cases such as delivery vehicles navigating through crowds on pedestrianized high-streets. Unlike static obstacles, pedestrians are active agents with complex, interactive motions. Planning AV actions in the presence of pedestrians thus requires modelling of their probable future behavior as well as detecting and tracking them. This narrative review article is Part II of a pair, together surveying the current technology stack involved in this process, organising recent research into a hierarchical taxonomy ranging from low-level image detection to high-level psychological models, from the perspective of an AV designer. This self-contained Part II covers the higher levels of this stack, consisting of models of pedestrian behavior, from prediction of individual pedestrians' likely destinations and paths, to game-theoretic models of interactions between pedestrians and autonomous vehicles. This survey clearly shows that, although there are good models for optimal walking behavior, high-level psychological and social modelling of pedestrian behavior still remains an open research question that requires many conceptual issues to be clarified. Early work has been done on descriptive and qualitative models of behavior, but much work is still needed to translate them into quantitative algorithms for practical AV control.

KW - autonomous vehicles

KW - datasets

KW - detection

KW - eHMI

KW - game-theoretic models

KW - microscopic and macroscopic behavior models

KW - pedestrian interaction

KW - pedestrians

KW - Review

KW - sensing

KW - signalling models

KW - survey

KW - tracking

KW - trajectory prediction

UR - http://www.scopus.com/inward/record.url?scp=85111304323&partnerID=8YFLogxK

U2 - 10.1109/TITS.2020.3006767

DO - 10.1109/TITS.2020.3006767

M3 - Review article

AN - SCOPUS:85111304323

VL - 22

SP - 5453

EP - 5472

JO - IEEE Transactions on Intelligent Transportation Systems

JF - IEEE Transactions on Intelligent Transportation Systems

SN - 1524-9050

IS - 9

M1 - 9151337

ER -

Pedestrian Models for Autonomous Driving Part II: High-Level Models of Human Behavior

Abstract

Bibliographical note

Keywords

Access to Document

Other files and links

Cite this