Policy Invariance under Reward Transformations for Multi-Objective Reinforcement Learning

Patrick Mannion; Sam Devlin; Karl Mason; Jim Duggan; Enda Howley

doi:10.1016/j.neucom.2017.05.090

Policy Invariance under Reward Transformations for Multi-Objective Reinforcement Learning

Patrick Mannion, Sam Devlin, Karl Mason, Jim Duggan, Enda Howley

Computer Science

Research output: Contribution to journal › Article › peer-review

Abstract

Reinforcement Learning (RL) is a powerful and well-studied Machine Learning paradigm, where an agent learns to improve its performance in an environment by maximising a reward signal. In multi-objective Reinforcement Learning (MORL) the reward signal is a vector, where each component represents the performance on a different objective. Reward shaping is a well-established family of techniques that have been successfully used to improve the performance and learning speed of RL agents in single-objective problems. The basic premise of reward shaping is to add an additional shaping reward to the reward naturally received from the environment, to incorporate domain knowledge and guide an agent’s exploration. Potential-Based Reward Shaping (PBRS) is a specific form of reward shaping that offers additional guarantees. In this paper, we extend the theoretical guarantees of PBRS to MORL problems. Specifically, we provide theoretical proof that PBRS does not alter the true Pareto front in both single- and multi-agent MORL. We also contribute the first published empirical studies of the effect of PBRS in single- and multi-agent MORL problems.

Original language	English
Article number	263
Pages (from-to)	60-73
Number of pages	14
Journal	Neurocomputing
Volume	263
Early online date	16 Jun 2017
DOIs	https://doi.org/10.1016/j.neucom.2017.05.090
Publication status	Published - 8 Nov 2017

Bibliographical note

Keywords

Reinforcement Learning
Multi-Objective
Reward Shaping

Access to Document

10.1016/j.neucom.2017.05.090

MO_PBRS_R3
© 2017 Elsevier B.V. This is an author-produced version of the published paper. Uploaded in accordance with the publisher’s self-archiving policy.
Accepted author manuscript, 516 KBLicence: CC BY-NC-ND

Cite this

@article{572f933084144270ad0bd551e8626d15,

title = "Policy Invariance under Reward Transformations for Multi-Objective Reinforcement Learning",

abstract = "Reinforcement Learning (RL) is a powerful and well-studied Machine Learning paradigm, where an agent learns to improve its performance in an environment by maximising a reward signal. In multi-objective Reinforcement Learning (MORL) the reward signal is a vector, where each component represents the performance on a different objective. Reward shaping is a well-established family of techniques that have been successfully used to improve the performance and learning speed of RL agents in single-objective problems. The basic premise of reward shaping is to add an additional shaping reward to the reward naturally received from the environment, to incorporate domain knowledge and guide an agent{\textquoteright}s exploration. Potential-Based Reward Shaping (PBRS) is a specific form of reward shaping that offers additional guarantees. In this paper, we extend the theoretical guarantees of PBRS to MORL problems. Specifically, we provide theoretical proof that PBRS does not alter the true Pareto front in both single- and multi-agent MORL. We also contribute the first published empirical studies of the effect of PBRS in single- and multi-agent MORL problems.",

keywords = "Reinforcement Learning, Multi-Objective, Reward Shaping",

author = "Patrick Mannion and Sam Devlin and Karl Mason and Jim Duggan and Enda Howley",

note = "{\textcopyright} 2017 Elsevier B.V. This is an author-produced version of the published paper. Uploaded in accordance with the publisher{\textquoteright}s self-archiving policy. ",

year = "2017",

month = nov,

day = "8",

doi = "10.1016/j.neucom.2017.05.090",

language = "English",

volume = "263",

pages = "60--73",

journal = "Neurocomputing",

issn = "0925-2312",

publisher = "Elsevier",

}

TY - JOUR

T1 - Policy Invariance under Reward Transformations for Multi-Objective Reinforcement Learning

AU - Mannion, Patrick

AU - Devlin, Sam

AU - Mason, Karl

AU - Duggan, Jim

AU - Howley, Enda

PY - 2017/11/8

Y1 - 2017/11/8

N2 - Reinforcement Learning (RL) is a powerful and well-studied Machine Learning paradigm, where an agent learns to improve its performance in an environment by maximising a reward signal. In multi-objective Reinforcement Learning (MORL) the reward signal is a vector, where each component represents the performance on a different objective. Reward shaping is a well-established family of techniques that have been successfully used to improve the performance and learning speed of RL agents in single-objective problems. The basic premise of reward shaping is to add an additional shaping reward to the reward naturally received from the environment, to incorporate domain knowledge and guide an agent’s exploration. Potential-Based Reward Shaping (PBRS) is a specific form of reward shaping that offers additional guarantees. In this paper, we extend the theoretical guarantees of PBRS to MORL problems. Specifically, we provide theoretical proof that PBRS does not alter the true Pareto front in both single- and multi-agent MORL. We also contribute the first published empirical studies of the effect of PBRS in single- and multi-agent MORL problems.

AB - Reinforcement Learning (RL) is a powerful and well-studied Machine Learning paradigm, where an agent learns to improve its performance in an environment by maximising a reward signal. In multi-objective Reinforcement Learning (MORL) the reward signal is a vector, where each component represents the performance on a different objective. Reward shaping is a well-established family of techniques that have been successfully used to improve the performance and learning speed of RL agents in single-objective problems. The basic premise of reward shaping is to add an additional shaping reward to the reward naturally received from the environment, to incorporate domain knowledge and guide an agent’s exploration. Potential-Based Reward Shaping (PBRS) is a specific form of reward shaping that offers additional guarantees. In this paper, we extend the theoretical guarantees of PBRS to MORL problems. Specifically, we provide theoretical proof that PBRS does not alter the true Pareto front in both single- and multi-agent MORL. We also contribute the first published empirical studies of the effect of PBRS in single- and multi-agent MORL problems.

KW - Reinforcement Learning

KW - Multi-Objective

KW - Reward Shaping

U2 - 10.1016/j.neucom.2017.05.090

DO - 10.1016/j.neucom.2017.05.090

M3 - Article

SN - 0925-2312

VL - 263

SP - 60

EP - 73

JO - Neurocomputing

JF - Neurocomputing

M1 - 263

ER -