Multigrid Reinforcement Learning with Reward Shaping

Marek Grzes; Daniel Kudenko

Multigrid Reinforcement Learning with Reward Shaping

Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Potential-based reward shaping has been shown to be a powerful method to improve the convergence rate of reinforcement learning It is a flexible technique to incorporate background knowledge into temporal-difference learning in a principled way. However, the question remains how to compute the potential which is used to shape the reward that is given to the learning agent. In this paper we propose a way to solve this problem in reinforcement learning with state space discretion. In particular, we show that the potential function can be learned online in parallel with the actual reinforcement learning process. If the Q-function is learned for states determined by a given grid, a V-functional for states with lower resolution can be learned in parallel and used to approximate the potential for ground learning. The novel algorithm is presented and experimentally evaluated.

Original language	English
Title of host publication	ARTIFICIAL NEURAL NETWORKS - ICANN 2008, PT I
Editors	V Kurkova, R Neruda, J Koutnik
Place of Publication	BERLIN
Publisher	Springer
Pages	357-366
Number of pages	10
Volume	5163 LNCS
Edition	PART 1
ISBN (Print)	978-3-540-87535-2
Publication status	Published - 2008
Event	18th International Conference on Arificial Neural Networks (ICANN 2008) - Prague Duration: 3 Sep 2008 → 6 Sep 2008

Conference

Conference	18th International Conference on Arificial Neural Networks (ICANN 2008)
City	Prague
Period	3/09/08 → 6/09/08

Cite this

@inproceedings{3b07f83424914507a491ebdbfe6e4783,

title = "Multigrid Reinforcement Learning with Reward Shaping",

abstract = "Potential-based reward shaping has been shown to be a powerful method to improve the convergence rate of reinforcement learning It is a flexible technique to incorporate background knowledge into temporal-difference learning in a principled way. However, the question remains how to compute the potential which is used to shape the reward that is given to the learning agent. In this paper we propose a way to solve this problem in reinforcement learning with state space discretion. In particular, we show that the potential function can be learned online in parallel with the actual reinforcement learning process. If the Q-function is learned for states determined by a given grid, a V-functional for states with lower resolution can be learned in parallel and used to approximate the potential for ground learning. The novel algorithm is presented and experimentally evaluated.",

author = "Marek Grzes and Daniel Kudenko",

year = "2008",

language = "English",

isbn = "978-3-540-87535-2",

volume = "5163 LNCS",

pages = "357--366",

editor = "V Kurkova and R Neruda and J Koutnik",

booktitle = "ARTIFICIAL NEURAL NETWORKS - ICANN 2008, PT I",

publisher = "Springer",

address = "Germany",

edition = "PART 1",

note = "18th International Conference on Arificial Neural Networks (ICANN 2008) ; Conference date: 03-09-2008 Through 06-09-2008",

}

TY - GEN

T1 - Multigrid Reinforcement Learning with Reward Shaping

AU - Grzes, Marek

AU - Kudenko, Daniel

PY - 2008

Y1 - 2008

N2 - Potential-based reward shaping has been shown to be a powerful method to improve the convergence rate of reinforcement learning It is a flexible technique to incorporate background knowledge into temporal-difference learning in a principled way. However, the question remains how to compute the potential which is used to shape the reward that is given to the learning agent. In this paper we propose a way to solve this problem in reinforcement learning with state space discretion. In particular, we show that the potential function can be learned online in parallel with the actual reinforcement learning process. If the Q-function is learned for states determined by a given grid, a V-functional for states with lower resolution can be learned in parallel and used to approximate the potential for ground learning. The novel algorithm is presented and experimentally evaluated.

AB - Potential-based reward shaping has been shown to be a powerful method to improve the convergence rate of reinforcement learning It is a flexible technique to incorporate background knowledge into temporal-difference learning in a principled way. However, the question remains how to compute the potential which is used to shape the reward that is given to the learning agent. In this paper we propose a way to solve this problem in reinforcement learning with state space discretion. In particular, we show that the potential function can be learned online in parallel with the actual reinforcement learning process. If the Q-function is learned for states determined by a given grid, a V-functional for states with lower resolution can be learned in parallel and used to approximate the potential for ground learning. The novel algorithm is presented and experimentally evaluated.

UR - http://www.scopus.com/inward/record.url?scp=58849111871&partnerID=8YFLogxK

M3 - Conference contribution

SN - 978-3-540-87535-2

VL - 5163 LNCS

SP - 357

EP - 366

BT - ARTIFICIAL NEURAL NETWORKS - ICANN 2008, PT I

A2 - Kurkova, V

A2 - Neruda, R

A2 - Koutnik, J

PB - Springer

CY - BERLIN

T2 - 18th International Conference on Arificial Neural Networks (ICANN 2008)

Y2 - 3 September 2008 through 6 September 2008

ER -

Multigrid Reinforcement Learning with Reward Shaping

Abstract

Conference

Other files and links

Cite this