Robustness analysis of SARSA(lambda): Different models of reward and initialisation

Marek Grzes; Daniel Kudenko

Robustness analysis of SARSA(lambda): Different models of reward and initialisation

Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

In the paper the robustness of SARSA(lambda), the reinforcement learning algorithm with eligibility traces, is confronted with different models of reward and initialisation of the Q-table. Most of the empirical analyses of eligibility traces in the literature have focused mainly on the step-penalty reward. We analyse two general types of rewards (final goal and step-penalty rewards) and show that learning with long traces, i.e., with high values of lambda, can lead to suboptimal solutions in some situations. Problems are identified and discussed. Specifically, obtained results show that SARSA(lambda) is sensitive to different models of reward and initialisation. In some cases the asymptotic performance can be significantly reduced.

Original language	English
Title of host publication	ARTIFICIAL INTELLIGENCE: METHODOLOGY, SYSTEMS, AND APPLICATIONS
Editors	D Dochev, M Pistore, P Traverso
Place of Publication	BERLIN
Publisher	Springer
Pages	144-156
Number of pages	13
ISBN (Print)	978-3-540-85775-4
Publication status	Published - 2008
Event	International Conference on Informational Technology and Environmental System Science - Jiaozuo Duration: 15 May 2008 → 17 May 2008

Conference

Conference	International Conference on Informational Technology and Environmental System Science
City	Jiaozuo
Period	15/05/08 → 17/05/08

Keywords

reinforcement learning
tern poral-difference learning
SARSA
Q-learning
eligibility traces

Cite this

@inproceedings{25fe5fc09816451eb6227cbe1167548b,

title = "Robustness analysis of SARSA(lambda): Different models of reward and initialisation",

abstract = "In the paper the robustness of SARSA(lambda), the reinforcement learning algorithm with eligibility traces, is confronted with different models of reward and initialisation of the Q-table. Most of the empirical analyses of eligibility traces in the literature have focused mainly on the step-penalty reward. We analyse two general types of rewards (final goal and step-penalty rewards) and show that learning with long traces, i.e., with high values of lambda, can lead to suboptimal solutions in some situations. Problems are identified and discussed. Specifically, obtained results show that SARSA(lambda) is sensitive to different models of reward and initialisation. In some cases the asymptotic performance can be significantly reduced.",

keywords = "reinforcement learning, tern poral-difference learning, SARSA, Q-learning, eligibility traces",

author = "Marek Grzes and Daniel Kudenko",

year = "2008",

language = "English",

isbn = "978-3-540-85775-4",

pages = "144--156",

editor = "D Dochev and M Pistore and P Traverso",

booktitle = "ARTIFICIAL INTELLIGENCE: METHODOLOGY, SYSTEMS, AND APPLICATIONS",

publisher = "Springer",

address = "Germany",

note = "International Conference on Informational Technology and Environmental System Science ; Conference date: 15-05-2008 Through 17-05-2008",

}

TY - GEN

T1 - Robustness analysis of SARSA(lambda): Different models of reward and initialisation

AU - Grzes, Marek

AU - Kudenko, Daniel

PY - 2008

Y1 - 2008

N2 - In the paper the robustness of SARSA(lambda), the reinforcement learning algorithm with eligibility traces, is confronted with different models of reward and initialisation of the Q-table. Most of the empirical analyses of eligibility traces in the literature have focused mainly on the step-penalty reward. We analyse two general types of rewards (final goal and step-penalty rewards) and show that learning with long traces, i.e., with high values of lambda, can lead to suboptimal solutions in some situations. Problems are identified and discussed. Specifically, obtained results show that SARSA(lambda) is sensitive to different models of reward and initialisation. In some cases the asymptotic performance can be significantly reduced.

AB - In the paper the robustness of SARSA(lambda), the reinforcement learning algorithm with eligibility traces, is confronted with different models of reward and initialisation of the Q-table. Most of the empirical analyses of eligibility traces in the literature have focused mainly on the step-penalty reward. We analyse two general types of rewards (final goal and step-penalty rewards) and show that learning with long traces, i.e., with high values of lambda, can lead to suboptimal solutions in some situations. Problems are identified and discussed. Specifically, obtained results show that SARSA(lambda) is sensitive to different models of reward and initialisation. In some cases the asymptotic performance can be significantly reduced.

KW - reinforcement learning

KW - tern poral-difference learning

KW - SARSA

KW - Q-learning

KW - eligibility traces

M3 - Conference contribution

SN - 978-3-540-85775-4

SP - 144

EP - 156

BT - ARTIFICIAL INTELLIGENCE: METHODOLOGY, SYSTEMS, AND APPLICATIONS

A2 - Dochev, D

A2 - Pistore, M

A2 - Traverso, P

PB - Springer

CY - BERLIN

T2 - International Conference on Informational Technology and Environmental System Science

Y2 - 15 May 2008 through 17 May 2008

ER -