Improving Optimistic Exploration in Model-Free Reinforcement Learning

Marek Grzes; Daniel Kudenko

Improving Optimistic Exploration in Model-Free Reinforcement Learning

Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

The key problem in reinforcement learning is the exploration-exploitation tradeoff. An optimistic initialisation of the value function is a. popular RI strategy. The problem of this approach is that the algorithm may have relatively low performance after many episodes of learning. In this paper, two extensions to standard optimistic exploration are proposed. The first one is based on different initialisation of the value function of goal states. The second one which builds on the previous idea explicitly separates propagation of low and high values in the state space. Proposed extensions show improvement in empirical comparisons with basic optimistic initialisation. Additionally, they improve anytime performance and help on domains where learning takes place on the sub-space of the large state space, that is, where the standard optimistic approach faces more difficulties.

Original language	English
Title of host publication	ADAPTIVE AND NATURAL COMPUTING ALGORITHMS
Editors	M Kolehmainen, P Toivanen, B Beliczynski
Place of Publication	BERLIN
Publisher	Springer
Pages	360-369
Number of pages	10
Volume	5495 LNCS
ISBN (Print)	978-3-642-04920-0
Publication status	Published - 2009
Event	9th International Conference on Adaptive and Natural Computing Algorithms (ICANNGA) - Kuopio Duration: 23 Apr 2009 → 25 Apr 2009

Conference

Conference	9th International Conference on Adaptive and Natural Computing Algorithms (ICANNGA)
City	Kuopio
Period	23/04/09 → 25/04/09

Cite this

@inproceedings{852584b1d1f44ca3afb3b9850e8570d4,

title = "Improving Optimistic Exploration in Model-Free Reinforcement Learning",

abstract = "The key problem in reinforcement learning is the exploration-exploitation tradeoff. An optimistic initialisation of the value function is a. popular RI strategy. The problem of this approach is that the algorithm may have relatively low performance after many episodes of learning. In this paper, two extensions to standard optimistic exploration are proposed. The first one is based on different initialisation of the value function of goal states. The second one which builds on the previous idea explicitly separates propagation of low and high values in the state space. Proposed extensions show improvement in empirical comparisons with basic optimistic initialisation. Additionally, they improve anytime performance and help on domains where learning takes place on the sub-space of the large state space, that is, where the standard optimistic approach faces more difficulties.",

author = "Marek Grzes and Daniel Kudenko",

year = "2009",

language = "English",

isbn = "978-3-642-04920-0",

volume = "5495 LNCS",

pages = "360--369",

editor = "M Kolehmainen and P Toivanen and B Beliczynski",

booktitle = "ADAPTIVE AND NATURAL COMPUTING ALGORITHMS",

publisher = "Springer",

address = "Germany",

note = "9th International Conference on Adaptive and Natural Computing Algorithms (ICANNGA) ; Conference date: 23-04-2009 Through 25-04-2009",

}

TY - GEN

T1 - Improving Optimistic Exploration in Model-Free Reinforcement Learning

AU - Grzes, Marek

AU - Kudenko, Daniel

PY - 2009

Y1 - 2009

N2 - The key problem in reinforcement learning is the exploration-exploitation tradeoff. An optimistic initialisation of the value function is a. popular RI strategy. The problem of this approach is that the algorithm may have relatively low performance after many episodes of learning. In this paper, two extensions to standard optimistic exploration are proposed. The first one is based on different initialisation of the value function of goal states. The second one which builds on the previous idea explicitly separates propagation of low and high values in the state space. Proposed extensions show improvement in empirical comparisons with basic optimistic initialisation. Additionally, they improve anytime performance and help on domains where learning takes place on the sub-space of the large state space, that is, where the standard optimistic approach faces more difficulties.

AB - The key problem in reinforcement learning is the exploration-exploitation tradeoff. An optimistic initialisation of the value function is a. popular RI strategy. The problem of this approach is that the algorithm may have relatively low performance after many episodes of learning. In this paper, two extensions to standard optimistic exploration are proposed. The first one is based on different initialisation of the value function of goal states. The second one which builds on the previous idea explicitly separates propagation of low and high values in the state space. Proposed extensions show improvement in empirical comparisons with basic optimistic initialisation. Additionally, they improve anytime performance and help on domains where learning takes place on the sub-space of the large state space, that is, where the standard optimistic approach faces more difficulties.

UR - http://www.scopus.com/inward/record.url?scp=78650724420&partnerID=8YFLogxK

M3 - Conference contribution

SN - 978-3-642-04920-0

VL - 5495 LNCS

SP - 360

EP - 369

BT - ADAPTIVE AND NATURAL COMPUTING ALGORITHMS

A2 - Kolehmainen, M

A2 - Toivanen, P

A2 - Beliczynski, B

PB - Springer

CY - BERLIN

T2 - 9th International Conference on Adaptive and Natural Computing Algorithms (ICANNGA)

Y2 - 23 April 2009 through 25 April 2009

ER -

Improving Optimistic Exploration in Model-Free Reinforcement Learning

Abstract

Conference

Other files and links

Cite this