Improving Optimistic Exploration in Model-Free Reinforcement Learning

Marek Grzes; Daniel Kudenko

doi:10.1007/978-3-642-04921-7_37

Improving Optimistic Exploration in Model-Free Reinforcement Learning

Marek Grzes, Daniel Kudenko

Computer Science

Research output: Contribution to conference › Paper › peer-review

Abstract

The key problem in reinforcement learning is the exploration-exploitation tradeoff. An optimistic initialisation of the value function is a popular RL strategy. The problem of this approach is that the algorithm may have relatively low performance after many episodes of learning. In this paper, two extensions to standard optimistic exploration are proposed. The first one is based on different initialisation of the value function of goal states. The second one which builds on the previous idea explicitly separates propagation of low and high values in the state space. Proposed extensions show improvement in empirical comparisons with basic optimistic initialisation. Additionally, they improve anytime performance and help on domains where learning takes place on the sub-space of the large state space, that is, where the standard optimistic approach faces more difficulties.

Original language	Undefined/Unknown
Pages	360-369
DOIs	https://doi.org/10.1007/978-3-642-04921-7_37
Publication status	Published - 2009

Access to Document

10.1007/978-3-642-04921-7_37

http://dx.doi.org/10.1007/978-3-642-04921-7_37

Cite this

@conference{2794e749da8d48cab7be1116eb447295,

title = "Improving Optimistic Exploration in Model-Free Reinforcement Learning",

abstract = "The key problem in reinforcement learning is the exploration-exploitation tradeoff. An optimistic initialisation of the value function is a popular RL strategy. The problem of this approach is that the algorithm may have relatively low performance after many episodes of learning. In this paper, two extensions to standard optimistic exploration are proposed. The first one is based on different initialisation of the value function of goal states. The second one which builds on the previous idea explicitly separates propagation of low and high values in the state space. Proposed extensions show improvement in empirical comparisons with basic optimistic initialisation. Additionally, they improve anytime performance and help on domains where learning takes place on the sub-space of the large state space, that is, where the standard optimistic approach faces more difficulties. ",

author = "Marek Grzes and Daniel Kudenko",

year = "2009",

doi = "10.1007/978-3-642-04921-7_37",

language = "Undefined/Unknown",

pages = "360--369",

}

TY - CONF

T1 - Improving Optimistic Exploration in Model-Free Reinforcement Learning

AU - Grzes, Marek

AU - Kudenko, Daniel

PY - 2009

Y1 - 2009

N2 - The key problem in reinforcement learning is the exploration-exploitation tradeoff. An optimistic initialisation of the value function is a popular RL strategy. The problem of this approach is that the algorithm may have relatively low performance after many episodes of learning. In this paper, two extensions to standard optimistic exploration are proposed. The first one is based on different initialisation of the value function of goal states. The second one which builds on the previous idea explicitly separates propagation of low and high values in the state space. Proposed extensions show improvement in empirical comparisons with basic optimistic initialisation. Additionally, they improve anytime performance and help on domains where learning takes place on the sub-space of the large state space, that is, where the standard optimistic approach faces more difficulties.

AB - The key problem in reinforcement learning is the exploration-exploitation tradeoff. An optimistic initialisation of the value function is a popular RL strategy. The problem of this approach is that the algorithm may have relatively low performance after many episodes of learning. In this paper, two extensions to standard optimistic exploration are proposed. The first one is based on different initialisation of the value function of goal states. The second one which builds on the previous idea explicitly separates propagation of low and high values in the state space. Proposed extensions show improvement in empirical comparisons with basic optimistic initialisation. Additionally, they improve anytime performance and help on domains where learning takes place on the sub-space of the large state space, that is, where the standard optimistic approach faces more difficulties.

UR - http://www.scopus.com/inward/record.url?scp=78650724420&partnerID=8YFLogxK

U2 - 10.1007/978-3-642-04921-7_37

DO - 10.1007/978-3-642-04921-7_37

M3 - Paper

SP - 360

EP - 369

ER -

Improving Optimistic Exploration in Model-Free Reinforcement Learning

Abstract

Access to Document

Other files and links

Cite this