Reinforcement Learning in RoboCup KeepAway with Partial Observability

Sam Devlin; Marek Grzes; Daniel Kudenko

doi:10.1109/WI-IAT.2009.151

Reinforcement Learning in RoboCup KeepAway with Partial Observability

Sam Devlin, Marek Grzes, Daniel Kudenko

Computer Science

Research output: Contribution to conference › Paper › peer-review

Abstract

Partially observable environments pose a major challenge to the application of reinforcement learning algorithms. In such environments, due to the Markov property frequently being violated in the system state representation, situations can occur where an agent has insufficient information to decide on the optimal action. In such cases, it is necessary to determine when information gathering actions should be executed, that is, when the agent needs to reduce uncertainty about the current state before deciding on how to act. One possible solution that has been proposed in past research is to manually code rules for execution of information gathering actions in the policy using heuristic (and likely faulty) knowledge. However, such a solution requires explicit expert knowledge about actions which are information gathering. In this paper a flexible solution is proposed which automatically learns when to execute information gathering actions and furthermore to automatically discover which actions gather information. We present an evaluation in the Robo{C}up Keep{A}way domain that empirically shows the robustness of the proposed approach and its success in learning under varying degrees of partial observability. Hence, it eliminates the need for hand-coded rules, is flexible in different situations and does not require knowledge about information gathering actions.

Original language	Undefined/Unknown
Pages	201-208
DOIs	https://doi.org/10.1109/WI-IAT.2009.151
Publication status	Published - 2009

Access to Document

10.1109/WI-IAT.2009.151

http://dx.doi.org/10.1109/WI-IAT.2009.151

Cite this

@conference{c36f3abb7dcb47ffa78b861d8e7f2ff6,

title = "Reinforcement Learning in RoboCup KeepAway with Partial Observability",

abstract = "Partially observable environments pose a major challenge to the application of reinforcement learning algorithms. In such environments, due to the Markov property frequently being violated in the system state representation, situations can occur where an agent has insufficient information to decide on the optimal action. In such cases, it is necessary to determine when information gathering actions should be executed, that is, when the agent needs to reduce uncertainty about the current state before deciding on how to act. One possible solution that has been proposed in past research is to manually code rules for execution of information gathering actions in the policy using heuristic (and likely faulty) knowledge. However, such a solution requires explicit expert knowledge about actions which are information gathering. In this paper a flexible solution is proposed which automatically learns when to execute information gathering actions and furthermore to automatically discover which actions gather information. We present an evaluation in the Robo{C}up Keep{A}way domain that empirically shows the robustness of the proposed approach and its success in learning under varying degrees of partial observability. Hence, it eliminates the need for hand-coded rules, is flexible in different situations and does not require knowledge about information gathering actions.",

author = "Sam Devlin and Marek Grzes and Daniel Kudenko",

year = "2009",

doi = "10.1109/WI-IAT.2009.151",

language = "Undefined/Unknown",

pages = "201--208",

}

TY - CONF

T1 - Reinforcement Learning in RoboCup KeepAway with Partial Observability

AU - Devlin, Sam

AU - Grzes, Marek

AU - Kudenko, Daniel

PY - 2009

Y1 - 2009

N2 - Partially observable environments pose a major challenge to the application of reinforcement learning algorithms. In such environments, due to the Markov property frequently being violated in the system state representation, situations can occur where an agent has insufficient information to decide on the optimal action. In such cases, it is necessary to determine when information gathering actions should be executed, that is, when the agent needs to reduce uncertainty about the current state before deciding on how to act. One possible solution that has been proposed in past research is to manually code rules for execution of information gathering actions in the policy using heuristic (and likely faulty) knowledge. However, such a solution requires explicit expert knowledge about actions which are information gathering. In this paper a flexible solution is proposed which automatically learns when to execute information gathering actions and furthermore to automatically discover which actions gather information. We present an evaluation in the Robo{C}up Keep{A}way domain that empirically shows the robustness of the proposed approach and its success in learning under varying degrees of partial observability. Hence, it eliminates the need for hand-coded rules, is flexible in different situations and does not require knowledge about information gathering actions.

AB - Partially observable environments pose a major challenge to the application of reinforcement learning algorithms. In such environments, due to the Markov property frequently being violated in the system state representation, situations can occur where an agent has insufficient information to decide on the optimal action. In such cases, it is necessary to determine when information gathering actions should be executed, that is, when the agent needs to reduce uncertainty about the current state before deciding on how to act. One possible solution that has been proposed in past research is to manually code rules for execution of information gathering actions in the policy using heuristic (and likely faulty) knowledge. However, such a solution requires explicit expert knowledge about actions which are information gathering. In this paper a flexible solution is proposed which automatically learns when to execute information gathering actions and furthermore to automatically discover which actions gather information. We present an evaluation in the Robo{C}up Keep{A}way domain that empirically shows the robustness of the proposed approach and its success in learning under varying degrees of partial observability. Hence, it eliminates the need for hand-coded rules, is flexible in different situations and does not require knowledge about information gathering actions.

UR - http://www.scopus.com/inward/record.url?scp=84856908031&partnerID=8YFLogxK

U2 - 10.1109/WI-IAT.2009.151

DO - 10.1109/WI-IAT.2009.151

M3 - Paper

SP - 201

EP - 208

ER -

Reinforcement Learning in RoboCup KeepAway with Partial Observability

Abstract

Access to Document

Other files and links

Cite this