Reinforcement Learning in RoboCup KeepAway with Partial Observability

TY - GEN

T1 - Reinforcement Learning in RoboCup KeepAway with Partial Observability

AU - Devlin, Sam

AU - Grzes, Marek

AU - Kudenko, Daniel

PY - 2009

Y1 - 2009

N2 - Partially observable environments pose a major challenge to the application of reinforcement learning algorithms. In such environments, due to the Markov property frequently being violated in the system state representation, situations can occur where an agent has insufficient information to decide on the optimal action. In such cases, it is necessary to determine when information gathering actions should be executed, that is, when the agent needs to reduce uncertainty about the current state before deciding on how to act. One possible solution that has been proposed in past research is to manually code rules for execution of information gathering actions in the policy using heuristic (and likely faulty) knowledge. However such a solution requires explicit expert knowledge about actions which are information gathering.In this paper a flexible solution is proposed which automatically learns when to execute information gathering actions and furthermore to automatically discover which actions gather information. We present an evaluation in the RoboCup KeepAway domain that empirically shows the robustness of the proposed approach and its success in learning under varying degrees of partial observability. Hence, it eliminates the need for hand-coded rules, is flexible in different situations and does not require knowledge about information gathering actions.

AB - Partially observable environments pose a major challenge to the application of reinforcement learning algorithms. In such environments, due to the Markov property frequently being violated in the system state representation, situations can occur where an agent has insufficient information to decide on the optimal action. In such cases, it is necessary to determine when information gathering actions should be executed, that is, when the agent needs to reduce uncertainty about the current state before deciding on how to act. One possible solution that has been proposed in past research is to manually code rules for execution of information gathering actions in the policy using heuristic (and likely faulty) knowledge. However such a solution requires explicit expert knowledge about actions which are information gathering.In this paper a flexible solution is proposed which automatically learns when to execute information gathering actions and furthermore to automatically discover which actions gather information. We present an evaluation in the RoboCup KeepAway domain that empirically shows the robustness of the proposed approach and its success in learning under varying degrees of partial observability. Hence, it eliminates the need for hand-coded rules, is flexible in different situations and does not require knowledge about information gathering actions.

KW - Belief state

KW - KeepAway

KW - partial observability

KW - POMDP

KW - reinforcement learning

UR - http://www.scopus.com/inward/record.url?scp=84856908031&partnerID=8YFLogxK

M3 - Conference contribution

SN - 978-1-4244-5331-3

VL - 2

SP - 201

EP - 208

BT - 2009 IEEE/WIC/ACM INTERNATIONAL JOINT CONFERENCES ON WEB INTELLIGENCE (WI) AND INTELLIGENT AGENT TECHNOLOGIES (IAT), VOL 2

A2 - BaezaYates, R

A2 - Berendt, B

A2 - Bertino, E

A2 - Lim, EP

A2 - Pasi, G

PB - IEEE Computer Society

CY - LOS ALAMITOS

T2 - IEEE/WIC/ACM International Conferences on Web Intelligence (WI)/Intelligent Agent Technologies (IAT),

Y2 - 15 September 2009 through 18 September 2009

ER -