Theoretical Considerations of Potential-Based Reward Shaping for Multi-Agent Systems

Sam Devlin; Daniel Kudenko

Theoretical Considerations of Potential-Based Reward Shaping for Multi-Agent Systems

Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Potential-based reward shaping has previously been proven
to both be equivalent to Q-table initialisation and guarantee
policy invariance in single-agent reinforcement learning.
The method has since been used in multi-agent reinforcement
learning without consideration of whether the theoretical
equivalence and guarantees hold. This paper extends
the existing proofs to similar results in multi-agent systems,
providing the theoretical background to explain the success
of previous empirical studies. Specically, it is proven
that the equivalence to Q-table initialisation remains and
the Nash Equilibria of the underlying stochastic game are
not modied. Furthermore, we demonstrate empirically that
potential-based reward shaping eects exploration and,
consequentially, can alter the joint policy converged upon.

Original language	English
Title of host publication	The 10th International Conference on Autonomous Agents and Multiagent Systems
Publisher	ACM
Pages	225-232
ISBN (Electronic)	0-9826571-5-3
ISBN (Print)	978-0-9826571-5-7
Publication status	Published - May 2011
Event	Tenth International Conference on Autonomous Agents and Multi-Agent Systems - Taipeh, Taiwan Duration: 2 May 2011 → …

Conference

Conference	Tenth International Conference on Autonomous Agents and Multi-Agent Systems
Country/Territory	Taiwan
City	Taipeh
Period	2/05/11 → …

Access to Document

p225-devlinFinal published version, 725 KB
Aamas11Accepted author manuscript, 317 KB

http://www.ifaamas.org/Proceedings/aamas2011/papers/D1_G45.pdf

Cite this

@inproceedings{d66d194f144941ef9d1057103e1c4864,

title = "Theoretical Considerations of Potential-Based Reward Shaping for Multi-Agent Systems",

abstract = "Potential-based reward shaping has previously been provento both be equivalent to Q-table initialisation and guarantee policy invariance in single-agent reinforcement learning.The method has since been used in multi-agent reinforcement learning without consideration of whether the theoretical equivalence and guarantees hold. This paper extendsthe existing proofs to similar results in multi-agent systems,providing the theoretical background to explain the success of previous empirical studies. Specically, it is proventhat the equivalence to Q-table initialisation remains andthe Nash Equilibria of the underlying stochastic game arenot modied. Furthermore, we demonstrate empirically thatpotential-based reward shaping eects exploration and, consequentially, can alter the joint policy converged upon.",

author = "Sam Devlin and Daniel Kudenko",

year = "2011",

month = may,

language = "English",

isbn = "978-0-9826571-5-7",

pages = "225--232",

booktitle = "The 10th International Conference on Autonomous Agents and Multiagent Systems",

publisher = "ACM",

address = "United States",

note = "Tenth International Conference on Autonomous Agents and Multi-Agent Systems ; Conference date: 02-05-2011",

}

Devlin, S & Kudenko, D 2011, Theoretical Considerations of Potential-Based Reward Shaping for Multi-Agent Systems. in The 10th International Conference on Autonomous Agents and Multiagent Systems . ACM, pp. 225-232, Tenth International Conference on Autonomous Agents and Multi-Agent Systems, Taipeh, Taiwan, 2/05/11. <http://www.ifaamas.org/Proceedings/aamas2011/papers/D1_G45.pdf>

TY - GEN

T1 - Theoretical Considerations of Potential-Based Reward Shaping for Multi-Agent Systems

AU - Devlin, Sam

AU - Kudenko, Daniel

PY - 2011/5

Y1 - 2011/5

N2 - Potential-based reward shaping has previously been provento both be equivalent to Q-table initialisation and guarantee policy invariance in single-agent reinforcement learning.The method has since been used in multi-agent reinforcement learning without consideration of whether the theoretical equivalence and guarantees hold. This paper extendsthe existing proofs to similar results in multi-agent systems,providing the theoretical background to explain the success of previous empirical studies. Specically, it is proventhat the equivalence to Q-table initialisation remains andthe Nash Equilibria of the underlying stochastic game arenot modied. Furthermore, we demonstrate empirically thatpotential-based reward shaping eects exploration and, consequentially, can alter the joint policy converged upon.

AB - Potential-based reward shaping has previously been provento both be equivalent to Q-table initialisation and guarantee policy invariance in single-agent reinforcement learning.The method has since been used in multi-agent reinforcement learning without consideration of whether the theoretical equivalence and guarantees hold. This paper extendsthe existing proofs to similar results in multi-agent systems,providing the theoretical background to explain the success of previous empirical studies. Specically, it is proventhat the equivalence to Q-table initialisation remains andthe Nash Equilibria of the underlying stochastic game arenot modied. Furthermore, we demonstrate empirically thatpotential-based reward shaping eects exploration and, consequentially, can alter the joint policy converged upon.

M3 - Conference contribution

SN - 978-0-9826571-5-7

SP - 225

EP - 232

BT - The 10th International Conference on Autonomous Agents and Multiagent Systems

PB - ACM

T2 - Tenth International Conference on Autonomous Agents and Multi-Agent Systems

Y2 - 2 May 2011

ER -