Multiagent reinforcement learning: Spiking and nonspiking agents in the Iterated Prisoner's Dilemma
Date
2011ISSN
1045-9227Source
IEEE Transactions on Neural NetworksVolume
22Issue
4Pages
639-653Google Scholar check
Keyword(s):
Metadata
Show full item recordAbstract
This paper investigates multiagent reinforcement learning (MARL) in a general-sum game where the payoffs' structure is such that the agents are required to exploit each other in a way that benefits all agents. The contradictory nature of these games makes their study in multiagent systems quite challenging. In particular, we investigate MARL with spiking and nonspiking agents in the Iterated Prisoner's Dilemma by exploring the conditions required to enhance its cooperative outcome. The spiking agents are neural networks with leaky integrate-and-fire neurons trained with two different learning algorithms: 1) reinforcement of stochastic synaptic transmission, or 2) reward-modulated spike-timing-dependent plasticity with eligibility trace. The nonspiking agents use a tabular representation and are trained with Q- and SARSA learning algorithms, with a novel reward transformation process also being applied to the Q-learning agents. According to the results, the cooperative outcome is enhanced by: 1) transformed internal reinforcement signals and a combination of a high learning rate and a low discount factor with an appropriate exploration schedule in the case of non-spiking agents, and 2) having longer eligibility trace time constant in the case of spiking agents. Moreover, it is shown that spiking and nonspiking agents have similar behavior and therefore they can equally well be used in a multiagent interaction setting. For training the spiking agents in the case where more than one output neuron competes for reinforcement, a novel and necessary modification that enhances competition is applied to the two learning algorithms utilized, in order to avoid a possible synaptic saturation. This is done by administering to the networks additional global reinforcement signals for every spike of the output neurons that were not responsible for the preceding decision. © 2011 IEEE.
Collections
Cite as
Related items
Showing items related by title, author, creator and subject.
-
Article
An extension of a hierarchical reinforcement learning algorithm for multiagent settings
Lambrou, Ioannis; Vassiliades, Vassilis; Christodoulou, Chris C. (2012)This paper compares and investigates single-agent reinforcement learning (RL) algorithms on the simple and an extended taxi problem domain, and multiagent RL algorithms on a multiagent extension of the simple taxi problem ...
-
Article
Multiagent reinforcement learning with spiking and non-spiking agents in the iterated prisoner's dilemma
Vassiliades, Vassilis; Cleanthous, A.; Christodoulou, Chris C. (2009)This paper investigates Multiagent Reinforcement Learning (MARL) in a general-sum game where the payoffs' structure is such that the agents are required to exploit each other in a way that benefits all agents. The contradictory ...
-
Conference Object
Multiagent reinforcement learning in the iterated prisoner's dilemma: Fast cooperation through evolved payoffs
Vassiliades, Vassilis; Christodoulou, Chris C. (2010)In this paper, we investigate the importance of rewards in Multiagent Reinforcement Learning in the context of the Iterated Prisoner's Dilemma. We use an evolutionary algorithm to evolve valid payoff structures with the ...