Multiagent reinforcement learning: Spiking and nonspiking agents in the Iterated Prisoner's Dilemma
Christodoulou, Chris C.
SourceIEEE Transactions on Neural Networks
Google Scholar check
MetadataShow full item record
This paper investigates multiagent reinforcement learning (MARL) in a general-sum game where the payoffs' structure is such that the agents are required to exploit each other in a way that benefits all agents. The contradictory nature of these games makes their study in multiagent systems quite challenging. In particular, we investigate MARL with spiking and nonspiking agents in the Iterated Prisoner's Dilemma by exploring the conditions required to enhance its cooperative outcome. The spiking agents are neural networks with leaky integrate-and-fire neurons trained with two different learning algorithms: 1) reinforcement of stochastic synaptic transmission, or 2) reward-modulated spike-timing-dependent plasticity with eligibility trace. The nonspiking agents use a tabular representation and are trained with Q- and SARSA learning algorithms, with a novel reward transformation process also being applied to the Q-learning agents. According to the results, the cooperative outcome is enhanced by: 1) transformed internal reinforcement signals and a combination of a high learning rate and a low discount factor with an appropriate exploration schedule in the case of non-spiking agents, and 2) having longer eligibility trace time constant in the case of spiking agents. Moreover, it is shown that spiking and nonspiking agents have similar behavior and therefore they can equally well be used in a multiagent interaction setting. For training the spiking agents in the case where more than one output neuron competes for reinforcement, a novel and necessary modification that enhances competition is applied to the two learning algorithms utilized, in order to avoid a possible synaptic saturation. This is done by administering to the networks additional global reinforcement signals for every spike of the output neurons that were not responsible for the preceding decision. © 2011 IEEE.
Showing items related by title, author, creator and subject.
Lambrou, Ioannis; Vassiliades, Vassilis; Christodoulou, Chris C. (2012)This paper compares and investigates single-agent reinforcement learning (RL) algorithms on the simple and an extended taxi problem domain, and multiagent RL algorithms on a multiagent extension of the simple taxi problem ...
Multiagent reinforcement learning with spiking and non-spiking agents in the iterated prisoner's dilemma Vassiliades, Vassilis; Cleanthous, A.; Christodoulou, Chris C. (2009)This paper investigates Multiagent Reinforcement Learning (MARL) in a general-sum game where the payoffs' structure is such that the agents are required to exploit each other in a way that benefits all agents. The contradictory ...
Multiagent reinforcement learning in the iterated prisoner's dilemma: Fast cooperation through evolved payoffs Vassiliades, Vassilis; Christodoulou, Chris C. (2010)In this paper, we investigate the importance of rewards in Multiagent Reinforcement Learning in the context of the Iterated Prisoner's Dilemma. We use an evolutionary algorithm to evolve valid payoff structures with the ...