Policy Representation Opponent Shaping via Contrastive Learning

To acquire results with higher social welfare in social dilemmas, agents need to maintain cooperation. Independent agents manage to navigate social dilemmas via opponent shaping. However, opponent shaping needs extra information of opponent. It is not always accessible in mixed tasks if agents are decentralized. To address this, We present PROS, which runs in a fully-independent setting and needs no extra information. PROS shapes the opponent with an extended policy that takes the opponent’s dynamics as additional input. Instead of receiving policy from the opponent, we discriminate the policy representation via contrastive learning. In terms of experiments, PROS reaches the optimal Nash equilibrium in iterated prisoners’ dilemma (IPD) and shows the same ability to maintain cooperation in Coin Game, a highly-dimensional version of IPD. The source code is available on https://github.com/RandSF/Policy-Representation-Opponent-Shaping .