Multi-agent Reinforcement Learning for Unmanned Aerial Vehicle Capture-the-Flag Game Behavior

This paper demonstrates the use of macro-action-based reinforcement learning to develop a decentralized game strategy for unmanned aerial vehicles in a real-world capture the flag game. Each agent chooses actions with only local observations. There is no global entity in control of the agents. Multi-agent reinforcement learning enables them to develop an effective team strategy, adapting to other agents by observing their behavior. We compare different choices for action spaces and show that sampling from a categorical output space is favorable over sampling from a continuous action space. All macro actions are implemented with hard-coded safety constraints such that no collisions between the agents or with the environment are possible. We successfully transfer simulation machine learning results to hardware. This approach is particularly favorable for real-world robotics because of the sample efficiency, potential for rapid iterations, and the ability to leverage strong and effective non-machine learned control techniques within the reinforcement learning approach.