HC-API: A Hierarchical Collaborative Agent Permutation Invariant Framework for Multi-agent Reinforcement Learning

Multi-Agent Reinforcement Learning (MARL) has demonstrated its effectiveness in learning collaboration, but most methods assume that all agents share a common action space, which might not hold in scenarios with heterogeneous agents. As a result, it becomes challenging to find a unified approach that accommodates the different action requirements of individual agents. Moreover, hybrid action spaces can lead to a combinatorial explosion in the number of possible joint actions, which further exacerbates the curse of dimensionality, making it impractical to explore the entire action space efficiently. For this, we propose a Hierarchical Collaborative Agent Permutation Invariant Framework (HC-API) to learn hierarchies policies, where high-level policies determine sub-task assignments for each agent, and low-level policies control actions within assigned sub-tasks. Besides, the backbone architecture API can achieve permutation invariance and permutation equivariance, which guarantees permuting the order of entities in the factored multiagent state space does not change the information. HC-API promotes better coordination and adaptability in scenarios with hybrid action spaces and enhances the robustness to diverse scenarios. We conducted extensive evaluations on a simulation platform, and the results validate the effectiveness and robustness of our proposed framework. The source codes are released on GitHub page .