Addressing Coupled Constrained Reinforcement Learning via Interative Iteration Design

Coupled constraints are a natural setting in many programming problems, such as edge computing, which makes agents more perplexed when updating policies. Existing primal-dual approaches often require additional conditions to be satisfied, and policy gradient methods easily fall into the inherent trade-off between maximizing task reward and satisfying constraints because the constraint information cannot be fed into the network. For such a setting, we design an Interactive Iteration (InIt) structure for policy gradient, which decouples high-dimensional coupled action space into multiple independent sub-action spaces to reduce the dimensionality of action space and the coupling of constraints. Then, we utilize policy gradient with strong convergence guarantees to sequentially solve the sub-actions and iteratively derive the optimal hybrid action to avoid the trade-off. Further, we propose a balancing loss for self-coupled actions, which enables the policy to pursue high task reward while complying with the objective constraints via the balancing feedback from the environment. Additionally, we conceive a notion of coupling compactibility to guide the decoupling of high-dimensional coupled action space to reduce model redundancy with minimal reward loss. Finally, we select two cases from the CCMDP scenario to evaluate our model, and the results demonstrate the superiority of our approach, especially for high-dimensional coupled action space.