Variable Hybrid Action Space Deep Q-Networks for Optimal Power Allocation and User Association in Heterogeneous Networks

Heterogeneous networks (HetNets) are essential in contemporary wireless communication networks as they help operators address challenges related to coverage, capacity, and quality. HetNets also enable the efficient use of resources and preparedness for future wireless technologies. The efficient allocation of limited resources to user equipments (UEs) is a critical problem that can be tackled using joint resource allocation and user association (JRAUA) techniques. However, this problem is challenging due to its non-convex and combinatorial nature, which is further complicated by the hybrid action space that involves both continuous actions and discrete actions. Consequently, determining the most efficient approach for JRAUA is a challenging endeavor. The objective of this study is to address the above challenge and improve the energy efficiency of networks by examining the JRAUA problem in downlink HetNets using orthogonal frequency division multiple access (OFDMA). This work presents a new strategy, called the Variable Hybrid Action Space—Deep Q-Network (VHAS-DQN), to optimize the learning policy and improve the average cumulative reward by incorporating constraints like wireless backhaul capacity of each UE and quality-of-service (QoS) into the learning process. To demonstrate the advantages of the JRAUA model in terms of energy efficiency while guaranteeing that the backhaul capacity constraints and quality of service (QoS) criteria are met, the simulation results from the VHAS-DQN were compared with other experimental case studies.