Hierarchical DRL Approaches for CVaR-Aware Cryptocurrency Portfolio Optimization

Dynamic portfolio selection refers to the ongoing process of distributing investment capital among different assets over several trading periods, while accounting for each investor’s balance between risk and return preferences. Automating this decision-making process through machine learning remains a complex challenge. Two major difficulties in portfolio optimization are identifying meaningful features and managing risk effectively to achieve consistent long-term gains. Although reinforcement learning (RL) methods are increasingly used in financial markets, many existing studies still neglect the risks posed by extreme or rare market events and the effects of worst-case scenarios on trading outcomes. In this research, we address these limitations by applying a hierarchical clustering algorithm to extract the most relevant features for our model. We also introduce a novel reinforcement learning framework called the Hierarchical Deep Q-Network (HDQN). This approach integrates the Deep Q-Network (DQN) structure within a hierarchical design to determine optimal asset weight allocations. The HDQN model incorporates Conditional Value-at-Risk (CVaR) as a key risk measure in its reward function, allowing asset allocations to adapt dynamically to both investor risk tolerance and the portfolio’s current exposure at each trading stage. To test the model’s performance, we apply it to a portfolio of eight cryptocurrencies, using daily data from July 2019 to April 2023. When compared with established benchmark models through performance indicators such as the Sharpe ratio, the HDQN framework demonstrates superior results in both constructing diversified portfolios and selecting individual assets effectively.