Efficiently managing municipal budgets requires innovative strategies for allocating funds across a broad spectrum of asset classes, including sewer systems and pavement networks, and their respective subclasses, such as arterial and collector roads, which often derive from varied funding sources. This study introduces an innovative method for optimizing cross-asset budget allocation through a hierarchical deep reinforcement learning framework. Utilizing the soft-actor critic algorithm at the system level, our approach facilitates a dynamic and adaptive distribution of budgets across and within asset classes, while linear programming at the asset level ensures optimal fund utilization. This combined strategy allows for customized budget allocation, catering to the specific needs and priorities of different asset classes or subclasses, regardless of their functional similarities or disparate funding sources. Our methodology demonstrates marked improvements in budget allocation efficiency and asset performance, highlighting the potential for refined and effective MRR planning and financial management within municipal infrastructure. This research illustrates the effectiveness of integrating advanced machine learning techniques with conventional optimization methods to address complex optimization challenges, providing valuable insights for municipal budget policy-making.