Deep reinforcement learning (DRL) for traffic signal control has become one of the important means of urban management and congestion relief. Multi-agent RL(MARL), based on independent learning, avoids the increase in complexity with the size of the road network through localized decision-making. However, traditional MARL methods rely on immediate rewards (e.g., queue length) and without considering long-term benefits, leading to limited adaptability in large-scale networks. To address this challenge, we propose HMLight, a novel framework based on Hierarchical Multi-Agent Deep Reinforcement Learning (HMDRL). HMLight adopts a two-level model: one model (Model-H) leverages long-term trend-aware rewards for auxiliary guidance, while the other (Model-L) integrates immediate rewards and auxiliary information to make the final decision. The auxiliary information is flexibly defined based on specific tasks. In this paper, we use them to (1) adjust Model-L’s learning scope dynamically, addressing experience of pollution caused by fixed-topology shared experience mechanisms in MARL (e.g., using a global replay buffer); (2) directly intervene in Model-L’s immediate decision-making for further improve performance. Experiments on both synthetic traffic grids and real-world traffic maps demonstrate that HMLight outperforms state-of-the-art MARL-based and TSC methods in performance.