Recent breakthroughs in autonomous robotics have established Robotic Mobile Fulfillment Systems (RMFS) as integral components of modern warehouse management. Nevertheless, achieving high order fulfillment efficiency remains a significant challenge, particularly in large-scale constrained environments. This study proposes a hierarchical reinforcement learning framework to enhance order fulfillment efficiency by fostering collaboration and coordination among robots while addressing constraints such as sparse rewards, limited communication, and partial-observability. By leveraging goal-directed curiosity-driven exploration, the framework enables robots to adapt their strategies to navigate warehouse environments efficiently, explore novel states, and maintain focus on completing their assigned tasks. Simulations in warehouse settings evaluated the performance of the approach against state-of-the-art baselines such as QMIX and Independent Q-Learning (IQL). Empirical results demonstrated significant improvements in order completion rates and overall operational efficiency, highlighting the transformative impact of hierarchical reinforcement learning in driving logistics operations within RMFS and advancing supply chain practices.