Due to rising labor costs and the increasing demand for intelligent production, the application of Automated Guided Vehicles (AGVs) for material transport is becoming increasingly widespread. Given the capability constraints of the AGVs and their highly dynamic nature, the AGV task allocation problem is much more complex than traditional task allocation problems, especially for immovable production equipment. Conventional techniques, such as heuristic and metaheuristic algorithms, often struggle to balance efficiency and real-time performance. This paper presents a new multi-agent reinforcement learning algorithm called Dense-to-Sparse Reward Switching (D2SRS) algorithm, utilizing both dense and sparse rewards to address the above problem. During the early stages of exploration, D2SRS guides the behavior of agents through dense rewards. In the later stages, it mitigates the noise issues associated with dense rewards by employing sparse rewards and further enhances the agents’ exploration of optimal strategy. Experimental results in a virtual workshop environment show that the number of orders completed in time increased by at least 7.77%, while the average processing time per order decreased by 13.2% compared to state-of-the-art heuristic algorithms and MARL with fixed reward settings.