Proximal Policy Optimization for Same-Day Delivery with Drones and Vehicles

With a surge demand for instant gratification in online-shopping, offering same-day delivery with heterogeneous fleets of drones and vehicles provides new insights for decision makers. However, decisions in real-time involving assignment and routing of vehicles and drones suffer “curse of dimensionality”, due to stochastic and dynamic orders, huge state spaces as well as associated and diverse decisions. In this paper, a deep reinforcement learning (DRL) based approach is presented to handle this dynamic decision problem. First, a routed-based Markov decision process is formulated to model the problem. Besides, a DRL-based algorithm combining proximal policy optimization and heuristics (PPOh) is developed to decide whether to accept customer requests, how to assign orders and plan routes of fleets. Evaluation on extensive computational experiments shows that PPOh outperforms the extant methods and evidently improves service rates of fleets under the same workload.