FlexPie: Accelerate Distributed Inference on Edge Devices with Flexible Combinatorial Optimization

The rapid advancement of deep learning has catalyzed the development of novel IoT applications, which often deploy pre-trained deep neural network (DNN) models across multiple edge devices (typically 4 \(\sim \) 6) for collaborative inference. However, conventional deployment methods frequently result in suboptimal inference times. Our study identifies that the inefficiency primarily stems from the model partitioning strategy employed by inference engines. Previous frameworks often rely on fixed partitioning schemes (e.g., one-dimensional or 2D-grid partitions) or limited optimizations (e.g., layer-wise adjustments or simple layer fusion), which fail to deliver optimal performance across varying model layers and testbed configurations. In this paper, we propose FlexPie, a solution to accelerate distributed inference on edge devices through flexible combinatorial optimization. FlexPie integrates an automated optimization procedure based on a data-driven cost model and dynamic programming, which efficiently finds an optimized model partition scheme in huge combinatorial spaces. Our evaluation on four commonly used DNN benchmarks demonstrates that FlexPie reduces inference time, achieving up to a 2.39 \(\times \) speedup over state-of-the-art methods.