Scholarlink

With the increasing deployment of complex deep learning models on edge devices, addressing the high inference latency caused by their inherent computational bottlenecks is paramount for enabling real-telligent applications. To mitigate the resulting high inference latency, we propose EdgeInferFlow, a novel framework designed to accelerate the distributed inference of chain-structured models across edge-device clusters. The framework is predicated on a two-level optimization methodology: model partitioning and distributed inference orchestration. At the partitioning level, EdgeInferFlow employs a dynamic load-balancing algorithm to mitigate computational load imbalance among partitioned sub-models. This algorithm is integrated with a distributed computation scheme for linear layers and a multi-stage strategy to minimize inter-node data dependencies and communication overhead. At the inference level, EdgeInferFlow constructs a distributed computational graph to orchestrate the data flow and parallel execution of tensors throughout the cluster. Experimental results demonstrate that our proposed method achieves a reduction in end-to-end inference latency of up to 70%.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

EdgeInferFlow: A Distributed Inference Acceleration Method for Deep Learning Chained Structure Models for Edge Devices

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Hanfeng Zhai,
  • Yifan Wang,
  • Xiaohui Peng,
  • Lei Li,
  • Xueqi Li

摘要

With the increasing deployment of complex deep learning models on edge devices, addressing the high inference latency caused by their inherent computational bottlenecks is paramount for enabling real-telligent applications. To mitigate the resulting high inference latency, we propose EdgeInferFlow, a novel framework designed to accelerate the distributed inference of chain-structured models across edge-device clusters. The framework is predicated on a two-level optimization methodology: model partitioning and distributed inference orchestration. At the partitioning level, EdgeInferFlow employs a dynamic load-balancing algorithm to mitigate computational load imbalance among partitioned sub-models. This algorithm is integrated with a distributed computation scheme for linear layers and a multi-stage strategy to minimize inter-node data dependencies and communication overhead. At the inference level, EdgeInferFlow constructs a distributed computational graph to orchestrate the data flow and parallel execution of tensors throughout the cluster. Experimental results demonstrate that our proposed method achieves a reduction in end-to-end inference latency of up to 70%.