Context Cloud-based black-box model deployment faces challenges related to latency and privacy due to data transmission across Wide Area Networks. On the other hand, Mobile-based black-box deployment prioritizes privacy at the expense of higher latency due to limited computational resources. To address these issues, Edge AI enables the deployment of black-box models across Mobile, Edge, and Cloud devices using a wide range of operators able to distribute a model’s components, terminate inference early, or even quantize a model’s computations, offering latency and privacy benefits. Existing surveys classify Edge AI model inference techniques into eight families, including Quantization, Early Exiting, and Partitioning, but they often treat these operators in isolation, overlooking their potential synergies and practical integration in real-world scenarios. Deciding what combination of operators to use across the Edge AI tiers to achieve specific latency and model performance requirements is still an open question for MLOps Engineers. Objective This study aims to empirically assess the accuracy vs inference time trade-off of different black-box Edge AI deployment strategies, i.e., combinations of deployment operators and deployment tiers. Method In this paper, we conduct inference experiments involving three deployment operators (i.e., Partitioning, Quantization, Early Exit), three deployment tiers (i.e., Mobile, Edge, Cloud) and their combinations on four widely-used Computer-Vision models to investigate the optimal strategies from the point of view of MLOps developers. The analysis is conducted in a containerized environment using CUDA for Cloud GPU acceleration and ONNX for model interoperability, covering a wide range of network bandwidths. Results Our findings suggest that Edge deployment using the hybrid Quantization + Early Exit operator could be preferred over Non-Hybrid operators (Quantization/Early Exit on Edge, Partition on Mobile-Edge) when faster latency is a concern at medium accuracy loss. However, when minimizing accuracy loss is a concern, MLOps Engineers should prefer using only a Quantization operator on Edge at a latency reduction or increase, respectively over the Early Exit/Partition (on Edge/Mobile-Edge) and Quantized Early Exit (on Edge) operators. In scenarios constrained by Mobile CPU/RAM resources, a preference for Partitioning across Mobile and Edge tiers is observed over Mobile deployment. For models with smaller input data samples (such as FCN), a network-constrained Cloud deployment can also be a better alternative than Mobile/Edge deployment and Partitioning strategies. For models with large input data samples (ResNet, ResNext, DUC), an Edge tier having higher network/computational capabilities than the Cloud/Mobile tier can be a more viable option than Partitioning and Mobile/Cloud deployment strategies. Smaller input data-sized models like FCN fit well in the Cloud, even with low bandwidth (<InlineEquation ID="IEq1"> <EquationSource Format="TEX">\(\le\)</EquationSource> </InlineEquation>10 Mbps). Larger input data-sized models, like ResNe(x)t and DUC, need more bandwidth (<InlineEquation ID="IEq2"> <EquationSource Format="TEX">\(\ge\)</EquationSource> </InlineEquation>50 Mbps) for Cloud latency convergence. Partitioned-based strategies for large intermediate-sized models like FCN and DUC also need at least 50 Mbps for latency convergence. Overall, the Cloud tier performs better than the Edge and Mobile tiers for Non-Partitioning operators when the MEC bandwidth is at least 50 Mbps. However, its latency performance declines in lower bandwidth scenarios. Furthermore, Mobile-Edge Partitioning-based strategies are a better alternative compared to Mobile-Cloud and Edge-Cloud alternatives.

On the impact of black-box deployment strategies for edge AI on latency and model performance

期刊论文
发表日期：

2026年4月27日

查看全文

Jaskirat Singh,
Emad Fallahzadeh,
Bram Adams,
Ahmed E. Hassan

摘要

Context

Cloud-based black-box model deployment faces challenges related to latency and privacy due to data transmission across Wide Area Networks. On the other hand, Mobile-based black-box deployment prioritizes privacy at the expense of higher latency due to limited computational resources. To address these issues, Edge AI enables the deployment of black-box models across Mobile, Edge, and Cloud devices using a wide range of operators able to distribute a model’s components, terminate inference early, or even quantize a model’s computations, offering latency and privacy benefits. Existing surveys classify Edge AI model inference techniques into eight families, including Quantization, Early Exiting, and Partitioning, but they often treat these operators in isolation, overlooking their potential synergies and practical integration in real-world scenarios. Deciding what combination of operators to use across the Edge AI tiers to achieve specific latency and model performance requirements is still an open question for MLOps Engineers.

Objective

This study aims to empirically assess the accuracy vs inference time trade-off of different black-box Edge AI deployment strategies, i.e., combinations of deployment operators and deployment tiers.

Method

In this paper, we conduct inference experiments involving three deployment operators (i.e., Partitioning, Quantization, Early Exit), three deployment tiers (i.e., Mobile, Edge, Cloud) and their combinations on four widely-used Computer-Vision models to investigate the optimal strategies from the point of view of MLOps developers. The analysis is conducted in a containerized environment using CUDA for Cloud GPU acceleration and ONNX for model interoperability, covering a wide range of network bandwidths.

Results

Our findings suggest that Edge deployment using the hybrid Quantization + Early Exit operator could be preferred over Non-Hybrid operators (Quantization/Early Exit on Edge, Partition on Mobile-Edge) when faster latency is a concern at medium accuracy loss. However, when minimizing accuracy loss is a concern, MLOps Engineers should prefer using only a Quantization operator on Edge at a latency reduction or increase, respectively over the Early Exit/Partition (on Edge/Mobile-Edge) and Quantized Early Exit (on Edge) operators. In scenarios constrained by Mobile CPU/RAM resources, a preference for Partitioning across Mobile and Edge tiers is observed over Mobile deployment. For models with smaller input data samples (such as FCN), a network-constrained Cloud deployment can also be a better alternative than Mobile/Edge deployment and Partitioning strategies. For models with large input data samples (ResNet, ResNext, DUC), an Edge tier having higher network/computational capabilities than the Cloud/Mobile tier can be a more viable option than Partitioning and Mobile/Cloud deployment strategies. Smaller input data-sized models like FCN fit well in the Cloud, even with low bandwidth ( \(\le\) 10 Mbps). Larger input data-sized models, like ResNe(x)t and DUC, need more bandwidth ( \(\ge\) 50 Mbps) for Cloud latency convergence. Partitioned-based strategies for large intermediate-sized models like FCN and DUC also need at least 50 Mbps for latency convergence. Overall, the Cloud tier performs better than the Edge and Mobile tiers for Non-Partitioning operators when the MEC bandwidth is at least 50 Mbps. However, its latency performance declines in lower bandwidth scenarios. Furthermore, Mobile-Edge Partitioning-based strategies are a better alternative compared to Mobile-Cloud and Edge-Cloud alternatives.