Breaking Boundaries: Can a Unified Hardware Abstraction Layer Simplify Transformer Deployments on Edge Devices?

The deployment of transformer models on edge devices like smartphones and tablets is pivotal for leveraging machine learning benefits in real-world scenarios. However, it brings forth challenges including hardware compatibility, memory efficiency, energy efficiency, and real-time performance. We introduce a versatile Hardware Abstraction Layer (HAL) to (1) bridge pre-trained transformer models with the target hardware for optimized deployment, and (2) incorporate intermediate representations (IR) as a crucial element. The IR facilitates seamless execution of models across diverse hardware backends, ensuring enhanced privacy, security, and functionality, especially in regions with limited internet connectivity. Our HAL, endowed with configurable parameters, dynamic model optimizations, and a modular design, caters to varied performance objectives, offering a unified layer that eases the deployment of IR while focusing on user-specified performance priorities. The main contribution of this work is the introduction of IR within the HAL framework, pushing the frontier in edge-device machine learning deployments to focus on latency, energy efficiency, or memory usage. Our results exhibit that the proposed HAL, with its IR component, significantly trims down deployment time and boosts inference efficiency, without compromising model accuracy on iPhone devices.