The increase of heterogeneous AI accelerators, particularly in edge computing, creates significant challenges for building unified and scalable distributed acceleration systems. Existing approaches often lack support for diverse hardware types across multiple nodes, limit scalability, or introduce significant communication overheads. This paper introduces a novel service-oriented infrastructure concept to overcome these limitations, enabling management of distributed, heterogeneous AI accelerators. The proposed architecture features three core modules: a versatile Model Database supporting various formats and artifacts, hardware-specific Runners acting as execution agents, and a central Scheduler for system-aware task allocation. Key contributions include: 1) A conceptual framework for truly distributed, heterogeneous AI acceleration services, contrasting with single-node or vendor-locked solutions; 2) A modular design enabling extensibility and seamless lifecycle management crucial for long-term edge deployments; and 3) A design incorporating direct data communication paths between sources, accelerators, and sinks, aimed at reducing end-to-end latency compared to proxy-based or application-offloading methods by eliminating intermediate hops. The evaluations show advantages in edge scalability, hardware flexibility, built-in failure recovery, and latency advantages over existing solutions, with a reduction of up to 50% in communication-based latency compared to existing approaches. This infrastructure maximizes resource utilization by unlocking access to a diverse, heterogeneous pool of accelerators, providing robust AI acceleration in complex, dynamic environments, particularly at the network edge.