We present a theoretical study of distributed Large Language Model inference on a low-cost Raspberry Pi 5 edge cluster. The proposed design uses pipeline parallelism where the transformer layers are partitioned across nodes and executed by lightweight, containerized microservices. This approach targets interactive token generation under constraints on memory capacity, CPU throughput and network bandwidth. Using a quantized transformer model as a reference workload, we outline an analysis in terms of time-to-first-token (TTFT), steady-state tokens-per-second (TPS) and inter-node communication overhead. The analysis suggests that distribution can mitigate single-node memory limits, while end-to-end responsiveness is often dominated by communication latency and synchronization frequency, highlighting a scalability trade-off in edge LLM deployment.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

A Theoretical Evaluation of Distributed Large Language Model Inference on a Raspberry Pi 5 Edge Cluster

  • Theodor-Radu Grumeza,
  • Grigore-Liviu Stăniloiu,
  • Alexandra-Emilia Fortiş

摘要

We present a theoretical study of distributed Large Language Model inference on a low-cost Raspberry Pi 5 edge cluster. The proposed design uses pipeline parallelism where the transformer layers are partitioned across nodes and executed by lightweight, containerized microservices. This approach targets interactive token generation under constraints on memory capacity, CPU throughput and network bandwidth. Using a quantized transformer model as a reference workload, we outline an analysis in terms of time-to-first-token (TTFT), steady-state tokens-per-second (TPS) and inter-node communication overhead. The analysis suggests that distribution can mitigate single-node memory limits, while end-to-end responsiveness is often dominated by communication latency and synchronization frequency, highlighting a scalability trade-off in edge LLM deployment.