Scholarlink

The deployment of large language models (LLMs) on edge devices faces significant challenges due to limited computational resources, memory bandwidth, and strict power constraints. While RISC-V architecture offers advantages in edge computing through its modularity, scalability, and open-source ecosystem, the LLM inference framework llama.cpp struggle to leverage the full potential of the RISC-V Vector Extension (RVV), leading to suboptimal performance. This study has achieved efficient LLM deployment in resource-constrained environments by optimizing critical operators in the llama.cpp framework using RVV instructions. We first identify performance bottlenecks, such as the f16 vector dot product and layer normalization, through runtime profiling. These operators are then redesigned with RVV’s SIMD capabilities, including half-precision floating-point support via the "zvfh" extension, to maximize parallel computation. Experimental results on a RISC-V-based BananaPi BPI-F3 platform demonstrate substantial improvements: the Gemma-2-2B model achieves a 1.7 × speedup in prefill prompt processing and a 53% acceleration in token generation during decoding, while the larger Llama-3.1-8B model shows 36% and 19% improvements in prefill and decode phases, respectively.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Accelerating LLM Inference on RISC-V Edge Devices via Vector Extension Optimization

  • 会议论文

  • 发表日期：

    2025年

查看全文
  • Zhilong Liu,
  • Long Peng,
  • Wenzhu Wang,
  • Ke Li,
  • Binrui Zeng,
  • Jie Yu,
  • Xiaodong Liu

摘要

The deployment of large language models (LLMs) on edge devices faces significant challenges due to limited computational resources, memory bandwidth, and strict power constraints. While RISC-V architecture offers advantages in edge computing through its modularity, scalability, and open-source ecosystem, the LLM inference framework llama.cpp struggle to leverage the full potential of the RISC-V Vector Extension (RVV), leading to suboptimal performance. This study has achieved efficient LLM deployment in resource-constrained environments by optimizing critical operators in the llama.cpp framework using RVV instructions. We first identify performance bottlenecks, such as the f16 vector dot product and layer normalization, through runtime profiling. These operators are then redesigned with RVV’s SIMD capabilities, including half-precision floating-point support via the "zvfh" extension, to maximize parallel computation. Experimental results on a RISC-V-based BananaPi BPI-F3 platform demonstrate substantial improvements: the Gemma-2-2B model achieves a 1.7 × speedup in prefill prompt processing and a 53% acceleration in token generation during decoding, while the larger Llama-3.1-8B model shows 36% and 19% improvements in prefill and decode phases, respectively.