Scholarlink

Federated Large Language Model (FedLLM) shows excellent potential in collaboratively training large language models (LLM) under the federated learning (FL) framework, which is benefiting from its privacy protection advantage. However, FedLLM faces the significant challenge of the non-IID problem. In the real world, there are often cross-source or even cross-domain language set data between clients. To address this challenge, we propose a new FedLLM framework FedALoRA by the method of personalized and efficient parameter fine-tuning (PEFT). Specifically, the proposed scheme combines the personalized aggregation method and the LoRA method. It can adaptively aggregate the downloaded global model and local model to the local target on each client while ensuring low training costs. This adaptation initializes the local model before each iterative training, enabling clients to learn general knowledge while enhancing their understanding of their own domain knowledge. Extensive experiments and analysis on cross-domain non-IID settings demonstrate the effectiveness and superiority of FedALoRA.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

FedALoRA: Adaptive Local LoRA Aggregation for Personalized Federated Learning in LLM

  • 会议论文

  • 发表日期：

    2025年

查看全文
  • Xinzhiyi Yi,
  • Chunqiang Hu,
  • Bin Cai

摘要

Federated Large Language Model (FedLLM) shows excellent potential in collaboratively training large language models (LLM) under the federated learning (FL) framework, which is benefiting from its privacy protection advantage. However, FedLLM faces the significant challenge of the non-IID problem. In the real world, there are often cross-source or even cross-domain language set data between clients. To address this challenge, we propose a new FedLLM framework FedALoRA by the method of personalized and efficient parameter fine-tuning (PEFT). Specifically, the proposed scheme combines the personalized aggregation method and the LoRA method. It can adaptively aggregate the downloaded global model and local model to the local target on each client while ensuring low training costs. This adaptation initializes the local model before each iterative training, enabling clients to learn general knowledge while enhancing their understanding of their own domain knowledge. Extensive experiments and analysis on cross-domain non-IID settings demonstrate the effectiveness and superiority of FedALoRA.