Scholarlink

<p>Transformer models have demonstrated remarkable success in natural language processing (NLP) tasks, but their deployment in distributed environments faces critical challenges including high computational demands, large memory footprints, and privacy concerns when handling sensitive data. Existing federated learning (FL) implementations with transformers often sacrifice either model performance for privacy or resource efficiency for accuracy. We propose the <i>F-Transformer</i>, a lightweight federated transformer framework that addresses these limitations through integrated privacy-preserving mechanisms and architectural optimizations. Our framework employs a compact architecture with 4 attention heads, 4 layers, and 64 embedding dimensions, achieving only 0.87 million trainable parameters. We implement an incremental FL strategy where local clients continuously train on newly arriving data while the global model aggregates updates through the FedAvg algorithm. The framework integrates privacy objectives directly into the optimization process through a novel regularization formulation. We evaluate the <i>F-Transformer</i> on the WikiText-2 dataset using validation perplexity as the primary metric, along with central processing unit (CPU) utilization, memory consumption, and training loss convergence. Our results demonstrate a validation perplexity of 5.9894, surpassing state-of-the-art (SOTA) models including BERT-Large, GPT-2, and SparseGPT while using significantly fewer parameters. The framework achieves 40% reduction in CPU utilization and 34% reduction in memory consumption compared to centralized training. These results establish the <i>F-Transformer</i> as an effective solution for privacy-preserving sequence generation in resource-constrained federated environments.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

F-Transformer: a federated transformer for efficient and privacy-preserving sequence generation

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年5月4日

查看全文
  • Nihar Patel,
  • Sanjana Brahmbhatt,
  • Fenil Ramoliya,
  • Om Vyas,
  • Anuja Nair,
  • Tarjni Vyas,
  • Nilesh Kumar Jadav,
  • Sudeep Tanwar,
  • Abdulatif Alabdultif

摘要

Transformer models have demonstrated remarkable success in natural language processing (NLP) tasks, but their deployment in distributed environments faces critical challenges including high computational demands, large memory footprints, and privacy concerns when handling sensitive data. Existing federated learning (FL) implementations with transformers often sacrifice either model performance for privacy or resource efficiency for accuracy. We propose the F-Transformer, a lightweight federated transformer framework that addresses these limitations through integrated privacy-preserving mechanisms and architectural optimizations. Our framework employs a compact architecture with 4 attention heads, 4 layers, and 64 embedding dimensions, achieving only 0.87 million trainable parameters. We implement an incremental FL strategy where local clients continuously train on newly arriving data while the global model aggregates updates through the FedAvg algorithm. The framework integrates privacy objectives directly into the optimization process through a novel regularization formulation. We evaluate the F-Transformer on the WikiText-2 dataset using validation perplexity as the primary metric, along with central processing unit (CPU) utilization, memory consumption, and training loss convergence. Our results demonstrate a validation perplexity of 5.9894, surpassing state-of-the-art (SOTA) models including BERT-Large, GPT-2, and SparseGPT while using significantly fewer parameters. The framework achieves 40% reduction in CPU utilization and 34% reduction in memory consumption compared to centralized training. These results establish the F-Transformer as an effective solution for privacy-preserving sequence generation in resource-constrained federated environments.