Scholarlink

Fine-tuning large Transformer models for short text classification poses significant computational challenges, limiting their widespread adoption in resource-constrained environments. This paper addresses this issue by providing a systematic empirical evaluation of Low-Rank Adaptation (LoRA) as a parameter-efficient alternative to full fine-tuning. We compared LoRA-augmented BERT, RoBERTa, and DistilBERT against their fully fine-tuned counterparts on three diverse benchmarks: R8, TREC, and SST-2. Our results demonstrate that LoRA dramatically improves efficiency, reducing the number of trainable parameters by over 98% and cutting training time by 10–40%, and lowering peak GPU memory consumption by 23–52%. Crucially, these resource savings are achieved while maintaining near-baseline accuracy. Our findings highlight RoBERTa’s robustness, consistently matching its baseline across all datasets. In contrast, models like BERT show significant hyperparameter sensitivity, yet can also attain near-baseline performance with targeted tuning. This study confirms that LoRA offers a compelling performance-efficiency trade-off, but underscores that its optimal application requires model-aware configurations.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

LoRA-Based Efficient Fine-Tuning of Transformer Models for Short Text Classification

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Xiaoxu Wang,
  • Azreen Azman

摘要

Fine-tuning large Transformer models for short text classification poses significant computational challenges, limiting their widespread adoption in resource-constrained environments. This paper addresses this issue by providing a systematic empirical evaluation of Low-Rank Adaptation (LoRA) as a parameter-efficient alternative to full fine-tuning. We compared LoRA-augmented BERT, RoBERTa, and DistilBERT against their fully fine-tuned counterparts on three diverse benchmarks: R8, TREC, and SST-2. Our results demonstrate that LoRA dramatically improves efficiency, reducing the number of trainable parameters by over 98% and cutting training time by 10–40%, and lowering peak GPU memory consumption by 23–52%. Crucially, these resource savings are achieved while maintaining near-baseline accuracy. Our findings highlight RoBERTa’s robustness, consistently matching its baseline across all datasets. In contrast, models like BERT show significant hyperparameter sensitivity, yet can also attain near-baseline performance with targeted tuning. This study confirms that LoRA offers a compelling performance-efficiency trade-off, but underscores that its optimal application requires model-aware configurations.