Scholarlink

This paper introduces LoRA-NAS, a method that integrates Neural Architecture Search (NAS) with Low-Rank Adaptation (LoRA) to determine optimal adaptation settings for specific tasks. This approach enables the creation of customized Large Language Model (LLM) architectures tailored to individual datasets. LoRA-NAS operates by leveraging an evolutionary algorithm to explore the space of efficient Parameter-Efficient Fine-Tuning (PEFT) configurations. It identifies the optimal layer-wise placement strategy and hyperparameters for LoRA adapters during the PEFT of LLMs. By treating adapter insertion as a search problem, LoRA-NAS aims to customize adapter designing based on task specifics, minimizing redundant parameter updates and accelerating adaptation. A comprehensive evaluation of LoRA-NAS was conducted using the LLaMA-7B model as its base model and benchmarking it against zero-shot classification with LLaMA and the LLaMA fine-tuned with LoRA on a range of natural language understanding tasks. Experimental results demonstrate that LoRA-NAS significantly outperformed reference methods across all evaluated tasks, showing improved accuracy, underscoring the benefits of using architecture search to achieve state-of-the-art results.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Leveraging Adapters for Neural Architecture Search in Large Language Models

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Alejandro Rosales-Pérez,
  • Adrián Pastor López-Monroy

摘要

This paper introduces LoRA-NAS, a method that integrates Neural Architecture Search (NAS) with Low-Rank Adaptation (LoRA) to determine optimal adaptation settings for specific tasks. This approach enables the creation of customized Large Language Model (LLM) architectures tailored to individual datasets. LoRA-NAS operates by leveraging an evolutionary algorithm to explore the space of efficient Parameter-Efficient Fine-Tuning (PEFT) configurations. It identifies the optimal layer-wise placement strategy and hyperparameters for LoRA adapters during the PEFT of LLMs. By treating adapter insertion as a search problem, LoRA-NAS aims to customize adapter designing based on task specifics, minimizing redundant parameter updates and accelerating adaptation. A comprehensive evaluation of LoRA-NAS was conducted using the LLaMA-7B model as its base model and benchmarking it against zero-shot classification with LLaMA and the LLaMA fine-tuned with LoRA on a range of natural language understanding tasks. Experimental results demonstrate that LoRA-NAS significantly outperformed reference methods across all evaluated tasks, showing improved accuracy, underscoring the benefits of using architecture search to achieve state-of-the-art results.