Large Language Models (LLMs) have demonstrated remarkable performance across various natural language processing tasks. However, adapting them to domain-specific problems like text-to-SQL generation remains computationally expensive, particularly in dynamic environments where schemas frequently evolve. Full fine-tuning for each new database schema is impractical due to high computational and memory costs. To address this challenge, we propose TraceTune, a parameter-efficient adaptation method that selectively fine-tunes the most relevant attention heads and bias terms identified through causal tracing. Experiments on multiple Text-to-SQL benchmarks demonstrate that TraceTune achieves comparable accuracy to full fine-tuning while significantly reducing memory and compute overhead. These results highlight TraceTune as an efficient and scalable solution for deploying Text-to-SQL systems in resource-constrained environments.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

TraceTune: Targeted Fine-Tuning of Attention Heads for Text-to-SQL

  • Saba Zamankhani,
  • Kai-Uwe Sattler

摘要

Large Language Models (LLMs) have demonstrated remarkable performance across various natural language processing tasks. However, adapting them to domain-specific problems like text-to-SQL generation remains computationally expensive, particularly in dynamic environments where schemas frequently evolve. Full fine-tuning for each new database schema is impractical due to high computational and memory costs. To address this challenge, we propose TraceTune, a parameter-efficient adaptation method that selectively fine-tunes the most relevant attention heads and bias terms identified through causal tracing. Experiments on multiple Text-to-SQL benchmarks demonstrate that TraceTune achieves comparable accuracy to full fine-tuning while significantly reducing memory and compute overhead. These results highlight TraceTune as an efficient and scalable solution for deploying Text-to-SQL systems in resource-constrained environments.