Prompt tuning has emerged as a highly effective technique in the field of natural language processing, particularly for adapting pre-trained language models to downstream tasks. Nevertheless, conventional prompt tuning techniques frequently encounter difficulties in effectively leveraging knowledge across multiple tasks, which can result in the isolation of knowledge and suboptimal performance in multitask learning scenarios. In this paper, we propose Multitask Embedding Reparameterization (MER), a novel approach designed to enhance prompt tuning by facilitating robust cross-task knowledge transfer. MER achieves this by reparameterizing task-specific prompt embeddings through a shared fusion module, enabling the model to capture and integrate knowledge from multiple tasks. This approach not only enhances model generalization but also diminishes the overall parameter footprint in comparison to conventional fine-tuning methodologies. Extensive empirical evaluations on the SuperGLUE benchmark demonstrate that MER consistently outperforms existing prompt tuning techniques, achieving substantial improvements in cross-task knowledge transfer and overall performance while maintaining high parameter efficiency. The results demonstrate the potential of MER as a promising solution for parameter-efficient and scalable multitask learning in a range of NLP applications.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Enhancing Prompt Tuning with Multitask Embedding Reparameterization

  • Xiayang Shi,
  • Yishuai Song,
  • Jingjing Liu,
  • Yinlin Li,
  • Yanna Sun

摘要

Prompt tuning has emerged as a highly effective technique in the field of natural language processing, particularly for adapting pre-trained language models to downstream tasks. Nevertheless, conventional prompt tuning techniques frequently encounter difficulties in effectively leveraging knowledge across multiple tasks, which can result in the isolation of knowledge and suboptimal performance in multitask learning scenarios. In this paper, we propose Multitask Embedding Reparameterization (MER), a novel approach designed to enhance prompt tuning by facilitating robust cross-task knowledge transfer. MER achieves this by reparameterizing task-specific prompt embeddings through a shared fusion module, enabling the model to capture and integrate knowledge from multiple tasks. This approach not only enhances model generalization but also diminishes the overall parameter footprint in comparison to conventional fine-tuning methodologies. Extensive empirical evaluations on the SuperGLUE benchmark demonstrate that MER consistently outperforms existing prompt tuning techniques, achieving substantial improvements in cross-task knowledge transfer and overall performance while maintaining high parameter efficiency. The results demonstrate the potential of MER as a promising solution for parameter-efficient and scalable multitask learning in a range of NLP applications.