Prompt-Partitioned Multi-task Learning for Universal Sentence Representations
摘要
Universal sentence encoders aim to underpin a wide spectrum of downstream tasks—semantic similarity, document retrieval, and question answering—within a shared embedding space. Yet converging heterogeneous supervision in a single model often triggers semantic interference: conflicting objectives tug representations in incompatible directions and erode cross-task generalisation. We introduce a prompt-partitioned multi-task framework that cleanly isolates task semantics via lightweight, discrete prompts (e.g., [SIM], [REL], [QA]) prepended to each input. These prompts steer the encoder toward task-specific sub-spaces without altering its architecture. To further counter data imbalance and label noise, we devise a large-language-model (LLM) pipeline that synthesises and rigorously filters training pairs in a prompt-aware fashion. Leveraging a RoBERTa-base encoder trained on 6.8 billion multilingual sentence pairs, we evaluate on 26 public benchmarks spanning classification, similarity, ranking, and retrieval. Our single-tower approach consistently outperforms strong baselines—including SimCSE, E5, and INSTRUCTOR—while retaining fast cosine inference. Extensive ablations confirm that both prompt partitioning and LLM-enhanced supervision are pivotal to the observed gains in multi-task sentence representation learning.