RouteLlama: Proactive Disentanglement for Robust Multi-domain Text Mining

Large Language Models (LLMs) degrade on heterogeneous, multi-domain corpora because gradients from one domain overwrite another, a core obstacle in KDD pipelines. Most remedies respond after a gradient conflict. RouteLlama instead enforces a preventive two-stage procedure that reshapes the representation space before any downstream task updates. Stage 1 trains PRISM (Partitioned Retrieval of Inter-/intra-domain Sample Mining) to disentangle domains through partitioned memories and a dual-temperature contrastive loss. Stage 2 freezes this geometry and learns a lightweight task adapter, keeping later updates from re-entangling domains. Across Amazon-29, MNLI, and SNLI, RouteLlama reduces gradient interference (cosine \(-0.45 \rightarrow -0.05\) ), improves multi-task performance (+1.1 pp Amazon macro-F1) and outperforms the strong single-task baseline (+0.6 pp), while locking MNLI matched/ mismatched accuracy at a 0.1 pp parity as a Pareto guardrail against negative transfer. The amortized ANN design keeps overhead near linear in the number of domains, making the approach practical for tens to low hundreds of domains.