Building a user model that incorporates diverse tasks remains a big challenge. While continual learning offers an alternative to multi-task learning by eliminating the need for retraining on all past tasks, prior works train the whole network backbone along with task-specific masks, which becomes computationally inefficient. Recent prompt-based parameter-efficient continual user modeling (PECUM) addresses this challenge by training only a few parameters, thus reducing the training cost. However, prompt tuning can yield homogeneous task embeddings and converge slowly compared to adapters. Hence, we propose a novel framework to integrate SVD-decomposed low-rank adapters into continual user modeling, which can be interpreted as a relaxed mixture of rank-1 experts. We further develop a novel attention framework that selectively weighs experts trained by semantically similar past tasks, and we jointly learn their attention coefficients along with newly added adapters, enabling interference-free knowledge transfer. We show the effectiveness of our proposed method on two real-world datasets.