Real-world data streams are unbounded, evolving, and often temporally dependent, posing key challenges such as concept drift, catastrophic forgetting, and the need for continuous learning. These challenges define a scenario called Evolving Streaming Time Series (ESTS). While Streaming Machine Learning enables fast adaptation and Continual Learning preserves past knowledge, neither alone addresses all ESTS needs. In the context of the emerging Streaming Continual Learning paradigm, this work introduces Continuous Piggyback (cPB), a novel method designed for ESTS. cPB adapts a pre-trained, frozen backbone model using learnable masks learned continuously. It uses transfer learning to boost adaptation to concept drifts and integrates temporal modeling through Recurrent Neural Networks. It supports both binary and real-valued masks and stores them persistently to avoid forgetting. Experiments on synthetic and real-world streams show that cPB outperforms established SML baselines and continuous RNNs, improving adaptation to concept drift while mitigating forgetting.