Test-time adaptation (TTA) improves model performance under distribution shifts by enabling on-the-fly adaptation during inference. However, many existing methods tend to overfit recent domains while forgetting previously acquired knowledge, limiting their long-term effectiveness in dynamic or mixed-domain scenarios. To tackle this challenge, we propose Mashup, a novel TTA approach that balances adaptation and knowledge retention without requiring source data. Mashup introduces a selective adaptation trigger based on Jensen-Shannon divergence (JSD), which detects significant distribution changes and avoids unnecessary updates. To preserve past knowledge, we maintain a compact buffer of historical knowledge vectors that capture adaptation dynamics. These vectors are adaptively fused with current information through a JSD-guided mechanism, enabling the model to integrate both recent and historical domain knowledge. Extensive evaluations on various datasets under both sequential and mixed-domain settings show that Mashup achieves state-of-the-art performance while effectively mitigating catastrophic forgetting.