Supervised learning models achieve notable performance in various downstream tasks such as image classification, at the cost of high computational overhead due to their data-hungry approach during the training or fine-tuning processes. A task-incremental approach can reduce the requirement of training data, as well as the possibility of overfitting. However, catastrophic forgetting is a common problem in any continual learning method. Addressing the problem of balancing plasticity and stability remains a challenge, despite several attempts to balance between stability and plasticity. To achieve a better stability-plasticity trade-off, we connect an additional auxiliary network to promote plasticity to an already learned model with enhanced stability. Through dynamic adjustment of synaptic weights based on data relevance, the proposed model augments plasticity while preserving critical knowledge, effectively mitigating catastrophic forgetting. Further, we maintain a dual feature space for the past and current tasks, for an enhanced balanced trade-off. Extensive experiments on benchmark datasets such as CIFAR-10, CIFAR-100, and TinyImageNet, demonstrate the superior accuracy and stability of our model compared to the existing methods. Overall, the proposed approach offers a robust solution for continual learning, providing enhanced adaptability and performance in a dynamic learning environment.