Continual learning remains a significant challenge in deep learning, as neural networks struggle to acquire new knowledge without compromising previously learned information, a phenomenon known as catastrophic forgetting. Class incremental learning is an implementation of continual learning that poses another big hurdle when requiring models to learn new classes, maintaining accuracy on previously learned classes without accessing past data. In this paper, we propose an approach to the class incremental learning problem that combines adaptive knowledge distillation with targeted parameter regularization based on importance estimation. Our method employs a multi-head architecture that allows for systematic expansion to accommodate new classes while preserving discrimination capabilities for previously learned ones. To mitigate catastrophic forgetting, we integrate a dual component loss: a distillation term to transfer previously learned knowledge to the current model and a parameter-specific regularization term derived from the Fisher information matrix to protect critical weights. Our experimental results on CIFAR-100 demonstrate superior performance compared to existing approaches, with particular advantages when learning new classes incrementally. Our proposed framework showcases the capability of continuous knowledge acquisition without compromising prior learning.