Class-Incremental Learning (CIL) aims to acquire knowledge of new classes while mitigating old-class catastrophic forgetting. Recently, with the development of foundation models (FMs), e.g., CLIP, leveraging them for incremental learning has become a more promising and practical endeavor. However, existing CIL methods based on FMs simply adopt common CIL methods or fine-tuning methods, without fully utilizing the rich prior knowledge of FMs, especially for the trained semantic space. This semantic space is trained by a large-scale dataset with a wide range of categories and diverse scenarios. To address this core strength of FMs, we propose a novel framework, termed VFGR (Virtual Feature Generation and Replay), which generates virtual features for both new-class learning and old-class replay. As for the new-class learning, we sample inliers of virtual features from a Gaussian kernel with small covariance to enhance the diversity of training data. Meanwhile, we sample outliers of virtual features with a large covariance to compact the decision boundary for each class, so as to leave enough space for incremental learning. As for the old-class replay, we mitigate knowledge forgetting by constructing a feature-level statistical distribution, which generates old class-wise virtual features during replay. Extensive experiments on prevalent CIL benchmarks, i.e., CIFAR100 and ImageNet100, showcase the superiority of VFGR, which improves 8.96% average accuracy (Avg) compared to CLIP without continual learning and surpass the recent state-of-the-art method (AttriCLIP) by 1.46% in Avg.