Class-incremental learning (CIL) is designed for allowing models to adapt to newly received knowledge while not forgetting previously learned knowledge without access to old training samples. In CIL, pre-trained models (PTMs) have become promising solutions to mitigate forgetting due to their inherent powerful feature extraction capabilities. A surge of research has focused on parameter-efficient tuning, a process in which only a small portion of the parameters are fine-tuned to adapt to the data distributions of downstream tasks. However, existing methods neglect the sample difficulty in the fine-tuning stage. PTMs assign equal weights to all available samples. Thus, PTMs tend to utilize class-specific features to assist with the classification tasks. Leading to overfitting the data distributions of downstream tasks when the training samples are limited. In light of this, we first introduce a novel sample difficulty measurement in CIL based on the relative Mahalanobis distance. The measurement achieves high consistency between human vision and the sample difficulty scores. Furthermore, we propose SDR, a Sample Difficulty-aware Regularizer to adaptively adjust the sample weights. SDR directs PTMs to focus more on samples with higher difficulty scores, thus enhancing generalizability and performance. In addition, we report the experimental results in 45 settings from seven datasets. Extensive experiments validate the effectiveness and superiority of SDR.