Low-bit quantization, specifically with 4-bit or 2-bit precision, involves compressing the weights and activations of deep learning models into representations with fewer bits, thereby significantly reducing computational and memory requirements. A key challenge in low-bit quantization is quantization imbalance, where certain weights or activations are overly compressed. This problem is especially acute for outliers, which are critical parameters that carry significant learning information. Existing approaches fall short in effectively managing outliers, potentially leading to a decline in the model's overall performance, particularly in 2-bit quantization scenarios. In this paper, we propose FECG-KD, a fisher enhanced and clustering guided knowledge distillation framework, which aims at addressing the outlier issue in extremely low-bit quantization. During the post-training quantization process, FECG-KD utilizes distilled gradient information and clustering techniques to more effectively preserve critical parameters. Moreover, we propose a calibration set augmentation approach that enhances gradient information, thus further optimizing the clustering process under low-bit quantization. Our framework achieves significant improvements in both quantization accuracy and model performance. In the 2-bit quantization scenario, FECG-KD outperforms the baseline in accuracy, with improvements of 1. 61%, 1. 91%, 1. 65%, 2. 17% and 3. 05% for ResNet-18, ResNet-50, MobileNetV2, RegNetX-600MF, and MNasx2, respectively.