Filter pruning is a vital technique for effectively compressing the convolutional neural networks to comply with hardware constraints. Traditional methods measure the filter importance by averaging the importance of samples, which fails to properly decouple the influence of the unique characteristics of each sample on the filter, leading to inaccurate measurement of filter importance. Consequently, we propose a novel residual energy metric to measure the filer importance by comprehensively considering the characeristics across different samples. Specifically, the residual energy is obtained by performing a series of discrete cosine transform of feature maps, and followed by adding the original energy of feature maps to serve as the residual energy metric. Then, the filter importance is computed by its metric differences across samples. Finally, filers of small importance are pruned. Experimental results on two challenging benchmarks of various models clearly demostrate the superiority of our method. On both the CIFAR-10 and ImageNet datasets, we attained higher accuracy compared to the baseline network when the pruning rate approached 50%. Notably, on the ImageNet dataset, we achieved superior results with an extreme pruning rate, reducing the parameters of the ResNet-50 model by 68.6% and the FLOPs by 76.7%.