Deep Neural Networks (DNNs) are dominant in the field of Computer Vision. However, their high computational complexity makes deployment on resource-limited devices challenging. Therefore, techniques, such as pruning, are desired to compress DNN models without accuracy degradation. Most conventional pruning methods significantly reduce the accuracy of the model. Therefore, the pruned models need time-consuming retraining to recover the accuracy. In many real scenarios, such costly retraining is impractical. To mitigate this problem, some methods attempt to preserve the accuracy by applying least squares reconstruction after pruning. Their pruning and reconstruction minimize the layer-wise error, which may not necessarily reflect the model performance on the target task. Therefore, their ability to preserve the accuracy is limited. In this paper, we propose Performance Preserving Pruning (PPP). Unlike conventional methods that minimize the layer-wise error, PPP leverages the gradient of the loss to approximately minimize the error of the loss. Thereby, PPP preserves the performance of pruned models more effectively than conventional methods.