The proliferation of advanced deepfake technologies and synthetic media has ignited extensive research in the realm of image forgery detection. In response to the lack of generalizability of past detection methods, prior-based deepfake detection methods have recently been proposed to leverage the feature space of pre-trained models for robust classification. However, they only use the final output of the prior model directly, without fully harnessing the efficacy of the prior. To this end, our approach delves into the internal knowledge structure of prior models and proposes a framework for deepfake detection employing both forward activation feature maps and backward convolutional kernel gradients of each layer as the internal representation of the prior. Further, leveraging the correlation characteristics of internal representations within the prior model, we introduce a novel Internal Prior Correlation (IPC) based deepfake detection method. By modeling the correlation of internal priors in both spatial and channel dimensions, we more accurately capture the subtle differences between fake and real images in the high-level feature space. Extensive experiments on both classic and state-of-the-art datasets validate the effectiveness and superiority of our method.