<p>Deepfakes have attracted considerable attention due to their detrimental impact, notably the production of convincing fake images and their unregulated dissemination. However, existing detectors suffer from a common defect that obsessing with low-level local artifacts or predefined forgery patterns while neglecting the intrinsic differences between the forged and genuine images. This leads to significant performance decline when the model encounters with unseen domains and forgeries. To address this issue, this paper proposes a novel framework, named as FTA-DFI, which leverages disentanglement learning and adversarial learning to analyze forgery traces amplified from the perspective of key facial features and frequency domain. FTA-DFI consists of the Fake Trace Amplification Strategy (FTAS) and the Distinctive Feature Identification Strategy (DFIS). FTAS suppresses method-specific features and identity information using the Face Features Random Mask and amplifies manipulated artifacts through the Frequency Domain Converter. DFIS employs the Feature Separation Disentangler and Intrinsic-Adversarial Learning modules to discern distinctive features between fake and genuine images, thereby further mitigating the overfitting of forgery-irrelevant information and specific forgery approaches. Extensive experimental results demonstrate that FTA-DFI outperforms current state-of-the-art models in generalization evaluations across mainstream datasets, validating the superior performance of the proposed architecture in terms of generalization capability. The code is available at <a href="https://github.com/zouzhengcs/FTA-DFI">https://github.com/zouzhengcs/FTA-DFI</a>.</p>