The rapid development of generative image models has raised concerns about misuse, especially in journalism and media. Therefore, developing tools for detecting fake images is essential. However, many current methods focus on short-term gains and lack long-term adaptability. This paper focuses on detecting deepfakes across various types of image data, such as faces, landscapes, objects, and scenes, using the visual-language CLIP model. Although CLIP has shown potential in deepfake detection, it has yet to clarify why it performs effectively in this task. Our analysis shows that CLIP’s combination of image features enhances the model’s generalization capability. By extracting image features trained for the deepfake detection task and generating captions through a text-decoding model, we demonstrate its effectiveness. Based on these findings, we introduce a novel method that enables the learning of forgery features and semantic features to improve generalization in image forgery detection. Extensive experiments show that our method achieves an accuracy of 98.3% on GAN-generated datasets and 95.9% on previously unknown diffusion model datasets. Our code is available at: https://github.com/genkerizer/CGL .