With the development of Artificial Intelligence Generated Content (AIGC), fake image detection has become increasingly challenging. Also leveraging the advanced capabilities of large language models (LLMs) in sequence prediction, we propose a novel perspective on fake image detection by fine-tuning pure LLMs. We introduce Fake-GPT, a LLM with 7 billion parameters which can differentiate between real and fake images. Unlike conventional image processing models, our approach directly process RGB pixel values without relying on any position embedding and visual-language feature alignment, thereby reducing model complexity and processing steps. Our research demonstrates the effective application of LLMs in detecting fake images, thereby expanding their application in non-textual domains. Extensive experiments conducted on various deepfake datasets show that Fake-GPT achieves competitive results compared with conventional image processing models, underscoring its potential as a new paradigm in the realm of image authentication.