Deepfake images, with artificially generated or partially altered fake content, appear more and more frequently in everyday life. In order to avoid misleading or misinforming potential recipients of such data, deepfake detection techniques are continuously developed and improved. Such approaches need to reliably detect images from various generation methods based on modern deep neural network architectures, such as generative adversarial networks (GAN) or diffusion models (DM). In this paper, we present an experimental analysis of the challenges associated with the detection of fake images generated from different sources. Inspired by previous research in this area, our experiments demonstrate the results of training classifiers using only the images generated by chosen diffusion models, as opposed to training detection models exclusively on images produced by GANs. We conducted the experiments on GenImage dataset using the state-of-the-art CLIP+ViT-L/14 backbone as feature extractor, combined with the recently proposed frequency masking approach. A comparative analysis revealed that a training set made up of diffusion model images can increase the average precision across a wide range of generators, resulting in a higher degree of model generalization. Moreover, training the detectors on DM data can result in relatively high accuracy in detecting also GAN-based deepfake images.