The continued release of increasingly realistic image generation models creates a demand for synthetic image detectors. To build effective detectors, we must first understand how factors like data source diversity, training methodologies and image alterations affect their generalization capabilities. This work conducts a systematic analysis and uses its insights to develop practical guidelines for training robust synthetic image detectors. Model generalization capabilities are evaluated across different setups (e.g., scale, sources, transformations), including real-world deployment conditions. Through extensive benchmarking of state-of-the-art detectors across diverse and recent datasets, we show that while current approaches excel in specific scenarios, no single detector achieves universal effectiveness. Critical flaws are identified in detectors and workarounds are proposed to enable practical applications that enhance accuracy, reliability and robustness beyond current limitations.