Foreign object detection is crucial for railway safety, preventing accidents and ensuring smooth operations. Current railway foreign object detection methods face two significant challenges: the scarcity of annotated real-world data and the inability to adapt to complex scenarios. This paper proposes a novel FARD (Fully Automated Railway Anomaly Detection System) approach to address these issues. FARD incorporates two key components: (i) A Diffusion model with inpainting technique to generate a diverse and realistic auxiliary dataset of railway anomalies, effectively representing real-world outliers. (ii) An integrated framework combining traditional object detection pipeline with reconstruction-based anomaly detection module for robust foreign object detection in railway environments. Experimental results demonstrate that FRAD outperforms traditional object detection methods in identifying anomalies on rail tracks by a large margin. This research offers a robust, data-efficient solution for railway foreign object detection that works well even with limited initial data.