The paper outlines a pipeline for the removal of transient objects from orthophotos to enhance the clarity and utility for orthophotos in military and civilian geo-databases generation as the main application. The presented deep-learning-based pipeline includes detecting the objects of interest, masking them out, and using the image and an enhanced inpainting mask to fill in these areas seamlessly. The approach combines semantic segmentation, utilizing an adapted DeepLabv3+ model, with shadow detection using Particle Swarm Optimization, and concludes with a generative inpainting process using a three-stage Generative Adversarial Network (3GAN) system for edge, segmentation, and texture inpainting. This method is applied to a well-known remote sensing dataset for detailed analysis, highlighting the integrated approach’s effectiveness in creating realistic, cleaned-up orthophotos.