With the rapid growth of remote sensing applications in various fields, the need for accurate semantic segmentation of remote sensing images has become increasingly urgent. However, the difficulty in acquiring high-quality semantically labeled data has limited the effectiveness of traditional semantic segmentation models that rely on labeled data in remote sensing. This has resulted in a need for alternative approaches to improve the accuracy of remote sensing data analysis. This paper presents a self-supervised semantic segmentation framework based on image inpainting to tackle this challenge. The proposed framework enhances the model's generalization ability by using image inpainting as a pre-task, learning a fine pixel-level representation, and applying an adversarial training technique to locate salient pixels in the image as the regions to be restored. The accuracy of both image inpainting and semantic segmentation is further improved.