Inspired by large-scale data pre-training, the Segment Anything Model (SAM) has emerged as a robust and adaptable framework that has significantly pushed the boundaries of image segmentation. Although SAM’s flexibility is widely recognized, the exploration of tailoring this model for specific visual concepts in remote sensing, such as automatically segmenting swimming pools in extensive image collections, has not been fully developed. To bridge this gap, we present the Context Remote Sensing Segment Anything Model (CRS-SAM), a specialized adaptation of SAM crafted for the semantic segmentation of remote sensing imagery. CRS equips SAM with a Feed Forward neural Network (FFN) -based similarity calculator, enabling SAM to learn from one image and apply its knowledge across multiple images. The workflow is as follows: an image and its corresponding mask are input into the network. The image is processed through SAM’s encoder to obtain feature embeddings, with the mask used to select the pertinent set of context feature embeddings. Subsequently, a test image undergoes the same encoding process to derive its feature embeddings, which are then fed into the FFN with context to compute a confidence map. Finally, preset guide points are used to extract prompt points from the test image, which are then input into SAM’s prompt encoder to obtain prompt embeddings. These prompt embeddings, combined with the image feature embeddings, are fed into the decoder to produce the ultimate target mask, thereby achieving efficient adaptation of SAM’s capabilities for remote sensing tasks with minimal training required. To assess the efficacy of this method, we lightly retrained our FFN model on the LoveDA dataset, demonstrating its remarkable in-domain and cross-domain transferability. This not only illustrates the model’s versatility but also underscores its potential for a wide range of applications in the remote sensing domain, where the ability to swiftly and accurately segment specific visual concepts without extensive manual prompting is of immense value.