Remote Sensing Object Detection (RSOD) presents unique challenges due to arbitrary object orientations, complex backgrounds, and data scarcity. Existing methods often rely on complex network architectures or auxiliary tasks for multi-task optimization, which increase model complexity and require additional data. In this work, we proposed an Adapter-based Few-shot Fine-Tuning framework for RSOD, termed AFFT. Specifically, to effectively capture global context and extract high-quality features, we adopted a pre-trained hierarchical Swin Transformer based on shifted windows attention mechanism as the backbone. This enables robust feature representation by efficiently modeling both local and global dependencies. Moreover, recognizing the limited availability of annotated remote sensing data, we designed a Group Equivariant Convolutions Multi-cognitive Visual Adapter (GCMONA) for few-shot fine-tuning. This approach significantly enhances model adaptability to new classes with minimal labeled samples, making it particularly well-suited for remote sensing scenarios. It is important that AFFT requires fine-tuning only 5% of the parameters with GCMONA, yet yields competitive or superior performance compared to full fine-tuning methods. Full-parameter training experiments on both DOTA-v1.0 and DOTA-v1.5 demonstrate the superior detection performance of our method in RSOD. Additionally, based on the pre-trained model in DOTA-v1.0, evaluation on our custom few-shot maritime dataset highlights the performance of the proposed few-shot fine-tuning technique when only limited data is available.