Pre-trained vision-language models (VLMs) have achieved remarkable performance across various downstream tasks. To further enhance the performance of VLMs on downstream tasks, adapter tuning has emerged as an efficient approach, enabling effective adaptation with minimal parameter updates. However, existing methods are susceptible to overfitting on task-specific data distributions, resulting in suboptimal performance. To address this challenge, we propose a novel framework, knowledge-decoupled adapter tuning (KDA-Tuning), which efficiently adapts VLMs to downstream tasks while preserving generalizability. KDA-Tuning employs a knowledge-decoupled adapter with a dual-branch structure consisting of one branch for general knowledge and another for task-specific knowledge, avoiding conflicts between their optimization. Additionally, we introduce corresponding supervision, including the general knowledge retention loss to encourage the models to preserve general knowledge, and the task-specific knowledge constraint loss to enhance cross-modal alignment for distinguishable feature representations. Extensive experiments demonstrate that KDA-Tuning outperforms existing methods on typical benchmarks.