Pre-trained Vision Transformers (ViTs) face challenges in effectively extracting features for multimodal tasks due to their initial training on single-modality data. Moreover, fine-tuning ViTs for such tasks requires adjusting a significant number of parameters, leading to substantial training and storage costs. In audio-visual multimodal learning, a major challenge is seamlessly integrating both audio and visual cues into the transfer learning process, which can be particularly challenging with pre-trained ViTs. In this paper, we introduce a cross-modal audio-visual parameter-efficient adapter (AV-PEA). By integrating AV-PEA into a frozen ViT, the transformer becomes adept at processing audio inputs without prior audio pre-training, incorporating a minimal set of trainable parameters into each block. AV-PEA also facilitates the exchange of essential audio-visual cues between modalities while keeping the majority of the transformer’s parameters frozen, thus preserving its established knowledge base. Experimental results demonstrate that AV-PEA consistently achieves superior or comparable performance to state-of-the-art methods in various audio-visual downstream tasks, including audio-visual event localization, audio-visual question answering, audio-visual segmentation, audio-visual retrieval, and audio-visual captioning. Additionally, AV-PEA allows seamless integration into these tasks while maintaining a minimal number of trainable parameters, typically accounting for less than 3.7% of the total parameters per task.