In recent years, the progress of deep learning has significantly promoted the development of RGB-D saliency object detection. However, existing methods are inefficient in feature processing. They either simply combine modalities through basic weighting or concatenation, failing to fully exploit complementary information, or they overlook local feature interactions, leading to detail loss. To address these issues, this paper proposes a more efficient approach during the encoder stage—the Cross-Modal Interaction Enhancement Module (CIE), which enhances local interactions to achieve more refined feature fusion, thereby effectively integrating RGB and depth features. During the decoder stage, we introduce the Hybrid Dilated Attention Fusion Module (HDA), which effectively processes three modal features—RGB, depth, and fusion—thereby enhancing the model’s performance. Experimental results demonstrate that our method outperforms other state-of-the-art approaches on four benchmark datasets: STERE, NJU2K, NLPR, and SSD, significantly enhancing performance.