RGB-D salient object detection aims to segment the most obvious object by fusing the RGB and depth modalities information. Existing works have designed many modal fusion modules that successfully enhance the feature information volume and improve the detection effect. However, most of these methods ignore the modal information asymmetry and insufficient priors in complex situations. To this end, we propose a novel asymmetry cross-modal fusion and multi-task learning network (ACMFMTLNet) framework that introduces the asymmetry cross-modal fusion module in the encoder process and adds the end-to-end object detection task to the decoder process. The asymmetry cross-modal fusion module implements the channel attention mechanism to reduce feature redundancy on the RGB branch and use the channel concatenation to improve the depth branch information abundance. The decoder part of the proposed model couples the object location task and segmentation task together, and the location task will provide priors to assist the segmentation task in direct and indirect ways. Extensive experiments are conducted on seven benchmark datasets, demonstrating the ACMFMTLNet’s superiority over the other nine SOTA works.