Unmanned aerial vehicles (UAVs) play a crucial role in tasks such as target detection due to their speed and high degree of freedom. However, the limited computing power of UAVs and the complexity of their communication environments pose significant challenges for real-time tasks. To ensure accuracy and reduce the amount of transmitted data, we propose a novel multi-task joint cross-modal semantic communication system, named “CMSCDet”, for object detection and image transmission. In this system, the UAV captures both visible and infrared images, which are fused in the mid-term and transmitted to a server using joint source-channel coding (JSCC). The system employs multi-level feature extraction, fusion, and reconstruction techniques to perform both object detection and image reconstruction tasks simultaneously. Simulation results demonstrate that the proposed method achieves performance improvement over 10% compared to the traditional JPEG+LDPC communication method and exhibits significant advantages over other benchmarks.