Infrared-RGB dual-modality small object detection is critical for all-weather aerial applications such as traffic monitoring and search-and-rescue missions. However, existing advanced methods like YOLOv12 face great challenges in handling complex backgrounds and significant scale variations in drone imagery. Moreover, many current cross-modality alignment and fusion strategies incur considerable parameter overhead. To address these issues, we propose DMR-YOLO (Dual-Modality Robust YOLO), a lightweight yet effective detection framework for small object detection in infrared-RGB UAV Imagery. The newly proposed DMR-YOLO achieves parameter-efficient multi-scale feature sharing via a Dynamic Cross-scale Adaptive Task-Decoupled Head (DCAT-Head) and introduces a Dynamic Modality-Aware Fusion (DMAF) module for modality-specific feature enhancement. The DMAF module integrates dynamic range compression with task-conditioned affine transformation to dynamically align and refine features across infrared (IR) and visible (RGB) modalities. Additionally, a Parallel Adaptive Downsampling (PAD) module is employed to preserve semantic integrity during feature fusion. Extensive experiments on the DroneVehicle (IR-RGB), POG (RGB), and HIT-UAV (IR) datasets demonstrate the superior performance and robustness of DMR-YOLO in UAV-based small object detection tasks.