Robust object detection in Unmanned Aerial Vehicle (UAV) surveillance is frequently compromised by dynamic environmental degradations such as low illumination, atmospheric obscurants (fog, smoke), and sensor failures. While multi-modal sensor fusion offers a potential solution by leveraging complementary data, existing approaches often rely on static integration rules that fail to adapt when specific sensors become unreliable. This rigidity can lead to “negative transfer,” where the inclusion of noisy or degraded data reduces performance below that of the single best sensor. To address this, this paper presents FuseFormerNet, an adaptive tri-modal fusion framework that robustly integrates RGB, thermal, and LiDAR data. The proposed architecture introduces a novel mid-level fusion mechanism comprising two key components: (1) a context-conditioned reliability gating module that dynamically regresses modality importance weights to suppress degraded sensors before fusion, and (2) a transformer-based fusion block that utilizes multi-head self-attention to facilitate content-aware feature refinement across modalities. The approach is benchmarked on the challenging R-LiViT dataset using a rigorous “detector-fair” protocol to isolate fusion benefits. Experimental results demonstrate that FuseFormerNet achieves 67.14% mAP@0.5, outperforming static tri-modal fusion by 1.41% and the RGB-only baseline by 4.80%. Crucially, the system exhibits superior robustness, maintaining high recall in darkness and synthetic smoke where single-modality detectors fail.