RGB-Thermal Video Object Detection (RGBT VOD) has emerged as a promising solution for visual perception under adverse illumination conditions. However, existing methods often assume well-aligned multimodal inputs, which limits their robustness in real-world scenarios where cross-modal misalignment and sensor disparities are prevalent. To address these challenges, we propose CAFNet, a novel Cross-modal Alignment and Fusion Network for misaligned RGBT VOD. Specifically, we introduce a Temporal Deformable Fusion (TDF) module to enhance temporal consistency across video frames, a Top-k Effective Spatial Attention (TESA) module to refine spatial saliency under modality noise, and a Cross Alignment Fusion (CAF) module that integrates deformable alignment with cross-modal attention for robust feature fusion. Our framework enables accurate and efficient object detection without relying on strict alignment or pixel-level correspondence. Extensive experiments on the UVT-VOD2024 dataset demonstrate that CAFNet achieves state-of-the-art performance in terms of both accuracy and speed, significantly outperforming existing RGBT VOD models under various misalignment and environmental conditions.