Dynamic sparse attention for lightweight multimodal sensor fusion on edge devices
摘要
Edge computing and multimodal sensor fusion together call for attention modules that fit inside tight hardware budgets. Standard dense attention, however, scales quadratically with sequence length and is hard to run in real time on devices with limited compute and memory. We address this with a Dynamic Sparse Attention (DSA) module and a matching deployment pipeline aimed at edge-side fusion. The central idea is an entropy-conditioned predictor that produces input-adaptive masks for each attention head, so computation is concentrated on the cross-modal pairs that carry most signal and pruned elsewhere. A heterogeneous feature alignment block maps tokens from different modalities into a shared space through modality-adaptive normalization, which compensates for the wide gaps in the raw sensor distributions. On the deployment side, we combine block-sparse reformulation, mixed-precision quantization, and hardware-aware scheduling so the dynamic sparsity actually translates into faster execution. Tests on nuScenes and KITTI show that DSA stays within 0.9% mAP of dense Transformer baselines while cutting inference latency by more than 50% and peak memory by roughly 45% on an NVIDIA Jetson Orin NX. Ablation runs confirm that each component pulls its weight, and the model degrades gracefully when one sensor stream is dropped at runtime.