In the field of gesture recognition, RGB-based visual methods often exhibit performance instability under varying lighting conditions and complex backgrounds, while methods relying solely on depth information struggle to capture rich textural details. Although multi-modal fusion is a recognized solution, systematic comparative studies on when to fuse and how to fuse, particularly within lightweight networks, remain limited. In light of this, this paper presents a systematic empirical analysis of fusion schemes for lightweight RGB-D gesture recognition using MobileNetV2 as the backbone. We systematically evaluate early and late fusion, and for mid-fusion approaches, we conduct an in-depth comparison of six distinct strategies: concatenation, channel-wise attention, cross-modal attention, spatially adaptive fusion, SE attention, and CBAM attention. Extensive experiments on the EgoGesture dataset demonstrate that placing the fusion operation at deep semantic layers (specifically, stage five) most effectively aggregates complementary information. Furthermore, among the strategies, the channel-wise attention-based approach achieves the best performance. Our final optimized model achieves a Top-1 accuracy of 95.34% on the 83-class gesture recognition task, which is 0.94% points higher than the RGB-only baseline (94.40%) and 1.45% points higher than the depth-only baseline (93.89%). This study provides practical insights and design guidelines for developing efficient RGB-D gesture recognition systems.