<p>With the rapid progress of autonomous driving technologies, camera–LiDAR-based multi-modal 3D object detection has attracted increasing interest. BEV-based approaches have become a prevalent paradigm, in which image features are lifted into Bird’s Eye View representations via depth estimation and subsequently fused with LiDAR BEV features. Nevertheless, errors in depth estimation and inefficient fusion mechanisms remain major bottlenecks for detection accuracy. To address these limitations, we introduce a novel multi-modal fusion network for 3D object detection, consisting of two principal components: Geometric Pseudo-Image Feature Fusion and the Feature Fusion Encoder. In particular, the Geometric Pseudo-Image Fusion module embeds LiDAR-derived 3D structural information into image features, enabling camera representations to retain reliable geometric cues (derived from the pixel-wise alignment between LiDAR point clouds and images) even in the presence of depth estimation inaccuracies. Furthermore, during the transformation of image features into BEV space, sparse point cloud depth is incorporated as a prior to enhance depth estimation quality. The Feature Fusion Encoder dynamically regulates the contributions of both modalities, alleviates cross-modal feature misalignment, and adopts adaptive convolutional kernels to better handle objects of varying scales. Additionally, temporal information is leveraged to improve the detection of occluded objects. Extensive experiments conducted on the nuScenes dataset validate the effectiveness and robustness of the proposed method.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

GPFusion: enhancing multi-modal 3D object detection via geometric pseudo-image fusion and adaptive feature fusion

  • Hongxin Xue,
  • Zeru Fang,
  • Luyang Li,
  • Liqun Kuang,
  • Huiyan Han

摘要

With the rapid progress of autonomous driving technologies, camera–LiDAR-based multi-modal 3D object detection has attracted increasing interest. BEV-based approaches have become a prevalent paradigm, in which image features are lifted into Bird’s Eye View representations via depth estimation and subsequently fused with LiDAR BEV features. Nevertheless, errors in depth estimation and inefficient fusion mechanisms remain major bottlenecks for detection accuracy. To address these limitations, we introduce a novel multi-modal fusion network for 3D object detection, consisting of two principal components: Geometric Pseudo-Image Feature Fusion and the Feature Fusion Encoder. In particular, the Geometric Pseudo-Image Fusion module embeds LiDAR-derived 3D structural information into image features, enabling camera representations to retain reliable geometric cues (derived from the pixel-wise alignment between LiDAR point clouds and images) even in the presence of depth estimation inaccuracies. Furthermore, during the transformation of image features into BEV space, sparse point cloud depth is incorporated as a prior to enhance depth estimation quality. The Feature Fusion Encoder dynamically regulates the contributions of both modalities, alleviates cross-modal feature misalignment, and adopts adaptive convolutional kernels to better handle objects of varying scales. Additionally, temporal information is leveraged to improve the detection of occluded objects. Extensive experiments conducted on the nuScenes dataset validate the effectiveness and robustness of the proposed method.