Cross-modal homography estimation is a fundamental task in computer vision. However, the significant modality gap between multimodal images poses a considerable challenge for this task. In this paper, we propose a novel framework, named Disentangled Representation Learning for Cross-modal Homography Estimation (DRLHomo), to address this challenge. Specifically, we design a feature disentanglement network to explicitly extract modality-shared and modality-unique features of image pairs using shared and unique feature encoders, respectively. Similarity loss and difference loss are employed to supervise this process. Furthermore, to prevent pathological disentanglement, where shared features are dominated by noise and unique features capture all modality information, we incorporate a decoder with reconstruction loss to enhance feature disentanglement. Ultimately, the homography is estimated based on the obtained modality-shared features. Extensive experiments on the GoogleMap and GoogleEarth datasets validate the effectiveness of DRLHomo in mitigating the modality gap and improving cross-modal homography estimation.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

DRLHomo: Disentangled Representation Learning for Cross-Modal Homography Estimation

  • Tianming Li,
  • Zhen Zhou,
  • Qing Zhu,
  • Jianqiao Luo,
  • Yaonan Wang

摘要

Cross-modal homography estimation is a fundamental task in computer vision. However, the significant modality gap between multimodal images poses a considerable challenge for this task. In this paper, we propose a novel framework, named Disentangled Representation Learning for Cross-modal Homography Estimation (DRLHomo), to address this challenge. Specifically, we design a feature disentanglement network to explicitly extract modality-shared and modality-unique features of image pairs using shared and unique feature encoders, respectively. Similarity loss and difference loss are employed to supervise this process. Furthermore, to prevent pathological disentanglement, where shared features are dominated by noise and unique features capture all modality information, we incorporate a decoder with reconstruction loss to enhance feature disentanglement. Ultimately, the homography is estimated based on the obtained modality-shared features. Extensive experiments on the GoogleMap and GoogleEarth datasets validate the effectiveness of DRLHomo in mitigating the modality gap and improving cross-modal homography estimation.