Contrastive Vision-Language Pre-training (CLIP) has shown groundbreaking results in zero-shot and few-shot learning. Its success in the 2D domain sparks the curiosity to explore if CLIP, pre-trained with large-scale image-text pairs, can be adapted for 3D recognition. This chapter discusses PointCLIP and PointCLIP V2 and shows that it is indeed possible to align CLIP-encoded point clouds with 3D category text representations. PointCLIP projects a point cloud into multi-view 2D images, which helps in revealing geometric information from 3D to 2D. For better extraction of global features, PointCLIP features an interview adapterInter-view adapter, significantly boosting the performance by just fine-tuning the lightweight adapter. After that, PointCLIP V2 is discussed to further improve the performance and enhance the generalization ability. V2 employs a shape projection module for the visual end, helping to synthesize more realistic depth maps and narrowing the domain gap between projected point cloudsPoint clouds and natural images. Simultaneously, V2 combines CLIP and large language models (LLMs) to generate 3D-specific text, improving the feature extraction of CLIP’s textual encoder. PointCLIP V1 and V2 demonstrate remarkable generalization capabilities in unified 3D open-world learningUnified 3D open-world learning, paving the way for better in-depth understanding and possible future improvements and extensions.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Adapting CLIP for 3D Understanding

  • Xiangyang Zhu,
  • Renrui Zhang,
  • Peng Gao

摘要

Contrastive Vision-Language Pre-training (CLIP) has shown groundbreaking results in zero-shot and few-shot learning. Its success in the 2D domain sparks the curiosity to explore if CLIP, pre-trained with large-scale image-text pairs, can be adapted for 3D recognition. This chapter discusses PointCLIP and PointCLIP V2 and shows that it is indeed possible to align CLIP-encoded point clouds with 3D category text representations. PointCLIP projects a point cloud into multi-view 2D images, which helps in revealing geometric information from 3D to 2D. For better extraction of global features, PointCLIP features an interview adapterInter-view adapter, significantly boosting the performance by just fine-tuning the lightweight adapter. After that, PointCLIP V2 is discussed to further improve the performance and enhance the generalization ability. V2 employs a shape projection module for the visual end, helping to synthesize more realistic depth maps and narrowing the domain gap between projected point cloudsPoint clouds and natural images. Simultaneously, V2 combines CLIP and large language models (LLMs) to generate 3D-specific text, improving the feature extraction of CLIP’s textual encoder. PointCLIP V1 and V2 demonstrate remarkable generalization capabilities in unified 3D open-world learningUnified 3D open-world learning, paving the way for better in-depth understanding and possible future improvements and extensions.