VAeViT: Fusing Multi-views for Complete 3D Object Recognition

The Vision Transformer (ViT) emergence has ushered in a new era in 2D image classification, yet the realm of 3D object recognition presents distinct challenges. To bridge this gap, we introduce VAeViT, a pioneering hybrid model that seamlessly integrates the strengths of Vision Transformers and Variational Autoencoders (VAE). VAeViT leverages VAE’s efficiency in feature representation to encode the 3D object views into a lower-dimensional latent space and ViT’s superiority for semantic feature extraction and incorporating the positional information to aggregate those low-dimension views into the most representative features for the whole object, providing comprehensive perception of 3D multi-view objects that enables accurate object recognition from any posture. Our experiments on the ModelNet10 and ModelNet40 benchmark datasets show that VAeViT achieves superior accuracy compared to existing models, even when utilizing smaller datasets. This highlights VAeViT’s potential to improve 3D object recognition in various computer vision applications significantly. For reproducibility, our code is available at https://github.com/AhmedYEita/VAeViT .