ViT-ED: Transformer Network for Image Similarity Measurement

Measuring image similarity correctly and reliability is a critical requirement with profound implications across various applications, including puzzle reconstruction and historial document retrieval. In this work, we introduce a new deep neural network called ViT-ED, which stands for Vision Transformer with Encoder and Decoder network, to solve this task of image similarity estimation. By utilizing the attention and cross-attention mechanisms from the Transformer architecture, ViT-ED is capable of incorporating both global and local dependencies between patches from the two input images to make better similarity measurement. Experimental results on benchmark datasets of the two related problems: puzzle reconstruction and image retrieval, show that our ViT-ED model significantly outperforms state-of-the-art approaches on these tasks, e.g. ViT-ED achieves \(19\%\) of improvement in terms of mean Average Precision over state-of-the-art approach on the HisFragIR20 benchmark dataset. These results suggest that ViT-ED could be a strong candidate to solve image similarity related problems.