Systematic Evaluation of Different Approaches on Embedding Search

This paper presents a comparative analysis of various methods of embedding search in insurance documents. The evaluation focuses on different SentenceTransformers models integrated within LangChain. Further, we assess the performance of the text-embedding-ada-002, Vicuna-13B, and a fine-tuned variant of the Vicuna-13B within the same pipeline. In an effort to broaden our evaluation, we also investigate a custom HuggingFace pipeline that compares the embeddings generated at the token level. Our findings reveal that the text-embedding-ada-002 model provides the most favorable results. Furthermore, in terms of open-source alternatives, the SentenceTransformers model all-miniLM-L12-v2 outperforms other models. To our knowledge, there is currently no published research addressing retrieval using embeddings on German insurance documents, thus underscoring the unique relevance of this study in this niche domain.