Quora is a crowdsourced platform for questions and answers. It has a big collection of user-created content. This includes science, technology, medicine, and business. We can use Quora in Retrieval-Augmented Generation (RAG) to enhance large language models (LLMs). We can do it by using its structured question-answer pairs as extra knowledge. To integrate Quora data into an RAG pipeline, we can use transformer-based embedding models. A vector database can store these embeddings. This helps to quickly find answers based on similarity searches. Integrating Quora into RAG brings benefits, but challenges remain. Using Application Programming Interface (API) of services that provide similarity search by LLMs can be very expensive. Also, Quora content often contains biases or low-quality answers. This work uses a hybrid approach and implements the question pair similarity locally without using APIs to reduce the cost of creating the RAG pipeline with LLMs. So, we can implement traditional machine learning models locally and provide strong filters to choose the best questions for LLM. In this work, we review and apply adaptive filtering algorithms to rank Quora responses based on similarity scores and language quality.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Enhanced Retrieval-Augmented Generation (RAG) with Question Similarity Filtering for QA Systems

  • Maryam Heidari,
  • Samira Zad,
  • Kianoosh Boroojeni,
  • Noel Philip,
  • Rohan Varghese

摘要

Quora is a crowdsourced platform for questions and answers. It has a big collection of user-created content. This includes science, technology, medicine, and business. We can use Quora in Retrieval-Augmented Generation (RAG) to enhance large language models (LLMs). We can do it by using its structured question-answer pairs as extra knowledge. To integrate Quora data into an RAG pipeline, we can use transformer-based embedding models. A vector database can store these embeddings. This helps to quickly find answers based on similarity searches. Integrating Quora into RAG brings benefits, but challenges remain. Using Application Programming Interface (API) of services that provide similarity search by LLMs can be very expensive. Also, Quora content often contains biases or low-quality answers. This work uses a hybrid approach and implements the question pair similarity locally without using APIs to reduce the cost of creating the RAG pipeline with LLMs. So, we can implement traditional machine learning models locally and provide strong filters to choose the best questions for LLM. In this work, we review and apply adaptive filtering algorithms to rank Quora responses based on similarity scores and language quality.