Need of OpenAI GPT-3 Embedding for Duplicate Text Detection to Improve Performance of CQA Website
摘要
Community-based question answering (CQA) platforms are websites that let users ask questions, get answers, and exchange information. One of these services’ biggest challenges is identifying duplicate questions and identifying high-quality answers from previous responses. Many efforts have been made to represent words as numerical vectors, including wordtovec. We converted preprocessed text input into a numerical vector by combining these techniques with OpenAI (artificial intelligence) embedding. Bidirectional encoder representations from transformers (BERT) is a model build by using encoder and transformer to analysis textual data which applied on QA system, sentiment analysis applications. Sentence-BERT and Distil-BERT are next version of BERT both used in same task. There are different generative pre-trained transformers (GPT) architecture like GPT-1, GPT-2, and GPT-3. We examine Sentence-BERT, Distil-BERT, and GPT-3 model on CQA system. Result demonstrate that any algorithm trained on OpenAI embedding yields excellent performance.