Quora Question Duplication Detection: An ML Approach for Identifying Semantically Equivalent Questions

This research paper delves into the field of Natural Language Understanding (NLU) by tackling the problem of identifying duplicate questions, using the Quora dataset as the primary source of data. Our study involves a comprehensive examination of the dataset and the application of machine learning techniques, specifically Random Forest and XGBoost models. Importantly, our results emphasize the efficiency of a straightforward CBOW neural network model, outperforming the effectiveness of more complex recurrent and attention-based models. The previous problem on Quora, characterized by numerous duplicate questions, has led to increased ambiguity and a diminished user experience. We also rigorously conduct error analysis, revealing subtleties and subjectivity in the dataset's labeling process. This investigation underscores the value of neural network-based approaches in tackling the complexities of duplicate question detection, which significantly contributes to the broader landscape of NLU research.