Scholarlink

<p>Duplicate questions on crowd-sourced question and answer websites such as Quora create redundancy and make information retrieval inefficient. This research conducts a systematic comparative analysis of machine learning and deep learning models for detecting semantic similarity in questions. Using the Quora Question Pairs dataset, we evaluate a spectrum of models: a classical TF-IDF baseline, feature-engineered Random Forest and XGBoost, a Siamese Manhattan LSTM (MaLSTM), and a fine-tuned BERT model. The study reveals a clear performance hierarchy. A key finding is that classical models with a limited set of hand-crafted linguistic features underperformed the simple TF-IDF baseline. While the MaLSTM network showed moderate improvement, the fine-tuned BERT model was unequivocally superior, achieving a statistically significant accuracy of 86.26%. This highlights the critical role of deep contextual embeddings for this task. However, BERT’s state-of-the-art performance comes at a significant computational cost, revealing a crucial trade-off between accuracy and resource efficiency. These findings provide a pragmatic guide for designing effective and scalable duplicate question detection systems.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Semantic similarity in community forum questions: Case study on Quora dataset

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2025年9月1日

查看全文
  • Anuj Bhardwaj,
  • Raza Hasan,
  • Salman Mahmood

摘要

Duplicate questions on crowd-sourced question and answer websites such as Quora create redundancy and make information retrieval inefficient. This research conducts a systematic comparative analysis of machine learning and deep learning models for detecting semantic similarity in questions. Using the Quora Question Pairs dataset, we evaluate a spectrum of models: a classical TF-IDF baseline, feature-engineered Random Forest and XGBoost, a Siamese Manhattan LSTM (MaLSTM), and a fine-tuned BERT model. The study reveals a clear performance hierarchy. A key finding is that classical models with a limited set of hand-crafted linguistic features underperformed the simple TF-IDF baseline. While the MaLSTM network showed moderate improvement, the fine-tuned BERT model was unequivocally superior, achieving a statistically significant accuracy of 86.26%. This highlights the critical role of deep contextual embeddings for this task. However, BERT’s state-of-the-art performance comes at a significant computational cost, revealing a crucial trade-off between accuracy and resource efficiency. These findings provide a pragmatic guide for designing effective and scalable duplicate question detection systems.