Efficient topic modeling for large-scale community question and answer websites

Community Question and Answering (Q&A) websites hold a wealth of unstructured text. Analyzing this text using topic modeling can offer valuable insights into recent trends and technology within these communities. However, current topic modeling methods have limitations, as they often require default parameter tuning and struggle to handle large datasets effectively. We introduce a new topic modeling technique called LEFT (LDA Entropy TFIDF-based Fuzzy Modeling) to address these challenges. We evaluated LEFT using twenty-eight datasets from the Stack Exchange websites, including Artificial Intelligence (AI), Software Engineering (SE), Data Science (DS), Information Security (IS), and Quantum Computing (QC). In our study, we compared the performance of LEFT with two state-of-the-art techniques: Latent Dirichlet Allocation (LDA) (Blei et al., J Mach Learn Res 3:993–1022, 2003) and Fuzzy Latent Semantic Analysis (FLSA) (Karami et al., Int J Fuzzy Syst 20:1334–1345, 2018). Our findings indicate that LEFT outperforms existing techniques. It shows a 13.71% and 22.26% improvement in Silhouette score and CH score for large-scale datasets compared to the LDA model. Moreover, LEFT demonstrates significant improvements over FLSA and delivers comparable results to the LDA model for small-scale datasets.