Topic Modeling with Latent Dirichlet Allocation (LDA) Using TF-IDF and Bag of Words
摘要
This paper explores LDA topic modeling using two distinct benchmark datasets: the 20newsgroup dataset and the Reuters21578 dataset. Our main objective is to compare the performance of the LDA model with term TF-IDF scores to baseline models. The study aims to demonstrate how the use of TF-IDF scores enhances the LDA model’s performance in the Reuters dataset to the point where it surpasses the baseline model (from coherence score (Cv) 0.4016–0.5281). However, in the 20newsgroup dataset, the LDA model with TF-IDF scores performs better than the baseline model (from coherence score (Cv) 0.5289–0.5971). These findings demonstrate the value of TF-IDF scoring in enhancing the quality of topic modeling, particularly in diverse textual datasets like newsgroups and news articles. They also stress how important it is to select appropriate text preparation techniques and modify parameters to enhance the performance of LDA models on various text corpora.