A Semi-automatic Method for Topic Discovery and Trend Analysis Across Text Corpora
摘要
Context: Discovering dominant topics across different text collections often requires extensive manual setup, especially when comparing how terminology evolves across communities. Traditional topic modeling pipelines are time-consuming to configure and are rarely designed for reuse across domains. Objective: This study presents a semi-automatic pipeline to streamline topic discovery and trend analysis in multiple corpora, while allowing researchers to review configurations. The contribution lies in its modular structure, which enables reproducible topic modeling across heterogeneous corpora with minimal manual effort. Method: The method integrates modular components for data scraping, cleaning, tokenization, and LDA-based topic modeling. An automated tuning step selects the number of topics using perplexity scores. Additionally, a lightweight module tracks the frequency of topic-related terms over time to identify shifts in focus. The pipeline is tested in the software architecture domain using both gray literature (StackOverflow, InfoQ, DZone) and academic sources (IEEE papers) to assess cross-domain applicability. Results:The pipeline produces coherent topics and reveals differences in terminology emphasis between practitioner and academic sources. The trend tracking component shows topic shifts that align with known changes in industry focus, such as increasing interest in observability and deployment practices. Conclusions: The proposed approach offers a replicable way to carry out topic discovery and trend analysis across domains. Its semi-automatic design reduces setup time, enhances consistency in topic modeling experiments, and supports exploratory analysis of discourse in large text collections. This contributes to better interpretation and tracking of emerging topics in dynamic, multi-source environments.