Uncovering Insights in Agroforestry: A Text Mining Analysis Using Web Scraping and Topic Modeling

Big data is so prevalent, text mining has become an essential tool to provide an overview and easy understanding of vast amounts of data. In this article, we present a study on agroforestry that collects information from Google Scholar using web scraping techniques. We describe the evolution of a web scraper using Python modules and packages to extract the obligatory data from their source HTML page. The collected data is organized into a CSV file and subjected to various topic modeling strategies, such as Latent Dirichlet Allocation (LDA), Bidirectional Encoder Representations from Transformers (BERT), and Latent Semantic Analysis (LSA) to analyze and present the data as topics. The article highlights the significance of web scraping in retrieving data and offers insights into the domain of agroforestry facilitating further analysis. This research demonstrates the power of text mining and topic modeling in exploring large datasets and deriving valuable insights.