TopicMapper: Geotagged URL Topic Extraction and Categorization
摘要
The paper presents TopicMapper, a tool designed to extract and categorize topics from geotagged URLs, focusing on web pages within specific geographical regions. The tool utilizes Apache Spark for large-scale data processing and employs the Latent Dirichlet Allocation (LDA) algorithm to identify topics within web pages. The system architecture involves parallelization and a multi-step pipeline, enabling efficient data handling and workload optimization. The study demonstrates the scalability of the tool through experiments on datasets of varying sizes. The results reveal that TopicMapper exhibits promising scalability, maintaining quasilinear performance as dataset sizes increase. The paper also introduces a method to avoid unnecessary recomputation by considering common areas between previously computed and new regions. Future work may involve GUI development for user-friendly interaction and refinement of the recomputation avoidance technique.