End-to-End System for Data Crawling, Monitoring, and Analyzation of E-Commerce Websites
摘要
In an era of rapidly developing internet technology and increasing social requirements of individuals, creating and exploiting a large amount of data has become necessary for businesses to improve their competitiveness. Crawling data quickly and accurately to satisfy user needs has become crucial for data mining Many web crawling data methods have been utilized by businesses to acquire the desired information but crawling systems for e-commerce have not yet. In this paper, we proposed an end-to-end crawling system for e-commerce websites. We investigate the Lomart and Shopeefood websites and build a data collection system to analyze the retrieved data and then represent them in chart form to facilitate the most intuitive monitoring of the data crawling process. The source code and data are available at https://github.com/DoManhQuang/data_crawling .