An Intelligent and Automated Web Data Extraction System for E-commerce

In the contemporary digital landscape, web scraping has emerged as an indispensable technique for extracting valuable data from the expansive World Wide Web. Literature reports various encouraging approaches for web scraping that are dependent on names of HTML elements and structure of webpages. However, such dependency makes the system rigid, inefficient and imposes a serious challenge on the accuracy particularly when the webpage changes, emphasizing the critical need for automation in the realm of E-commerce. This paper addresses the limitation of the rigid conventional methods by proposing an efficient approach to outperform even in the real-world scenario of dynamically and rapidly changing contents of the web page. The intelligent and versatile data extraction approach devised in this research employs use of a set of keywords which consistently endure on the webpage, ensuring independence from variable names or xpath. The approach proposed in this research is rigorously tested in a real-time industrial environment to extract the data from more than 20,000 orders and has given 100% accurate results despite the synthetically incorporated changes in the structure and the HTML elements. The proposed method is confirmed to be superior to the one developed using generative AI in terms of its dependency on the structure of the webpages and allows seamless and accurate data extraction, regardless of the changes in the webpages. This research study thus not only presents an innovative methodology for automated web scraping but also paves the way for future advancements ensuring businesses stay ahead in the rapidly evolving digital landscape.