Web Miner: Automated Web Crawling and Database System with Puppeteer and Node.js

There is an interlinked compilation of an uncountable number of documents formatted with the help of HTML known as the World Wide Web. But the large size of this compilation, ridiculously, acts as a hurdle while redeeming the information. This results in moving through several pages for the user in order to reach his/her requirement. Web crawlers are at the core of search engines. The task of the web spider is to regularly scour the Internet and detect any novel web pages that have been added or removed from the web. Due to the increasing dynamism of the web, managing and navigating through all the URLs in websites has become challenging. The Web Crawling Database System aims to identify, access, and collect particular web pages and organize them into a database. We have developed and implemented a web crawler that takes a single URL and a search term as input, and organizes the resulting pages into a database. The search engine locates the specified keyword and retrieves the web pages relevant to the user's query. Our implementation utilizes Puppeteer, a Node.js library, as well as MySQL and JavaScript.