Large Scale Web Crawling and Distributed Search Engines: Techniques, Challenges, Current Trends, and Future Prospects

The heart of any substantial search engine is a crawler. A crawler is a program that collects web pages by following links from one web page to the next. Due to our complete dependence on search engines for finding information and insights into every aspect of human endeavors, from finding cat videos to the deep mysteries of the universe, we tend to overlook the enormous complexities of today’s search engines powered by the web crawlers to index and aggregate everything found on the internet. The sheer scale and technological innovation that enabled the vast body of knowledge on the internet to be indexed and easily accessible upon queries is constantly evolving. In this paper, we look at the current state of the massive apparatus of crawling the internet, specifically focusing on deep web crawling, given the explosion of information behind an interface that cannot be extracted from raw text. We also explore distributed search engines and the way forward for finding information in the age of large language models like ChatGPT or Bard. Our primary goal is to explore the junction of large-scale web crawling and search engines in an integrative approach to identify the emerging challenges and scopes in massive data where recent advancements in AI upend traditional means of information retrieval. Finally, we present the design of a new asynchronous crawler that can extract information from any domain into a structured format.