Web crawling is an essential technique for collecting data in various domains, but traditional crawlers often struggle with inefficiencies such as irrelevant data collection, redundant crawling, and resource overuse. This paper explores the enhancement of web crawling efficiency through the integration of adaptive scheduling algorithms and ChatGPT. Adaptive scheduling dynamically prioritizes URLs based on real-time parameters such as relevance, server load, and crawling depth, enabling a more targeted and resource-efficient approach. Meanwhile, ChatGPT is an intelligent content analysis tool, offering functionalities such as relevance filtering, keyword expansion, and summarization. The proposed system combines these methodologies, presenting a robust architecture in which ChatGPT supports decision-making within an adaptive crawling framework. Performance metrics, such as crawl speed, relevance accuracy, and resource usage, are used to evaluate system efficiency. A case study on academic data collection shows significant improvements in speed and relevance over traditional crawlers. This approach paves the way for intelligent next-generation web crawlers capable of high-precision, scalable data collection by addressing challenges such as dynamic content and API rate limits.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Enhancing Web Crawling Efficiency with Adaptive Scheduling Algorithms and ChatGPT Integration

  • Nguyen Minh Tuan,
  • Phayung Meesad

摘要

Web crawling is an essential technique for collecting data in various domains, but traditional crawlers often struggle with inefficiencies such as irrelevant data collection, redundant crawling, and resource overuse. This paper explores the enhancement of web crawling efficiency through the integration of adaptive scheduling algorithms and ChatGPT. Adaptive scheduling dynamically prioritizes URLs based on real-time parameters such as relevance, server load, and crawling depth, enabling a more targeted and resource-efficient approach. Meanwhile, ChatGPT is an intelligent content analysis tool, offering functionalities such as relevance filtering, keyword expansion, and summarization. The proposed system combines these methodologies, presenting a robust architecture in which ChatGPT supports decision-making within an adaptive crawling framework. Performance metrics, such as crawl speed, relevance accuracy, and resource usage, are used to evaluate system efficiency. A case study on academic data collection shows significant improvements in speed and relevance over traditional crawlers. This approach paves the way for intelligent next-generation web crawlers capable of high-precision, scalable data collection by addressing challenges such as dynamic content and API rate limits.