Challenges in Web Bot Detection and Detection Evasion Technologies

Web bots are automated programs designed to execute tasks on the Internet. Typically, Web bots are utilized to crawl search engines, increasing hit rates during user searches and ensuring that websites function correctly. However, some Web bots have been misused for malicious activities, such as hoarding goods and launching brute force attacks. Various methods for detecting Web bots have been developed and studied to combat such misuse. Conversely, techniques to circumvent these detection methods have also been explored. In this paper, we examine Web bot detection methods and the corresponding evasion techniques, drawing on information from contemporary Web bot-related research papers and technical blogs authored by experts in the field. Specifically, we assess the effectiveness of browser fingerprinting techniques within the context of Web bot detection and discuss the inherent challenges associated with this detection.