In simple terms, a big part of what makes the scraping/crawling of modern websites difficult is the need to implement the crawling/scraping procedures in such a way that the scraping instance(s) appears and acts as if they were human. Without addressing this issue so directly, the previous Chaps. 7 to 8 provided insights into several techniques that work toward “human-like” appearance and actions (such as using an actual browser to interact with the website for scraping). However, there are still many websites that can detect and block automated traffic, even when using a browser. These websites may use techniques such as CAPTCHAs or blocking certain user agents or IP addresses. While these measures protect websites from too extensive automated traffic, they can also hinder the more polite web mining efforts in the context of legitimate data collection for academic research purposes. In addition, and rather related to human-like actions than human-like appearance, some websites require users to log in to access certain content, which can be a challenge for web scrapers. Of course, the idea here is that you are in a situation where you have legitimate access to the website and thus have the right credentials to log in. However, if you want to automate such a process, you need to be able to handle the login process in your web scraping code.