Social media and websites are major data sources for social scientists, businesses, and policymakers, providing insights into human behavior, market trends, and public opinion. This chapter introduces major approaches for collecting web data. The first part of the chapter provides instructions on Internet data collection by using the Application Programming Interface (API), covering topics such as authentication and authorization, GET and POST HTTP data requests from specific resources, and the JSON format frequently used in returned data. The second part reviews Web content mining for cases when API service is unavailable with tools like BeautifulSoup and handling dynamic content using Selenium. Ethical and legal considerations are highlighted, such as respecting website terms of service and avoiding actions that disrupt web services. Finally, the chapter concludes with a Python lab teaching how to develop programs for API data acquisition from the Amadeus global airline and hotel reservation service and for scraping a demo online bookstore.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Web Data Scraping

  • Andrei P. Kirilenko

摘要

Social media and websites are major data sources for social scientists, businesses, and policymakers, providing insights into human behavior, market trends, and public opinion. This chapter introduces major approaches for collecting web data. The first part of the chapter provides instructions on Internet data collection by using the Application Programming Interface (API), covering topics such as authentication and authorization, GET and POST HTTP data requests from specific resources, and the JSON format frequently used in returned data. The second part reviews Web content mining for cases when API service is unavailable with tools like BeautifulSoup and handling dynamic content using Selenium. Ethical and legal considerations are highlighted, such as respecting website terms of service and avoiding actions that disrupt web services. Finally, the chapter concludes with a Python lab teaching how to develop programs for API data acquisition from the Amadeus global airline and hotel reservation service and for scraping a demo online bookstore.