Crawling 101: Crawling a static web page

We will begin with building our first crawler. The goal is that the crawler takes a static web page as …

Quick tips: How to avoid getting blocked

Getting blocked from crawling can happen very fast. Maybe we exceed the maximum number of requests, maybe we send too …

Crawling 104: Retrieving data from APIs

Next up, we will be dealing with probably the most convenient way of retrieving data for our research projects, that …

Pro tip: Get notified by email when the crawler has done its job

Crawling can take hours or days to complete. Sometimes crawling is done regularly, say, once a week or even daily. …

Quick tips: Friendly crawling

Data should be crawled responsibly so that it does not have a detrimental effect on the web site being scraped. …

Crawling 102: Collecting web data with Selenium and Python

Selenium is a powerful tool for collecting web-data. With the help of Selenium, we can collect data from pages that …