Web scraping and data collection

Half the data you need is sitting on a web page with no export button. We go from a single request to a crawler you can schedule, learning to find the hidden API before parsing HTML, and to be a guest the site doesn't want to ban.

Duration: 2 days · Level: intermediate

For: developers, analysts and anyone who needs to collect data from the web.

Prerequisites: the basics of Python.

Pricing Contact
Web scraping and data collection

Day 1 — Fetching and parsing

  • A refresher on HTTP: methods, headers, status codes
  • Requests and httpx
  • Encodings, or why your accents turned into gibberish
  • Parsing HTML with BeautifulSoup, lxml and selectolax
  • Finding your way with CSS selectors and XPath
  • Forms, sessions and cookies
  • The API hidden behind the page: reading the network tab
  • Sites full of JavaScript: driving a browser with Playwright
  • Storing the results in CSV and SQLite

Day 2 — Scaling up, politely

  • Being polite: robots.txt, rate limiting, user agents
  • What the law and terms of use allow
  • Retries, timeouts and resilience
  • Not downloading twice: caching with diskcache
  • Crawling many pages at once with asyncio and httpx
  • Cleaning and validating the data with Pydantic
  • Scheduling your scraper
  • When a framework is worth it: an overview of Scrapy

Elevate Your Python Proficiency!

Contact Examples of programs