Concurrency and performance

"Python is slow" is only half true, and the other half is usually the algorithm. We measure before optimising, fix the cheap things first, then choose between threads, processes and asyncio knowing why. Native code comes last, when numbers say it's worth it.

Duration: 2 to 3 days · Level: advanced

For: Python developers whose programs are too slow, or who wait on the network or the disk.

Prerequisites: a good command of Python; generators and context managers help.

Concurrency and performance

Day 1 — Measure first

  • Where does the time go? timeit and cProfile
  • Watching a live program with py-spy
  • Memory: tracemalloc and friends
  • Algorithmic complexity, without the maths
  • The lesser-known structures: deque, Counter, bisect, heapq
  • Doing less: generators and lazy evaluation
  • Caching with functools.lru_cache
  • Caching big data sets on disk with diskcache
  • The numpy exception: vectorise instead of looping

Day 2 — Doing several things at once

  • Concurrency vs parallelism
  • The GIL, and free-threaded Python 3.13+
  • Threads, locks and queues
  • Using every CPU with multiprocessing
  • concurrent.futures: one API for both
  • Non-blocking I/O with asyncio
  • Tasks, TaskGroup and timeouts
  • Async HTTP requests with httpx
  • Mixing sync and async code

Day 3 — Optional: out of the process

  • Long jobs in a task queue with huey
  • Retries, scheduling and results
  • When to reach for native code
  • A taste of Cython
  • Compiling your program with Nuitka
  • Rust extensions with PyO3 and maturin
  • Choosing, case by case

Elevate Your Python Proficiency!