Graphical exploration of concurrency in Python

Part 2: Multiprocessing API requests with Python

A series exploring concurrency in Python to speed up data retrieval from remote APIs.

This article is part of a series: Graphical exploration of concurrency in Python. Part 1: Defining and timing an API function in Python.

Even though the problem we’re dealing with is a waiting problem, not a computation problem we will still see some benefit from multiprocessing because the gains from some concurrency outweigh the overhead of spinning up processes.

I’m running this on a PC with an Intel 8700k processor. This has 6 CPU cores plus Hyper-Threading so to the OS it looks like 12 cores. The key advantage of multiprocessing over threading is that:

each process can run interpreted Python code independently.

So, we can split up a bunch of work, pass it off to 12 processes, each running on their own CPU “core” and let the kernel handle the scheduling. The downside is that processes are not free. There’s some spin-up time involved.

Here’s the code that uses jira_query() to allocate a series of API requests to multiple processes (one per CPU core).