The Ford F-Series is the best selling thing with 4 wheels in North America and has been since the early 80s. The sheer number of F-150s sold each year is hard to comprehend.

  • About 6% (1 our of 16) of all vehicles on the road today are Ford F-150s.
  • If we assume dealerships are open 12 hours a day 365 days a year that would mean that every 15 seconds a new Ford F-Series rolls off the dealer lot in North America.
  • The F-150 alone generates more revenue than all major US sports leagues combined.

All this to say that…


Electric vehicles are only able to store a fraction of the energy of internal combustion engine (ICE) vehicles and getting that energy onboard by charging a battery (currently) takes significantly longer than filling up a gas tank. Despite that limitation, electric vehicles can now manage a competitive range with comparatively little energy and charging times continue to improve.

A Tesla Model 3 Standard Range Plus has the equivalent energy of 5.6 litres of gas (1.2 gallons) but it can travel 423 km with that relatively small amount of energy. …


Graphical exploration of concurrency in Python

A series exploring concurrency in Python to speed up data retrieval from remote APIs.

This article is part of a series: Graphical exploration of concurrency in Python. Part 1: Defining and timing an API function in Python.

In this series, we’ve used the example of retrieving 1334 Jira issues in batches of 50 from the Jira API to illustrate 4 different ways of coordinating the 27 separate network requests required.

The altair plotting library was used throughout to generate charts showing the relative start and end times of each individual request (a link to the full code is available at the bottom of this article). …


Graphical exploration of concurrency in Python

A series exploring concurrency in Python to speed up data retrieval from remote APIs.

This article is part of a series: Graphical exploration of concurrency in Python. Part 1: Defining and timing an API function in Python.

Recap: Sequential vs. Multiprocessing vs. Threading

In the last section, we moved from running requests on multiple processes, each with their own process id and memory allocation to multiple threads running under the pre-existing process that’s running our code. This showed an improvement in overall runtime by eliminating the startup time of all those processes.

In this section we’ll run all the requests on a single thread and, instead of Python’s threading library handling the coordination of concurrency, we’ll use a combination of…


Graphical exploration of concurrency in Python

A series exploring concurrency in Python to speed up data retrieval from remote APIs.

This article is part of a series: Graphical exploration of concurrency in Python. Part 1: Defining and timing an API function in Python

Recap: Sequential vs. Multiprocessing

Sequential

Each request waited for the previous one to finish. The total time to complete 27 requests was 31.4 seconds and the CPU was only working ~2% of the time.


Graphical exploration of concurrency in Python

A series exploring concurrency in Python to speed up data retrieval from remote APIs.

This article is part of a series: Graphical exploration of concurrency in Python. Part 1: Defining and timing an API function in Python.

Even though the problem we’re dealing with is a waiting problem, not a computation problem we will still see some benefit from multiprocessing because the gains from some concurrency outweigh the overhead of spinning up processes.

I’m running this on a PC with an Intel 8700k processor. This has 6 CPU cores plus Hyper-Threading so to the OS it looks like 12 cores. The key advantage of multiprocessing over threading is that:

each process can run interpreted…


Graphical exploration of concurrency in Python

A series exploring concurrency in Python to speed up data retrieval from remote APIs.

The Problem

When using APIs most of the time is spent waiting for a response. When collecting information in bulk from an external service this can result in a lot of wasted time. This article explores how to go from sequential requests to concurrent requests using Python’s Multiprocessing, Threading, and Async features.

Introduction

In a previous article, we wrote a simple function that worked around the Jira API limit for the number of issues returned by a single query. …


Introduction

APIs are great. They separate access and control of your data from the UI that’s built around it. It’s like a UI built for a computer. This article is going to cover the basics of interacting with the Jira API to retrieve data using Python and load it into a Pandas DataFrame. From there you can clean, filter, chart, or store it.

We will also cover some basic write operations through the API: creating issues, transitioning issues, linking issues, and adding comments. The Jira API is very rich but these basic building blocks open up a lot of opportunities for…


The Magic Keyboard is (finally) Apple’s answer for iPad users who have ever longing looked at the built-in kickstand that Microsoft built into its Surface line. It’s an interesting device because Apple is trying to solve the “how to attach a keyboard to a big piece of glass” problem in a different way than we’ve seen before.

After using it for a few days I’m surprised how different the floating design makes it feel from both a traditional laptop or a kickstand style solution. Three things stood out to me about this floating, double hinge design. It:

  1. brings the screen…


The Problem

APIs and document databases sometimes return nested JSON objects and you’re trying to promote some of those nested keys into column headers but loading the data into pandas gives you something like this:

df = pd.DataFrame.from_records(results["issues"], columns=["key", "fields"])

The problem is that the API returned a nested JSON structure and the keys that we care about are at different levels in the object.

Derek Mortensen

I write about programming, statistics and tech. Studied electrical engineering (B.Sc., M.Sc.) and I work in software.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store