AsyncIO's blocking call problem

I frequently work with FastAPI[1]^[1]source: trust me, an asynchronous Python web framework built for performance[2]^[2]it’s pretty darn fast.

Every so often I’ll check in on a FastAPI server’s metrics and see a graph that looks something like this:

Mysterious drops in request throughput, and there’s no corresponding CPU spike. If you’ve managed a Python backend server[3]^[3]With more than 3 concurrent users, you might already know what’s going on.

This is the kind of graph that sends you digging: the service is supposed to be high-concurrency, but something feels suspiciously sequential.

The event loop

asyncio is built around an event loop, which lets concurrent functions co-operatively give up control when they need to wait for something else to happen. While one coroutine is waiting on a network request, another one can run.

A FastAPI endpoint might look like this:

@app.get("/users/{user_id}")
async def get_user(user_id: int):
    user = await fetch_user(user_id)
    return user

That await is the important part. While fetch_user() is waiting on I/O, the event loop can go work on some other request.

At a high level, it looks something like this:

In the synchronous version, request-1 blocks the worker until it finishes, then request-2 runs, then request-3. In the async version, request-1 can hit an await, yield control, and let the loop make progress on request-2 or request-3 while request-1 is still waiting on the database or network.

Only one coroutine is actively running Python code at a time. One event loop runs callbacks and tasks in a single thread. While a task is running no other task can run in the same thread. The others make progress when the current one hits an await and yields control back to the loop. If one of those coroutines calls a blocking function instead, everything behind it waits.

Used correctly, asyncio can handle a lot of requests without needing a pile of threads[4]^[4]Self-proclaimed ‘perfect fit for I/O-bound code’.

The catch is that real-world codebases are rarely fully asyncio-compatible[5]^[5]Django has a whole async_unsafe mechanism because this boundary is easy to get wrong in mixed codebases. There’s usually some mix of async and blocking functions, and it’s often hard to tell where the boundary is.

Blocking the loop

Calling a blocking function from an async context can cripple an asyncio app. One blocking call can make every other request wait[6]^[6]“just don’t do that”, I hear you say. I agree.
But I challenge you to present any significant codebase that doesn’t have a stray time.sleep(0.1) somewhere.

There are ways to safely run blocking code in asyncio – for example, offloading it to a thread with asyncio.to_thread()[7]^[7]My savior on many occasions (or to a separate process for CPU-bound work because of the GIL). But the lack of tooling makes this a very easy footgun.

Consider this snippet:

import asyncio
import time
 
async def foo():
    # An async function that calls a blocking function.
    # Many linters catch this
    time.sleep(1)
 
def bar():
    # A straight up blocking function
    time.sleep(1)
 
async def baz():
    # An async function that correctly wraps a blocking call in an executor
    loop = asyncio.get_running_loop()
    await loop.run_in_executor(None, time.sleep, 1)
 
async def main():
    await foo()
    bar()
    await baz()
 
if __name__ == "__main__":
    asyncio.run(main(), debug=True)

The calls to foo() and bar() both block the event loop for a second. In this toy example there isn’t much concurrency to disrupt, but in a real service those pauses mean every other request handled by the same loop stops making progress too.

This is also why APM tools like SigNoz or New Relic are helpful but incomplete here. They can tell you that something is wrong, but not always which call caused it.

Runtime debugging

asyncio’s debug mode[8]^[8]It really don’t do much… enables a couple extra runtime checks that can help surface performance issues:

• callbacks taking longer than 100 milliseconds are logged
• the execution time of the I/O selector is logged if it takes too long to perform an I/O operation

Running the above snippet with debug mode enabled yields this output:

Executing <Task pending name='Task-1' coro=<main() running at main.py:23>
wait_for=<Future pending cb=[_chain_future.<locals>._call_check_cancel() at python/3.11.4/lib/python3.11/asyncio/futures.py:387, Task.task_wakeup()]
created at python/3.11.4/lib/python3.11/asyncio/base_events.py:427>
cb=[_run_until_complete_cb() at python/3.11.4/lib/python3.11/asyncio/base_events.py:180]
created at python/3.11.4/lib/python3.11/asyncio/runners.py:100> took 2.012 seconds

Aside from not being terribly legible, the debug logs don’t actually point us to the root of the issue.

We know main() exceeded the blocking time threshold, but this doesn’t help us identify the actual blocking calls inside it.

Even if debug mode could identify the root cause, these checks are runtime-only. Unless your tests exercise exactly the bad path, it’s easy for blocking calls to slip through and only get noticed later under real traffic.

Toward static analysis

mypy changed my relationship with Python. The instant in-IDE feedback gave me confidence my code wouldn’t crash in production. I run mypy locally as a pre-commit check, and it catches broken call signatures in seconds – before I waste time pushing and waiting hours for CI to tell me what I already broke.

I want the same experience for asyncio codebases.

There appears to be some energy being put towards identifying blocking calls[9]^[9]Sad StackOverflow thread from 2020. Existing tools like flake8-async[^15] and ruff’s ASYNC rules[10]^[10]Rust btw catch direct blocking calls, but they miss the nested case of async -> sync -> blocking. The ones that do catch nested calls only do so at runtime.

I explore one possible solution in Static analysis of blocking calls in async Python. I’m trying to build a static analyzer that can identify calls to blocking functions inside async contexts before they hit production.