CONTEXT FOR THIS WRITE-UP

Preserved from the existing engineering journal. Code, records, and diagrams are illustrative. Historical performance figures in these notes are not independently reproduced benchmarks or current project claims.

01 — The Problem

How CPU-bound operations silently choke FastAPI async event loops

FastAPI is celebrated for high-concurrency asynchronous I/O. However, many developers mistakenly write CPU-bound operations (like OpenCV image decoding or NumPy array operations) directly inside `async def` route handlers.

Because Python's `asyncio` event loop runs on a single thread, any synchronous CPU-bound task blocks the entire loop, freezing all concurrent incoming web requests until the computation finishes.

02 — The Solution

Offloading CPU tasks to thread pools without sacrificing async ergonomics

By delegating heavy image processing and model inference to a bounded `ThreadPoolExecutor` or `run_in_threadpool`, the main event loop remains unblocked, serving hundreds of simultaneous health checks and WebSocket heartbeats.

Non-Blocking CPU Delegation in FastAPI
from fastapi import FastAPI, UploadFile, File
from starlette.concurrency import run_in_threadpool
import cv2
import numpy as np

app = FastAPI()

def cpu_bound_image_preprocess(raw_bytes: bytes) -> dict:
    # Heavy CPU synchronous decoding and color transform
    np_arr = np.frombuffer(raw_bytes, np.uint8)
    image = cv2.imdecode(np_arr, cv2.IMREAD_COLOR)
    processed = cv2.resize(image, (640, 640))
    return {"shape": processed.shape, "mean_intensity": float(processed.mean())}

@app.post("/api/v1/preprocess")
async def preprocess_image(file: UploadFile = File(...)):
    raw_bytes = await file.read()
    
    # Offload CPU-bound task to threadpool; keeps asyncio event loop free
    result = await run_in_threadpool(cpu_bound_image_preprocess, raw_bytes)
    
    return {"status": "success", "data": result}

Why this mattersUsing run_in_threadpool prevents a single 80ms image decoding job from blocking 50 other concurrent requests from being received or acknowledged.

LOOKING BACK

What I learned.

01

async def is for I/O; def is for CPU

FastAPI automatically runs standard `def` routes in an external threadpool, while `async def` executes directly on the main event loop.

02

Profile your event loop latency in production

Monitoring event loop lag with Prometheus metrics immediately surfaces hidden synchronous bottlenecks before they impact end users.

THANKS FOR READING.

The useful part of a technical problem is often the question it leaves you with.

Continue the conversation