The Problem: Post-Response Code Freezes on Cloud Run

If you use FastAPI's BackgroundTasks (or asyncio.create_task) on Google Cloud Run with default CPU allocation, your background work might never run—or run minutes later when the next request arrives. The platform throttles CPU to near-zero after the HTTP response is sent, freezing any coroutine mid-await.

The Setup

A webhook receiver acknowledged events in under a second. The handler validated, deduplicated, and handed off the actual job via BackgroundTasks.add_task(dispatch_job, payload) before returning a 202. Textbook fire-and-forget.

@app.post("/webhook")
async def webhook(req):
    ...validate...
    background_tasks.add_task(dispatch_job, payload)
    return JSONResponse(202, ...)

On a normal server, this works fine. On Cloud Run with request-based CPU, the instance gets CPU only while a request is in flight. The moment the response goes out, CPU is throttled to ~0. The background task freezes at the first await—usually mid-network call—until the next request arrives and unfreezes it.

The Symptom That Fooled Me

The background task doesn't fail. No error, no crash. The log line before the network call prints fine (runs during the request's CPU window). The next line, after the await, doesn't print for minutes—or never, if no traffic arrives and the instance scales to zero.

request in ──► handler runs (full CPU)
└─ schedules background task
response out ◄── 202 sent
└─ CPU throttled to ~0        ← background task freezes HERE,
usually mid-network-call
...minutes pass...
next request in ──► instance gets CPU again ──► frozen task resumes, completes

This pattern is hard to catch because:

  • It's fast on busy days, slow or lost on quiet ones—opposite of an overload problem.
  • Retry-happy callers (chat platforms, webhook senders that retry on timeout) generate the very traffic that unfreezes the task, turning "broken" into "occasionally a few minutes slow."
  • It works perfectly locally and under load testing, where the instance stays saturated with requests.

Confirming the Hypothesis

The cheapest diagnostic: inject traffic. If a "stuck" background task completes the instant you curl the service's health endpoint a couple of times, you've proven CPU starvation. No debugger needed.

The Fix: Move Work Inline

The real fix is to not schedule the latency-critical part as a background task. In this case, the RPC that launches the job costs about one second—well within most webhook ack budgets (5 seconds or more). So just await it before responding:

@app.post("/webhook")
async def webhook(req):
    ...validate, dedupe...
    recent[key] = now()          # dedup marker
    try:
        await launch_job(payload)   # ~1s: starts the job, doesn't wait for it
    except Exception:
        recent.pop(key, None)       # let the sender's retry re-dispatch
        raise                       # → 5xx → sender retries
    return JSONResponse(202, ...)

Two critical details:

  • On failure, roll back the dedup marker before re-raising. Otherwise your dedup swallows the sender's retry, and the job never gets a second chance.
  • Let the failure surface as a 5xx. A 202 that lies about success is worse than an honest error.

When Inline Isn't an Option

If the caller enforces a hard ack deadline (e.g., 3 seconds) and pre-dispatch work is inherently slow (an LLM classification step), switch the platform mode, not the code:

gcloud run services update  --no-cpu-throttling

Persist this in your IaC/deploy config—not as a one-off CLI command. The next deploy from your pipeline can silently revert to request-based billing if the flag isn't in the deploy definition. Also reconsider minimum instance count, since instance-based billing changes the cost model from per-request to per-instance-time.

Testing the Difference

Most test-client frameworks execute background tasks before handing the response back. So a happy-path test ("was dispatch_job called?") passes whether the call is inline or backgrounded. The assertion that distinguishes them is the failure path: trigger a dispatch failure and check that you get a 5xx and the dedup marker is cleared. Only inline dispatch can produce that combination.

Lessons Learned

  1. Post-response code has no CPU guarantee on request-billed serverless. Treat "after the response" as "whenever the next request arrives, maybe never."
  2. Latency-critical side effects belong on the request path. A job-launch RPC that only starts work is often cheap—about a second—well within most webhook timeout budgets. Await it, then respond.
  3. Make inline failure retryable end-to-end. Roll back dedup markers and return 5xx so the sender's retry actually re-dispatches.
  4. When inline is impossible, change the platform mode, not the code. Switch to instance-based CPU allocation and persist the flag in deploy config.
  5. Confirm the hypothesis by injecting traffic. Cheap, decisive, non-invasive.
  6. Audit the pattern fleet-wide. The same convenience API tends to get copy-pasted across every receiver in a codebase.

Pitfalls and False Signals

  • The first log line printing does not mean the task ran. It ran for the last few milliseconds of leftover request CPU, then froze at the first await.
  • "It works in staging" proves nothing. Health checks and teammates clicking around provide exactly the wake-up traffic that masks this.
  • Don't blame the downstream API. The frozen call is usually a healthy RPC; its client-side coroutine simply isn't being scheduled.
  • Retries hide the bug. Senders that retry unacked deliveries generate the traffic that unfreezes the task.

Quick Checklist

  • List every place a handler schedules work after the response.
  • For each: is the platform CPU-throttled outside of requests? (Cloud Run default: yes.)
  • Can the work move inline within the caller's timeout budget? Move it; on failure, roll back dedup markers and return 5xx.
  • If it can't move: flip to instance-based CPU allocation in deploy config, not a live-only update.
  • Add a failure-path test: dispatch error → 5xx and dedup marker cleared.
  • To confirm a suspected stall in prod: watch logs while sending synthetic requests. Instant completion means starvation confirmed.

One-line summary: On request-billed serverless, "after the response" means "whenever the next request happens to arrive"—so launch anything that matters before you respond, or pay for always-on CPU.