A voice agent lives or dies on latency. At 2.4 seconds a turn it feels like a bad cell connection and the person on the other end starts talking over it; under 1.2 seconds it feels human enough that they stay on the call. Our outbound agent at Outlyst was clearing 2.4s on warm calls and getting worse under load - past roughly 200 concurrent sessions, response times spiked unpredictably and a fraction of calls dropped outright.
The instinct in that situation is to reach for a bigger architecture. It was the wrong instinct.
The latency is in the middle, not the edges
Retell AI handles the speech side - recognition and text-to-speech. Our backend only answers structured tool calls in between. So the round-trip a caller actually feels is dominated by what we do in those middle few hundred milliseconds, not by the model or the audio. The problem was not the design; it was measurement - find where the milliseconds go and stop losing them. The target was an average under 1.2s, held stable past 2,000 concurrent sessions, and no horizontal scaling, because throwing machines at it would have quietly killed the unit economics.
Profile the process that is actually under load
I reached for py-spy rather than cProfile, and the reason matters. py-spy samples the running process from the outside, so I could attach it to the production backend under real traffic without restarting anything or distorting the numbers. cProfile's instrumentation would have moved exactly the timings I was trying to read.
A profiler that changes the timing it measures is measuring the wrong program.
# Illustrative: sample the live backend and render a flame graph
py-spy record --pid $(pgrep -f 'uvicorn') --duration 60 --output profile.svgThe flame graph and the asyncio task traces surfaced two bottlenecks the request-rate dashboards had never shown:
- A synchronous ORM call on every tool invocation, blocking the event loop.
- A connection pool sized for the wrong shape of load - tuned for short HTTP request bursts, not long-lived websocket sessions.
Get the blocking call off the event loop
One synchronous database call on the hot path does not just slow that one request - it stalls every coroutine sharing the loop. So I moved the hot path off SQLAlchemy and onto asyncpg.
That was a deliberate trade. SQLAlchemy's async support is real, but its abstraction layers show up in a flame graph as frames you cannot cut. asyncpg is the actual driver: no ORM, no relationship modelling, no migrations at request time - none of which the inference backend needs. What it needs is fast reads and writes against a schema it already knows.
Size the pool for sessions, not requests
The pool problem was subtler. A websocket call holds its session open for minutes, not milliseconds. A pool sized for HTTP request volume gave us a few dozen connections trying to serve thousands of concurrent, long-lived sessions, so callers queued on connection acquisition and the latency surfaced everywhere at once. Sizing the pool against the observed in-flight session distribution, rather than peak request rate, removed the contention without adding a single machine.
Stop doing independent work in sequence
A single user turn often needed data from several places at once - CRM, calendar, contact enrichment. Those calls were independent but running one after another, so the turn paid for the sum of them. asyncio.gather() collapses them into one await:
# Illustrative: independent tool calls in one turn, run concurrently
crm, calendar, contact = await asyncio.gather(
fetch_crm(caller_id),
fetch_calendar(caller_id),
enrich_contact(caller_id),
)Now the turn is as slow as its slowest call, not the sum of all of them.
There was one more change that was less about speed than waste: a lightweight gatekeeper-detection classifier that runs before the main inference loop. Cheap-and-fast beats expensive-and-smart - there is no reason to spend LLM tokens on a receptionist who is only going to transfer the call. Detected gatekeepers route to a callback scheduler instead of dead-ending, and each gated call saves roughly three to five minutes of GPU time.
What it added up to
Mean call latency dropped from 2.4s to 1.1s - a 54% reduction - with no horizontal scaling. The system now holds 2,100+ concurrent stateful websocket sessions without dropping any, where it used to degrade past 200. The downstream effects tracked the latency: a 25% lift in lead conversions, 27 qualified leads out of the gatekeeper-aware routing, and 100+ staff hours a week reclaimed once contact sync ran off automated extraction instead of manual entry.
Where the next milliseconds live
The plumbing is mostly drained now. The next 200ms will not come from the surrounding I/O; it will have to come from the inference itself - speculative decoding, smaller models fine-tuned to the specific tool-call patterns we see, or moving the gatekeeper classifier onto a co-located CPU model so it costs almost nothing to run. That is a different kind of work, and it is where I would point next.