Autonomous Voice Agent
Cutting voice-agent latency 54%, from 2.4s to 1.1s
- Timeline
- Oct 2025 - Mar 2026
- Role
- AI / Machine Learning Engineer @ Outlyst
- Status
- In Production
- Primary stack
- FastAPI · Retell AI · AsyncIO · PostgreSQL
Headline metrics
- What it tackles
- At 2.4s a turn, a voice agent sounds like a bad phone line and the prospect starts talking over it. Under 1.2s it feels human enough that they stay on the call.
- What it delivers
- 1.1s mean call latency, a 54% cut, with no horizontal scaling and no change to the model - the win came out of profiling, not architecture.
The problem in full
A voice agent's quality is dominated by latency. A 2.4-second response feels like a bad cell connection; under 1.2 seconds it feels human enough that the prospect stays on the call. The Outlyst voice agent was clearing 2.4s on warm calls, and response times spiked unpredictably under load.
The hard part is that Retell AI handles speech recognition and TTS - the backend just answers structured tool calls - but the round-trip from ASR through inference and back is dominated by what we do in those middle hundreds of milliseconds. Profiling, not architecture redesign, was the actual problem.
Goal: get average call latency under 1.2s, and do it without horizontal scaling that would have killed the unit economics.
System design
I instrumented the FastAPI inference backend with py-spy and asyncio task tracing. The traces showed two bottlenecks the metrics dashboards had missed: a synchronous ORM call on each tool invocation that blocked the event loop, and a connection pool sized for the wrong concurrency profile - pools sized for HTTP request bursts, not long-lived websocket sessions.
Replaced the synchronous ORM with asyncpg for direct PostgreSQL access on the hot path. Restructured the connection pool sizing around observed session lifetime rather than peak request rate. Parallelised independent tool calls with asyncio.gather() so a single user turn could query CRM, calendar, and contact-enrichment in parallel instead of in sequence.
Built a lightweight gatekeeper-detection classifier that runs before the main inference loop, so we don't burn LLM tokens on receptionists who'll just transfer the call. Detected gatekeepers route to a callback scheduler instead of a dead-end transfer.
Added structured CRM sync via automated extraction pipelines, removing the manual data entry step between a completed call and a usable contact record.
Key technical decisions
asyncpg over SQLAlchemy
SQLAlchemy's async support is real but layered with abstractions that show up in flame graphs. asyncpg is the actual driver, no ORM, and the inference backend doesn't need migrations or relationship modeling at request time - just fast reads and writes against a known schema.
py-spy over cProfile
py-spy samples without instrumentation, so we could profile the production process under real load without restarting it or distorting timing. cProfile would have changed the timing it was measuring.
Pool sizing for sessions, not requests
A websocket call holds a session for minutes, not milliseconds. Sizing the pool for HTTP request volume gave us tens of pooled connections trying to serve thousands of long-lived sessions. Sizing it for the observed in-flight session distribution fixed the contention without adding infrastructure.
Gatekeeper classifier before inference
Cheap-and-fast filter beats expensive-and-smart. Detecting 'this is a receptionist, not the prospect' with a small classifier saves ~3-5 minutes of call time per gated call. It also routes those calls to a callback scheduler instead of dead-ending.
The results in full
Mean call latency dropped from 2.4s to 1.1s - a 54% reduction - without horizontal scaling, across 2,100+ outbound calls over the contract.
What I'd do next
What I'd change: instrument event-loop lag from day one. The request-rate dashboards looked healthy right up until latency spiked, because they never measured the thing that was actually saturating - time-to-yield inside the event loop. py-spy found in an afternoon what the dashboards had been hiding for weeks.
The next 200ms of latency reduction is going to come from the LLM inference itself, not the surrounding plumbing - speculative decoding, smaller fine-tuned models for the specific tool-call patterns, or moving the gatekeeper classifier to a co-located CPU model. The plumbing is mostly drained.
Continue reading
VoiceFlow
Retell call exporter with local Whisper transcription
FinLaw-UK
Graph-augmented RAG for UK financial regulation
Jobzyl
One search across 20 job boards, with ATS resume matching