How We Build a Voice AI Agent: Our 4-Week Process
Voice is the least forgiving agent surface: a slow or clumsy reply and the caller hangs up. This is the process we use to ship voice agents that people actually stay on the line with, with the latency numbers and design rules we hold ourselves to.
- Published
Week 1 — call reasons and the latency budget
- Listen to 100+ real calls or read their transcripts. Tag each by reason. Like WhatsApp, it collapses to 6–10 types — and the top three are usually 70% of volume.
- Write the call flows for the top three only. Each flow is a short script of what the agent must collect and what it must never say. Everything else hands off.
- Set the latency budget and design to it: under 800 ms from the caller finishing to the agent starting to speak. That budget is split across speech-to-text, model, and text-to-speech, and it rules out slow retrieval mid-turn.
- Confirm integrations: the calendar or CRM the agent books into, and how transfers reach staff (SIP, forwarding, a queue).
| Stage | Target | How we hit it |
|---|---|---|
| End-of-speech detection | < 150 ms | Tuned VAD, no fixed silence timers |
| Speech-to-text | < 200 ms | Streaming STT, partials used early |
| Model first token | < 300 ms | Short context, cached system prompt, right model per step |
| Text-to-speech first audio | < 150 ms | Streaming TTS, sentence-level chunking |
| Total | < 800 ms | Measured on every call, alerted above 1 s |
Week 2 — the streaming stack
- Wire telephony to a streaming pipeline: audio in → streaming STT → model with tools → streaming TTS → audio out. Everything streams; nothing waits for a full sentence.
- Implement barge-in properly: when the caller speaks, cut TTS within 100 ms, keep what was said, and let the model recover the thread. Callers interrupt constantly.
- Build the tools: check_availability, book, reschedule, lookup, transfer_with_summary, take_message. Same rules as any agent — strict schemas, model-readable descriptions.
- Write the escalation rules in code: any clinical, legal, financial or emotional content transfers. The agent says it is transferring, and the human receives a spoken and written summary.
- Add the disclosure line at call start and make sure the agent never pretends to be human if asked.
Weeks 3–4 — evals, shadow mode, launch
- Build an eval set of 150+ scripted calls covering each flow, accents, background noise, interruptions and the emotional edge cases. Grade on the action taken (booked, transferred, message taken), not on wording.
- Shadow mode: the agent listens to live calls and proposes what it would do, staff see it, nobody hears it. Compare against what staff actually did for a week.
- Launch on after-hours and overflow first. Every call is recorded and transcribed; the first-week review is where the last 10% of quality comes from.
- Instrument the three numbers that matter: answer rate, containment (calls completed without a human), and bookings made. Report them weekly to the owner.
Four weeks, and a phone line that is answered every time. We build voice agents for $8k–$25k, integrated with your systems, and we teach the streaming stack and latency engineering in a two-day workshop.
Frequently asked questions
Which speech and language models do you use?
We pick per deployment for latency and language coverage, and we keep the stack swappable. The eval set is what lets us change a component without regressions.
Can the agent make outbound calls?
Yes — reminders, confirmations, follow-ups. Outbound needs consent handling and calling-hours rules, which we build in from the start.
How do you handle poor audio or strong accents?
They are in the eval set from day one, and the agent is designed to confirm critical details — names, dates, numbers — by reading them back.
