How AI Voice Agents Actually Work

The Four-Stage Voice AI Pipeline

Modern AI voice agents chain ASR, NLU, an LLM, and TTS in a streaming pipeline that processes and responds to human speech in under 500 milliseconds. When you talk to a modern AI voice agent, four distinct systems work in a tight loop to create the illusion of a natural conversation. Automatic Speech Recognition (ASR) converts your voice into text. A Natural Language Understanding (NLU) layer extracts intent and entities. A Large Language Model (LLM) generates a contextual response, often calling tools or retrieving knowledge. Finally Text-to-Speech (TTS) renders that response as audio. The challenge isn't building any one of these components — off-the-shelf APIs exist for every layer. The challenge is running all four in parallel with streaming I/O so the caller hears a response start while they're still finishing their own sentence. This guide walks through the full architecture, the latency budget, and the engineering tradeoffs that separate sub-500ms systems from the 2-3 second latency of earlier voice AI.

Stage 1 — Automatic Speech Recognition (ASR)

ASR converts raw audio into streaming text using transformer-based models like Whisper or Deepgram Nova, emitting partial transcripts every 100-200ms so downstream systems can start processing immediately. The caller's microphone produces a continuous audio stream, typically 16kHz 16-bit PCM. ASR models process this in rolling windows, producing interim hypotheses and committing to final transcripts at sentence boundaries. Modern ASR uses encoder-decoder transformers trained on tens of thousands of hours of multilingual speech. Streaming vs. batch ASR Production voice agents stream audio chunks to ASR, receiving partial transcripts every 100-200ms — not a single transcript at the end. Voice Activity Detection (VAD) A small model decides when the caller starts and stops speaking, triggering endpointing logic and barge-in detection. Multilingual code-switching Best ASR models handle mid-sentence language switches (English to Spanish to Hindi) without reloading. Acoustic robustness Trained on noisy call audio, telephone bandwidth (8kHz), and varied accents for real-world reliability.

Stage 2 — NLU & LLM Reasoning

The LLM receives the transcript plus conversation history, system prompts, and tool definitions, then decides whether to answer, call a function, or retrieve knowledge from a vector database. Once ASR produces a committed transcript, the LLM takes over. Modern voice agents don't treat NLU and response generation as separate steps — a single LLM call handles intent classification, entity extraction, reasoning, and response generation simultaneously, guided by a structured system prompt. LLM inputs for each turn: Compare the details in our intent classification & entity extraction guide and the voice AI vs IVR comparison . Add voice AI to your website in 10 minutes

• • System prompt defining agent persona, capabilities, and constraints
• • Conversation history (trimmed to fit context window, typically 8-32K tokens)
• • Tool/function definitions for booking, CRM lookup, payment, etc.
• • Retrieved context from RAG over the business's knowledge base
• • Latest user transcript from ASR
• Try AnveVoice Free
• • 50K tokens free, no card
• • No credit card required
• • One-line embed

Stage 3 — Text-to-Speech (TTS)

TTS models like ElevenLabs, OpenAI TTS, and Azure Neural Voices convert LLM output into audio using neural vocoders, often starting playback before the full response is generated. The TTS layer is what the caller actually hears. Modern neural TTS uses a two-stage architecture: an acoustic model predicts mel-spectrograms from text, and a vocoder (WaveNet, HiFi-GAN) converts spectrograms to waveform. Critically, TTS receives streaming tokens from the LLM and begins synthesizing audio chunks while the LLM is still generating the rest of the sentence. Streaming synthesis TTS begins producing audio after the first ~8 tokens arrive from the LLM, slashing perceived latency. Voice cloning A 30-second voice sample creates a custom brand voice, trained via few-shot fine-tuning on speaker embeddings. Prosody control SSML tags and emotion conditioning let the agent sound empathetic, enthusiastic, or neutral as context requires. Cost vs. quality High-quality neural TTS runs $0.15-$0.30 per 1K chars; faster low-cost models hit $0.015 but sound more robotic.

Turn-Taking, Interruptions & Back-Channel Cues

Turn-taking logic uses VAD plus a turn-detection model to distinguish genuine interruptions ('actually, wait—') from back-channel cues ('uh-huh') the agent should ignore. Human conversation isn't strictly sequential — we overlap, interject, and acknowledge. Naive voice agents get this wrong: they either stop on every 'uh-huh' or steamroll over real interruptions. Modern systems ship a dedicated turn-detection model that classifies the caller's vocalization in real time. Three types of user vocalization:

• • Genuine interruption ("Actually, hold on...") — agent stops TTS, processes new input
• • Back-channel acknowledgment ("uh-huh", "right") — agent continues speaking
• • End-of-turn signal — caller finished, agent should respond

The Sub-500ms Latency Budget

Best-in-class voice AI achieves sub-500ms end-to-end latency by overlapping ASR, LLM, and TTS streams and using geographic co-location with telephony providers. Conversational feel requires responses to arrive within one human turn — roughly 500-900ms. Hitting that target under real-world network conditions is non-trivial. Every component must stream, and the pipeline must run in parallel rather than sequentially. Typical latency breakdown: See our voice AI comparison guide for latency benchmarks across platforms. Need help choosing? Book a free 15-minute consultation

• • ASR endpointing: 150-300ms (silence detection)
• • LLM time-to-first-token: 200-500ms
• • TTS first-audio-byte: 100-200ms
• • Network + jitter buffer: 50-150ms
• • Total perceived latency: 500-1,150ms depending on optimization

Tool Calls & Real-World Actions

Voice agents take real actions by emitting structured tool calls (JSON function invocations) that trigger API requests to CRMs, calendars, payment processors, and website DOM automation. A voice agent that can only talk is a chatbot in disguise. The real value comes from taking actions — booking an appointment, updating a CRM, processing a payment, filling a form. Modern LLMs support function calling : the model returns a structured JSON payload specifying which tool to invoke and with what arguments. AnveVoice adds an extra layer: DOM actions that let the agent navigate pages, click buttons, and fill forms on the live website the visitor is using. Learn more in our AI that takes actions on websites guide and agentic AI examples for business .

Frequently Asked Questions

Common questions about voice AI architecture cover the four-stage pipeline, LLM choices, interruption handling, typical latency, and the differences between voice AI and IVR. Want to see a sub-500ms voice agent in action? Try AnveVoice free — 50K tokens, no credit card required.