VoiceAI
← Back to blog

The Voice Stack Explained: STT, LLM, and TTS

Admin User · 9 May 2026 · 7 reads

Breaking Down the Voice Pipeline

Behind every voice AI agent is a pipeline that runs in under a second, thousands of times per conversation. That pipeline has three layers, and the choices you make at each layer define the quality, cost, and capabilities of your agent.

Layer 1: Speech-to-Text (STT)

STT converts the caller's spoken audio into text. Speed and accuracy here directly impact how natural the conversation feels — a slow or inaccurate STT means your agent mishears or delays its response.

Key providers:

  • Deepgram Nova-2 — the current benchmark for low-latency STT. Sub-300ms transcription with high accuracy across accents.
  • OpenAI Whisper — excellent accuracy, especially for technical vocabulary. Slightly higher latency in hosted form.
  • Google STT — mature, reliable, broad language support.
  • AssemblyAI — strong on speaker diarisation and sentiment analysis.

For voice agents, latency beats raw accuracy. Deepgram's streaming API delivers partial transcripts as the user speaks, enabling your agent to start processing before they've finished talking.

Layer 2: Large Language Model (LLM)

The LLM is the brain. It receives the transcribed text, processes it within the context of the conversation, and generates a response. For voice, the LLM needs to be:

  • Fast — first token latency under 200ms is the target
  • Concise — voice responses should be 1–3 sentences, not paragraphs
  • Consistent — staying on-script and avoiding hallucinations

Popular choices:

  • GPT-4o — strong reasoning, fast via OpenAI's API
  • Claude 3.5 Haiku — excellent speed/quality tradeoff for voice
  • Groq-hosted Llama — extremely fast inference, great for latency-sensitive agents

Layer 3: Text-to-Speech (TTS)

TTS converts the LLM's text response back into audio. The voice quality here is what your users actually hear — it's the face of your agent.

  • ElevenLabs — highest quality, most natural prosody. Slight latency overhead.
  • OpenAI TTS — good quality, fast, affordable at scale.
  • Google WaveNet / Neural2 — reliable, broad language support.
  • Azure Neural TTS — enterprise-grade, SSML support, low latency.

Optimising the Full Pipeline

Total response latency is the sum of STT + LLM + TTS. A well-optimised stack looks like:

  • STT streaming → first LLM token in ~400ms total
  • LLM streaming → feed tokens to TTS as they arrive
  • TTS streaming → start playing audio before the full response is generated

Platforms like Vapi handle this orchestration automatically. If you're building from scratch with Pipecat or LiveKit Agents, you'll implement the streaming pipeline yourself — but gain full control over every component.