Speech-to-Speech vs Cascaded Voice AI: Which Should You Deploy in 2026?

A cascaded system lets you check every reply before the caller hears it; speech-to-speech wins on conversational feel. The evidence on speed, accuracy, and cost, and how to test both.

Deciding between speech-to-speech and cascaded models comes down to one question: does anything have to happen between your agent deciding what to say and the caller hearing it, like checking a refund offer against your policy? If it does, build a cascade, which chains three stages. First, a speech-to-text (STT) model transcribes what the caller says. Next, a large language model (LLM) reasons about it using the policies you’ve set, calls any tools, and writes the reply as text. Last, a text-to-speech (TTS) model speaks that reply.

Because the LLM’s reply is text, an automated check can read it before anything is spoken: your own policy rules, a guardrail, or a second model. If the reply offers a refund your policy forbids, the check changes or blocks it, and the TTS model never speaks it. That’s the checkpoint, and no one has to listen to anything in the middle. If nothing needs that checkpoint, choose speech-to-speech (S2S). One model hears the caller’s audio and answers in audio, and the conversation feels more natural.

The cascade also lets you debug, fail over to a backup, and upgrade one stage at a time. What it costs you is feel, plus some speed, mostly on plain turns, where the agent answers without calling a tool. On tool calls and task accuracy, the models you pick matter far more than the architecture, so choose on the checkpoint and settle the rest by running both versions of your agent through the same simulated calls in Coval.

Here’s what a cascade costs you and what it buys. The first three rows come from EVA-Bench, a September 2026 research benchmark in which a simulated caller talked to 12 voice systems (seven cascades, three speech-to-speech models, and two hybrids of the two) across 213 enterprise scenarios. Its task-accuracy and experience scores are the share of attempts that pass, from 0 to 1, so higher is better. The last three rows come from vendor documentation, covered below.

CriterionCascadedSpeech-to-speechVerdict
Response time2.16 to 4.29 s plain turns; 2.31 to 5.37 s with a tool1.01 to 1.29 s plain turns; 1.87 to 2.85 s with a toolS2S; close on tool turns
Task accuracy and tools0.205 to 0.504 task accuracy0.163 to 0.467 task accuracyTie: the build decides
Conversational feelExperience score 0.273 at bestExperience score 0.406 to 0.589Speech-to-speech
Checking replies before speechReply exists as text before TTSTranscripts and OpenAI’s guardrails run beside the audio, not before itCascaded
Debugging, fallback, upgradesTrace, back up, and swap each stageOne model; a retirement swaps the whole agentCascaded
CostPer stage, or bundled at $4.50/hrAudio tokens or per-minute billingTie: compare per completed task

How cascaded and speech-to-speech agents work, and the hybrids in between

The checkpoint lives wherever the reply exists as text. In a cascade, that’s the handoff between the LLM and the TTS stage. A speech-to-speech model goes from audio to audio, and any text is a by-product.

Two flows compared. In a cascade, caller audio goes to speech-to-text, the LLM writes the reply as text, a check step (your policy rules, a guardrail or a second model) can change or block it, then text-to-speech speaks it to the caller, so a refund your policy forbids is blocked before it is spoken. In speech-to-speech, one model hears and answers in audio, and transcripts and guardrails run beside the audio rather than in front of it. Where a reply can be checked before the caller hears it.

Vendors now ship the shapes in between. OpenAI documents three paths: a chained pipeline, the Realtime API (its speech-to-speech model), and GPT-Live, a voice front end that keeps your existing text workflow as its backend. LiveKit’s half-cascade has a speech-to-speech model do the listening and reply in text, which a separate TTS then speaks, so the reply exists as text you can check. Deepgram’s Voice Agent API is a cascade sold as one API. Our voice AI models guide covers the models behind each path.

Speech-to-speech answers plain turns faster; tool calls close most of the gap

Speed is the first thing people expect a cascade to give up, and on plain turns it does. Speech-to-speech answers about a second faster, but once the agent calls a tool, the best cascades close most of that gap. In EVA-Bench, the fastest cascade (Whisper for speech-to-text, Qwen3.5-27B as the LLM, and Voxtral for text-to-speech) answered tool turns in 2.31 s, faster than both GPT-Realtime-1.5 (2.40 s) and Gemini-3.1-Flash-Live (2.85 s). Only GPT-Realtime-mini, at 1.87 s, was quicker.

Newer speech-to-speech models aren’t automatically faster. Kwindla Hultman Kramer of Pipecat, the open-source voice agent framework, times models across 30-turn scripted conversations in his aiewf-eval benchmark. In its September 16, 2026 results, GPT-Realtime-1.5 still had the fastest median response on turns without tools, at 1,120 ms. GPT-Realtime-2.1 and Gemini 3.8 Live were both slower. Our own speech-to-speech board times the whole wait at p95 instead, and over the seven days to November 9, 2026, it ran from 1,101 ms for PolyAI’s Dialog-RSN-1 (a hybrid that hears the caller’s audio directly and hands its reply to a separate TTS voice) to 2,597 ms for Gemini 3.1 Flash Live, with Gemini 3.8 Live at 1,253 ms. Generation alone doesn’t predict speed, so time the specific model you’d ship.

Inside a cascade, the LLM sets the pace, because the speech stages are quick. On our STT and TTS leaderboards (30-day averages to October 28, 2026), Nova 3 returns a final transcript 86 ms after the caller stops speaking, and Universal 3.6 Pro takes 121 ms. Flash v2.5 makes its first sound 198 ms after synthesis starts, and Sonic 3.6 takes 292 ms. That adds up to roughly 0.3 to 0.4 s for both speech stages. Pipecat’s voice-readiness benchmark, which tested 46 text LLM setups from seven providers, says an LLM that takes much above 700 ms to its first answer token is too slow for most voice agents. So when a cascade feels slow, start with the LLM, and measure voice latency across the whole turn.

Task accuracy depends on the build, not the architecture

Accuracy doesn’t pick a side. In the matched tests published in 2026, the architectures tie and the builds don’t. EVA-Bench’s task-accuracy score counts the attempts that complete the task, follow policy, and speak key details correctly. The best cascade (Nova-3 + GPT-5.4 + Sonic 3) scored 0.504, and the best speech-to-speech model (GPT-Realtime-1.5) scored 0.467. That’s 0.037 apart, while the seven cascades alone ranged from 0.205 to 0.504 and the three speech-to-speech models from 0.163 to 0.467.

Tool use doesn’t favor the cascade either. Full-Duplex-Bench-v3, a 100-scenario test from National Taiwan University and NVIDIA, puts voice models through natural speech and multi-step tool use. On its tool tasks, GPT-Realtime’s first-attempt pass rate was 0.600 (on a 0-to-1 scale), against 0.450 for a Whisper + GPT-4o + OpenAI TTS cascade.

Single runs flatter everyone. When EVA-Bench required a scenario to pass all five of its trials, the best cascade fell from 0.504 to 0.217 and GPT-Realtime-1.5 from 0.467 to 0.283. Whichever you build, run each scenario more than once before you trust its score.

Speech-to-speech wins on turn-taking and conversational feel

The price a cascade really pays is how the call feels, and speech-to-speech wins that clearly. EVA-Bench’s experience score is the share of attempts that pass on all three of turn timing, conversation progress, and concise spoken replies. The three speech-to-speech models scored 0.406 to 0.589, and no cascade scored above 0.273.

When callers changed their request mid-sentence in Full-Duplex-Bench-v3, its cascade’s pass rate was 0.176 against GPT-Realtime’s 0.588. Speech-to-speech also copes better with accents. Under French-accented speech in EVA-Bench, the share of calls where cascades completed the task fell by 10 percentage points on average (17 for the worst system), while no speech-to-speech model’s score dropped significantly. Speech-to-speech’s weak spot was coffee-shop background noise, which EVA-Bench found “exposed S2S experience failures.”

Cascades let you check every reply before the caller hears it

This is what the cascade gets in return. OpenAI’s own voice agents guide describes the chained path as the way to “inspect or transform intermediate text.” OpenAI’s Realtime stack also gives you transcripts and guardrails, and Gemini Live gives you transcripts, but they run beside the audio, not in front of it.

  • OpenAI Realtime API: the caller transcript comes from a separate transcription pass that is off by default and runs asynchronously, and OpenAI says to treat it “as guidance of input audio content rather than precisely what the model heard,” so any check on the caller’s words works from an approximation.
  • OpenAI Agents SDK: realtime output guardrails run on accumulated transcript text and cancel the response when one trips, but “some audio may already be buffered when the tripwire fires.”
  • Gemini Live native audio: the models answer only in audio; to get the reply as text, you transcribe that audio.

A half-cascade puts the checkpoint back: an OpenAI Realtime session can return text only, which you check and then send to your own TTS.

If compliance is your reason for wanting the checkpoint, HIPAA doesn’t demand one. Its Security Rule requires audit controls that record and examine system activity (more in our HIPAA voice AI guide).

Cascades are easier to debug, fail over, and upgrade

The chain pays off again after launch, because each stage can be traced, backed up, and replaced on its own. LiveKit’s fallback adapters switch STT, TTS, and LLM providers on any error. A speech-to-speech model, by contrast, listens, reasons, and speaks as one unit, so there’s no single stage to swap.

Debugging starts with knowing which stage is slow. When a cascaded agent sends OpenTelemetry traces (timing records from each step of a call), our trace metrics report how long the STT, LLM, and TTS stages each take to start responding, plus tool-call counts per tool.

Upgrades are where the difference bites. Google lists November 17, 2026 as the earliest shutdown for Gemini 3.1 Flash Live, replaced by Gemini 3.8 Live, which removes affective dialogue (the feature that lets Gemini adapt its response style to the caller’s expression and tone) and makes async function calling the default. OpenAI removes gpt-realtime, gpt-4o-realtime, and gpt-realtime-mini on January 20, 2027. In a cascade, a retirement like that swaps one stage; in speech-to-speech, it swaps the whole agent.

Neither is cheaper by default; compare cost per completed task

Price won’t settle it for you, because cost depends on call length and who does the talking. OpenAI bills Realtime audio as tokens and resends the whole conversation for each response, so later turns cost more. At GPT-Realtime-2.1’s list prices, that’s $0.0192 per minute of caller audio and $0.0768 per minute of agent audio, not counting the resent conversation. Gemini 3.8 Live lists $0.005 per minute in and $0.018 out. GPT-Live 1 costs $0.05 per voice minute plus your backend’s usage. On the cascade side, Deepgram’s bundled Voice Agent API is a flat $4.50 per hour ($0.075 a minute) on its full stack. The cheapest list price buys nothing if its calls fail more often. OpenAI’s cost guide recommends comparing cost per successful task alongside completion rate, and that’s the right test.

Speech-to-speech vs cascaded: which to choose, and how to test both

So go back to the refund your agent shouldn’t promise, and ask whether that promise has to be stopped before the caller hears it. Your answer picks the architecture:

  1. If any reply must be checked, changed, or blocked before it’s spoken, or you need per-stage fallback, build a cascade.
  2. If nothing needs that check and the feel of the call matters most, use speech-to-speech, and run your checks on the recorded audio after each call.
  3. If you need the check but want speech-to-speech’s ear for accents and mid-sentence changes, try a hybrid: a half-cascade, or a voice front end with a text backend. Kwindla told us that much of what Pipecat helps customers deploy now “feels like a thinking fast and slow split.” Measure its feel before you commit, though. EVA-Bench’s two hybrids (Gemini-3-Flash + Gemini-3.1-Flash-TTS, and Ultravox-Realtime) each pair a model that understands audio directly with a separate TTS. They scored 0.000 and 0.029 out of 1 on conversational experience, below the top cascade on that score (0.273) and every speech-to-speech model.

That settles the architecture, but only a matched test tells you which build to ship. Here’s how to run both through the same calls with Coval simulation:

  1. Connect both builds. The cascade through Pipecat Cloud, LiveKit, or a phone number; speech-to-speech through our direct OpenAI Realtime, Gemini Live, Grok Realtime, or Deepgram Voice Agent connections.
  2. Write one test set. Plain turns, a tool call, a mid-sentence correction, a number to read back, and a refund your policy forbids, run with personas that have a regional accent, background noise, and a high interruption rate.
  3. Score both the same way. Latency and Interruption Rate are timing metrics computed directly from the call. An Audio Binary LLM Judge answers your question (“Did the agent promise a refund the policy forbids?”) from the recording, so you score what the caller heard. Tool evidence comes from traces on the cascade or the Deepgram Voice Agent connection’s recorded tool calls.
  4. Compare variants in one run. Mutations swap the model, voice, instructions, or temperature on the OpenAI Realtime or Gemini Live connection, with results grouped by variant.
  5. Repeat and schedule. Save the setup as a template, re-launch it to rerun every scenario, and put it on a schedule so a model change shows up in your results before your callers hear it. The same setup runs vendor bakeoffs between providers.

Run that set on both builds and you’ll hear which one kept the refund off the call, and what it paid for that in speed and feel.

Test both architectures Run your cascade and a speech-to-speech build through the same simulated calls, and let the scores pick.

Get deployment-ready.