Voice AI Benchmark Methodology 2026: How Coval Measures Latency and Accuracy

Two voice AI benchmarks can test the same text-to-speech model, publish different latency numbers, and both be honest. A number on a leaderboard is made up of a handful of factors that each benchmark sets for itself, and until you know them, you can’t tell whether that number says anything about your agent.

A voice AI benchmark methodology is those factors, written down: what audio or text every model runs on, where the stopwatch starts and stops, how errors are counted, and what gets averaged over what window. We set each one the way a caller would experience it. Coval’s benchmark tests almost 40 providers across speech-to-text (STT), text-to-speech (TTS), and speech-to-speech models, and every model in a category gets the same inputs. On speech-to-text, those run from a clean control set to the accents, noise, and phone-line compression of real calls. The clocks that time a caller’s wait stop where it really ends: at the final transcript your agent needs, or the first sound your caller hears. Word errors are pooled across the whole test set, so a short clip can’t outweigh a long one. Results are a rolling 30-day average shown next to their spread, and the runner and method are open source.

Below, we take each factor in turn and show, with our own benchmark, what it does to a result. Then we read one row of our leaderboard together and show how to run the same method on your own callers.

The inputs: audio that sounds like real calls, shared by every model

The inputs decide which callers a number describes. Score a model only on clean read speech and you learn how it handles clean read speech. LibriSpeech, a widely used clean read-speech set, is also “heavily represented in provider training data,” as our methodology puts it, so models’ error rates on it come out artificially low.

So for speech-to-text we build the inputs around the audio a phone call actually delivers, plus one clean set to measure it against, and every model in a category gets exactly the same thing. Each model is called by its exact versioned ID, such as nova-3, never a floating alias like “latest” that a provider can quietly point at a newer model. Each run draws one random sample of clips, and every model is scored on that same draw.

Our STT board runs on seven audio sets, listed in our dataset catalog. Six come from WildASR, Boson AI’s diagnostic benchmark built from real human speech, and one from the public test data of Pipecat, an open-source framework for building voice agents.

Audio set	Clips	What it tests
Pipecat voice-agent turns	897	Spontaneous turns, with fragments and fillers
WildASR accents	845	English across demographic accents
WildASR clean	284	Undistorted read speech: the baseline
WildASR clipping	284	The clean clips with peak distortion
WildASR far-field	284	The clean clips re-rendered as if recorded far from the microphone
WildASR noise gaps	284	The clean clips with noise bursts and silences inserted
WildASR phone codec	284	The clean clips passed through telephony compression

The clean set is the control. The four rows below it test noisy audio, and they aren’t new recordings. If a model’s word error rate (WER) climbs between the clean set and the phone-codec set, the phone line did that, not a harder sentence: each degraded set “uses the same utterance and transcript as the WildASR clean baseline, so the WER change isolates the effect of the test condition.” Our best STT providers roundup shows which models hold up.

Text-to-speech runs on text instead: 30 short customer-service prompts covering order tracking, appointments, account verification, and tech support, plus longer agent passages, so every voice reads the same lines.

The clock: stopped where the wait actually ends

“Latency” can name very different waits. A text-to-speech stopwatch can stop when the first byte of audio arrives or when the first audible sound does, and those two moments can be far apart. So each clock below starts and stops at a defined event. These are our definitions:

A voice-activity model decides when someone has stopped talking, and sharing one means every provider’s TTFS timer starts from the same instant. Speech-to-speech models hear the caller and answer in audio directly, so we time them from the call recording, not from events the provider reports. For how these per-component latencies add up across a full turn, see our guide to measuring voice AI latency.

Two conditions hold for every provider we time. Connection setup (the TCP and TLS handshakes and any session setup before the request) is excluded for every provider. And our workers run in us-east-1, so a provider serving from another region carries the longer trip to and from its servers in its numbers.

Here’s how far the stop point moves a result. Over the 30 days to October 28, 2026, Cartesia’s Sonic 3.5 took 291 ms on average to produce its first audible sound. Only 160 ms of that was the wait from sending the request to the first byte of audio arriving back, which is where a first-byte stopwatch would stop. The other 131 ms was leading silence: inaudible samples at the start of the audio stream that a caller sits through before hearing a word. Time it to the first byte and Sonic 3.5 looks almost twice as fast as it sounds.

We measured TTFA to the first byte of audio until May 2026, then moved the stop point to the first audible sample. Our TTFA benchmark post shows what that change did to the leaderboard.

When a latency can’t be measured fairly, we don’t publish it. Some STT endpoints send partial transcripts on a fixed schedule rather than as soon as text is ready, so their first-partial time would measure the schedule, not the model. We leave those endpoints out of that ranking instead of publishing a misleading number.

The score: errors counted the same way for every model

A four-word clip with one wrong word shouldn’t count as much toward a model’s score as a 41-word clip with none. Under our old scoring, it carried exactly as much.

Word error rate is the share of words a transcript gets wrong against a reference transcript: substitutions plus deletions plus insertions, divided by the number of reference words. Lower is better. Until September 2026, our headline WER was the average of each clip’s own rate. Those two clips score 25% and 0%, so their average is 12.5%. Pool their errors and words instead, one error in 45 words, and you get about 2.2%. Since September, our headline WER has been pooled that way across the whole test set.

Numbers from before and after that change aren’t comparable, just like TTFA numbers from before and after May. That’s our rule for every scoring change: results are “marked incomparable rather than quietly blended,” so no trend line mixes two methods.

Before anything is counted, the reference and the model’s transcript both pass through the same text normalizer, Whisper’s EnglishTextNormalizer, so formatting differences don’t count as errors. “Twelve” and “12” count as the same word.

Text-to-speech gets a word error rate too, as a measure of intelligibility. One fixed STT model, OpenAI’s whisper-1, transcribes every synthesized clip, and that transcript is scored against the prompt with the same formula. With the same listener for every voice, a difference in WER comes from the voices. That tells you whether speech can be understood, not whether it sounds natural. Listener-preference scores, such as mean opinion scores (people rating clips from 1 to 5) or head-to-head arena votes, answer that separate question.

On the speech-to-speech board, the question is whether the model did what it was told. Instruction adherence is the share of evaluated requirements a model satisfied, with every model running the same multi-turn simulated calls under the same instructions.

How to read a number on a voice AI benchmark leaderboard

Every factor above shows up in a single leaderboard row, if you know where to look. Results combine runs from the last 30 days, and the headline is the 30-day average. Next to it sit p50 (half the samples came in at or under it), p95 (95% did), and the max, the single worst sample in the window. The sample count tells you how much evidence sits behind the row.

Here are Cartesia Sonic 3.5’s numbers for the 30 days to October 28, 2026:

Statistic	Time to first audio	Word error rate
Average	291 ms	1.6%
p50	279 ms	0.0%
p95	394 ms	13.2%
Max	6,167 ms	52.6%
Samples	2,875	2,822

The latency column is tight. The average and p50 sit 12 ms apart, and p95 is about 100 ms above the average, so the typical wait is predictable. The 6,167 ms max is a single sample, the worst of 2,875, while p95 says 19 of every 20 samples came in at or under 394 ms.

The word error rate column, the intelligibility score from the previous section, reads differently. An average of 1.6% next to a p50 of 0.0% means at least half the prompts came back word-perfect and a few missed badly, which the 13.2% p95 and 52.6% max confirm. For an agent that reads back order numbers and appointment times, those few misses are the ones a caller notices.

You can also check our work. The runner is open source under Apache-2.0: clone it, add your own provider API keys, and run it. You’ll get a result comparable to current measurements, not a copy of a leaderboard cell, because each run draws a fresh random sample and the leaderboard combines many runs across the window.

Run the same voice AI benchmark methodology on your own callers

A public leaderboard narrows the field. It can’t tell you how a model handles your callers: their accents and languages, the phone lines they call from, and the names, dates, and account numbers your agent has to get right. As our benchmarking team put it in our August 2026 AMA, “making those benchmarks contextual to your data is really going to give you higher signal.”

The same method carries over to your own data in two steps.

Compare the components on your audio. The Coval app’s Benchmarks feature compares STT and TTS providers on your own audio and text, up to 10 providers per run. STT runs report word error rate and latency; TTS runs report TTFA at p50 and p95, total latency, and audio duration. Benchmarks is in Beta and available on request.

Then test the finalists inside your agent. In a simulation, Coval plays the caller, talks to your agent end to end, and scores the conversation. You pick who calls with a persona: their voice, language and regional accent, background noise (from a built-in library or your own recordings), and how they behave. You pick what they call about with a test set, a list of scenarios. Then run the same test set against each candidate model or vendor, and score every run on two things a public leaderboard can’t measure for you:

• the Latency metric, the average silence gap between your caller finishing a turn and your agent starting to respond, counting only gaps of 0.5 seconds or longer;
• an API State check, which calls your backend after the conversation and confirms that the booking, refund, or account change actually happened.

That’s a vendor bakeoff: define the calls once, run the exact same calls against each candidate, and decide on your callers’ results instead of a leaderboard average.

Read the method first, then let your calls decide

A voice AI benchmark number means exactly what its inputs, clock, score, and window say. Read those before the headline, and a leaderboard becomes a fast, honest way to build a shortlist. Then run that shortlist through your own callers, in your own agent, and let those results make the call.