How to Evaluate a Voice Provider: A Field Guide from Cartesia and Coval

A voice can sound right in a ten-second demo and still fail the first week of real calls. It can hesitate before speaking, stumble over account numbers, mispronounce a product name, or drift during a long conversation. None of those failures are captured by asking which demo sounds best.

Cartesia and Coval approach this problem from different sides. Cartesia builds real-time voice models. Coval independently benchmarks speech models and evaluates the agents built around them. We agree on one point: choosing a provider is not a single ranking exercise. It is a sequence of tests that gets closer to your deployment at every stage.

Key takeaways

• Use an independent benchmark to build a shortlist, not to make the final decision.
• Measure objective performance on your own inputs, then test subjective voice quality with blind listening.
• Compare providers inside the same end-to-end agent under the same scenarios.
• Keep the evaluation running after launch, and turn production failures into regression tests.

Stage	The question it answers	Evidence to collect	Move on when
1. Independent benchmark	Which providers are credible candidates?	Current latency and intelligibility data, plus blind listening	Two or three candidates clear your minimum requirements
2. Your data	Which candidate handles your content best?	A fixed input set, metric distributions, and listening results	One or two candidates meet the same prewritten rubric
3. End-to-end simulation	Does the voice work inside the agent?	Multi-turn outcomes, tool use, turn-taking, and failure traces	The complete agent clears release thresholds
4. Production evaluation	Does it keep working for real callers?	Live quality trends, outliers, and new failure clusters	This stage does not end

Stage 1: Use an independent benchmark to build the shortlist

Start with an independent benchmark to avoid spending a full evaluation cycle on every provider in the market. Coval’s live Voice AI Benchmarks measure text-to-speech latency and intelligibility across current models with the same runner and datasets. The benchmark code is public, so teams can inspect how each result is produced.

If Cartesia is on your shortlist, its dedicated provider page brings the models Coval measures and their current benchmark results into one view.

Use the leaderboard for the questions it actually answers. For text-to-speech, that means objective measures such as time to first audio and word error rate. Those metrics are useful because they are collected consistently across providers. They do not tell you what a model costs, whether a particular voice fits your brand, or how it will perform on your scripts.

Naturalness needs a different instrument. The Voice Arena plays two anonymous outputs for the same text so listeners can choose without seeing the provider. Pricing should be compared directly from provider quotes for your expected volume, contract terms, and support requirements.

The output of Stage 1 should be a shortlist, usually two or three providers, plus a written record of why each candidate advanced. If a requirement is not represented on the leaderboard, do not smuggle it into the ranking. Put it in the next stage.

Stage 2: Test the finalists on your own data

Public benchmarks control the inputs so providers can be compared fairly. Your evaluation should do the opposite: hold the providers constant and vary the inputs across the content your agent will actually say.

Build a fixed set before you listen to the outputs. Include the phrases most likely to expose differences:

1. Opening lines and other high-frequency utterances.
2. Account numbers, confirmation codes, dates, currencies, and addresses.
3. Product names, proper nouns, abbreviations, and industry terms.
4. Compliance disclosures and long-form responses where pacing matters.
5. Every language, accent, and locale you plan to support.
6. Inputs taken from real failure cases, with sensitive data removed.

Write the acceptance rubric before running the candidates. That keeps the team from changing the definition of “good” after hearing a preferred voice. The rubric should separate measurable requirements, such as latency or skipped words, from listening judgments, such as warmth, pacing, and brand fit.

Time to first audio (TTFA): Coval’s TTS methodology measures the time until the audio arrives and adds any leading silence before the first audible sample. A provider may publish time to first byte using a different start or end point. Neither number is useful without its definition.

Compare candidates using the same measurement path, the same region, the same connection setup, and a distribution such as median and tail latency rather than one fast request.

For listening tests, hide the provider and voice names, randomize the order, and ask reviewers to score the same dimensions. Keep the raw outputs. They become evidence for the decision and a regression set for future model versions.

Stage 2 ends when the shortlist has been tested against the same inputs and the same prewritten rubric. It does not end when one person finds a favorite demo.

Stage 3: Put each voice inside the complete agent

A single utterance cannot reveal whether a voice works in a conversation. The text-to-speech provider sits inside a chain that includes speech recognition, turn detection, an LLM, tools, telephony, and network conditions. A strong isolated result can still produce a poor call when those parts interact.

At this stage, swap one provider at a time inside the agent you plan to ship. Run the same simulation test set against every candidate. Keep the prompt, tools, persona, audio conditions, and success metrics fixed so a provider change is the variable under test.

Coval Simulate

Test the complete agent, not an isolated voice.

Run the same personas and scenarios across providers, then inspect the outcome and trace behind every failure.

The scenarios should cover more than clean, cooperative calls. Test interruptions, double-talk, silence, background noise, long responses, tool latency, transfers, retries, and conversations long enough to expose pacing or voice drift. Include the business outcome too. A natural voice that consistently fails the handoff is not the better production choice.

This stage also tells you where the provider decision stops. If every candidate fails the same scenario, the problem may be turn detection, prompting, a tool, or the telephony path rather than the voice model. The trace from a full simulation gives the team something more useful than a lower composite score: a failure to fix.

Move forward when the complete agent, with the selected voice, clears the release thresholds your team set in advance.

Stage 4: Keep evaluating after launch

Provider selection is not finished when traffic starts. Models, infrastructure, routing, prompts, and callers change. The same evaluation logic needs to continue on production conversations.

Use production observability to track the outcomes that matter for the use case, then segment failures by provider, model version, language, call type, or deployment change. Watch distributions and outliers, not only averages. A stable median can hide a growing tail of callers waiting too long or receiving a broken pronunciation.

The operating loop is simple:

1. Detect a production failure or a meaningful shift in a metric.
2. Inspect the conversation and identify the component that failed.
3. Turn the case into a sanitized simulation or single-turn test.
4. Run it before the next provider, model, prompt, or agent release.

This is what makes a provider evaluation durable. The work from Stages 2 and 3 becomes a reusable test suite rather than a spreadsheet that expires after procurement.

Make the decision with evidence you can rerun

No single score can choose a voice provider for you. The useful question is whether each stage produces evidence that is closer to the system your callers will experience.

Start broad with independent data. Narrow the field with your own inputs and blind listening. Test the finalists inside the complete agent. Then keep measuring after launch and feed failures back into the suite.

That process gives model builders, application teams, and buyers a shared record of why a provider was chosen and what would cause the decision to change. It also makes the next evaluation faster. When Cartesia or another provider releases a new model, you already have the data, scenarios, and thresholds needed to test it.