Voice AI Background Noise Testing: What Breaks STT

A joint piece from Coval and ai-coustics. Coval is a testing and simulation platform that helps teams validate how their voice AI agents actually perform under real-world conditions. ai-coustics builds real-time audio models for voice AI, including Tyto, an audio insight model, and Quail Voice Focus, which isolates the primary speaker.

The myth

Most voice AI teams treat “audio quality” as one problem: suppress anything that is unpleasant or disturbing to a human listener, and you’re safe. ai-coustics’ benchmarking data says that’s the wrong mental model. Worse, teams often spend engineering effort suppressing the one thing that was never going to break their agent, while missing the thing that actually does.

Benchmarking real voice AI audio shows something different. Based on those findings, ai-coustics built Tyto: a lightweight model that analyzes the real-world audio flowing into a voice AI stack (a voice agent, a speech-to-text pipeline, whatever’s listening) and predicts whether that audio is likely to cause a downstream failure. Its main output is a risk score between 0 and 1: low means the audio is fine; high means a failure is likely.

Tyto backs its headline risk score with six qualitative dimensions that explain what kind of problem is present: noise, interfering speech, speaker reverb, packet loss, codec degradation, speaker loudness.

What the data shows

To find out which of these actually break voice AI systems, ai-coustics tested each degradation type by artificially adding it to clean speech recordings. They measured downstream transcription with word error rate (WER) and checked how voice activity detection (VAD) models responded. These controlled benchmarks use reference transcripts to measure accuracy. In production, Tyto estimates downstream risk from incoming audio without requiring a reference transcript.

Benchmark methodology

ai-coustics ran these tests September 11–25, 2026, using 59 English, call-like clips from its internal corpus: 40.9 minutes of speech, with a median clip length of 42.1 seconds. Human transcripts provided the WER reference; human frame-level speech/non-speech labels at 100 Hz provided the VAD reference.

Each clip was tested under 88 degradation conditions across five families: stationary noise, reverberation, reduced Opus bitrate, packet loss, and competing speech. Including a clean anchor for each clip, this produced 5,251 clips and approximately 60 hours of audio per STT engine.

The eight streaming STT configurations were AssemblyAI universal-3-5-pro, Deepgram nova-3, Soniox stt-rt-v5, Mistral voxtral-mini-transcribe-realtime-2602, ElevenLabs scribe_v2_realtime, Gladia solaria-1, Speechmatics enhanced, and Azure Speech (no model parameter). VAD results used Silero VAD 6.2.1 against the human frame labels.

These are controlled English-language benchmark results from the configurations above. They do not establish universal thresholds for every phone call or production voice agent. No listening test was conducted, so the benchmark does not compare model accuracy with human listening performance.

Results and limits

Open the full-size benchmark chart.

WER and VAD accuracy across five degradation types, tested on AssemblyAI Universal-3.5-Pro RT. Shaded regions show the ranges ai-coustics identifies as typical for calls. The chart also shows how Tyto’s risk score correlates with WER and VAD error rate for each degradation type.

Commercial speech-to-text (STT) models can respond differently to the same audio conditions. The chart above illustrates ai-coustics’ results for AssemblyAI; it does not show per-engine results for all eight configurations.

In this benchmark, stationary noise, reverberation, codec degradation, and packet loss were less disruptive to transcription than competing speech over much of the tested range. Transcription degraded more as noise became severe; the point at which that happens depends on the engine and test condition. Reverberation affected both transcription and voice activity detection, while severe compression and packet loss also reduced performance. The chart illustrates one tested STT configuration, rather than a guarantee of performance across providers or production calls.

Competing speech caused substantial insertion errors in the illustrated configuration. A second person talking in the background, or a TV, can end up in the transcript as if the caller had said it. It can also cause voice activity detection to treat background speech as an active speaker or an interruption. Either way, it breaks the flow of the conversation.

Testing for it: the Coval approach

A transcription error can become a wrong order. Testing the full conversation shows whether the agent catches it, recovers, and completes the task correctly.

Picture a drive-thru: an engine idles as a customer orders one burger. In the next lane, someone orders three. A few seconds later, the agent confidently confirms three burgers with the first customer. The customer replies, “No, I said one.” Does the agent correct the order, or carry the mistake into checkout?

Coval simulates conversations with voice agents so teams can test these situations and compare the results. A test case gives the simulated caller a task, such as placing an order. A persona defines how that caller speaks and the sounds around them. Together, they let you ask a practical question: how does the same agent handle this order with an engine running, with another person talking, and with no background sound at all?

Choose from Coval’s background sounds or upload your own recording, then adjust the volume to compare how different audio conditions affect the same task.

Start with those three conditions, keeping the caller’s voice, volume, and behavior fixed. Repeat each across the same set of orders. Now you can see whether the agent struggles broadly with noise or specifically with competing speech. Custom background recordings let you bring in the sounds of a particular restaurant, call center, or other setting.

The first result to check is the final order, including the quantity sent to the ordering tool when that trace is available. Does the agent meet the expected behavior? Coval’s metrics then help explain how the conversation unfolded: did the agent hesitate, repeat a question, talk over the caller, or need another prompt? Transcription checks highlight discrepancies worth reviewing. Check for dropouts or repeated audio when the recording suggests those problems.

Example drive-up evaluation grouped by persona. Coval brings audio-artifact measurements, recovery, and task outcomes into one report. The same view can compare personas configured with different background sounds.

If you find a failed order, follow it from the report into its recording, transcript, and available traces. Listen for the competing order and check where the agent’s response diverges, using its own STT trace if available. Then follow the customer’s correction through to the final action. ai-coustics’ Tyto analysis of the same caller audio can add context about competing speech around that moment, giving you a hypothesis to investigate.

That makes the next decision easier to evaluate. Try another STT configuration against the same orders and audio conditions. Use a statistically significant number of simulations to check whether your agent captures the right quantities, keeps the conversation moving, and still performs well on clean calls. Save those cases and their pass criteria for regression testing in the next release, then expand coverage to different callers and combinations of conditions.

Fixing it: how Quail Voice Focus helps

Coval’s tests show where and why an agent struggles. Quail Voice Focus is ai-coustics’ answer to what to do about it.

Voice Focus is a real-time speaker isolation model for single-speaker human-to-agent scenarios. It uses recent audio context to select a primary speaker and suppress competing speech, background noise, and residual echo. Test the enhanced audio with your STT provider, including quiet target speech, overlap, and speaker changes: stronger suppression can also remove speech you intended to retain. For ai-coustics’ dedicated VAD, use the original input audio; evaluate third-party VAD behavior separately.

Voice Focus sits upstream of whatever STT a team chooses. That makes it a complement to Coval’s stack comparisons, useful no matter which provider gets tested.

That scope lines up with what the benchmarking data found. Voice Focus targets interfering speech and background media speech directly, both shown earlier to break transcripts and trip false VAD triggers.

Suppression isn’t always the right call, though. In fraud detection, for instance, a second voice coaching a caller through a loan application is a signal you want to detect. Suppressing it defeats the purpose. That’s an argument for applying Voice Focus deliberately, per use case.

Closing

Most voice AI teams have a hard time tracing a pipeline failure back to audio. Coval and ai-coustics close that gap from three different angles. Coval’s simulations show where an agent struggles under real conditions, and Tyto adds a risk score and six qualitative dimensions to help explain the audio conditions. Quail Voice Focus fixes what’s fixable, applied deliberately depending on the use case.

Try Coval’s testing platform · Try Tyto

Further resources

• ai-coustics’ open-source dataset of phone calls with interfering speech: https://huggingface.co/datasets/ai-coustics/aic_test_calls_en
• Coval’s open-source benchmarks: https://benchmarks.coval.ai/