Improve Agent Performance With Voice Benchmarks: AMA Replay and Show Notes

Awesome to see some folks trickling in. We'll get started in a minute or two.

Hello, everyone. Thanks so much for joining today. Can everyone hear? Today is our second AMA, and we're going to talk through our benchmarks. We host the largest open-source voice benchmarks specifically designed for voice AI.

We consistently hear from customers that choosing voice AI models is difficult. There are STT, TTS, and LLM options, plus the choice between speech-to-speech and cascaded systems. Today, we're going to talk through the benchmarks our team has built. I'm Brooke, founder of Coval. We're building an evaluation platform to help enterprises scale to millions of calls, including Fortune 500 companies and organizations in financial services and healthcare. Our team brings experience from Waymo and works closely with model providers across the community.

Our benchmarks team—Cale, Madina, and Cooper—has built speech-to-speech, TTS, and STT benchmarks and developed the metrics on top of them. We're excited to share what we've built. Benchmarks help answer which models to use and when it makes sense to explore a new model for an application already in production. We'll share the framework behind our benchmarks, walk through the live leaderboards, and discuss how different models are performing.

And then we're going to have a sneak peek and our first look at the voice arena that we just launched. being able to judge the naturalness of different models. and finally we're going to have an AMA question and answer of about all different models that we're seeing in production and any other questions that you guys might have. So feel free to throw your questions in the chat and we will get to them towards the end of today's webinar. Amazing. And so yeah just a reminder on So a lot most voice systems today are cascaded systems. So it's a combination of STT, LLM, and TTS.

So basically the model is trying to perceive what is happening in the world around us. what should I say next and then how can I actually say that thing out loud. So this is actually very similar to an autonomous system which is how do I make sure that how do I perceive the world around me and then actually reason about what to do next or plan my next steps and then take that action. and so these are all the parts of the stack that we benchmark as well as we then benchmark the the end capabilities.

Speech-to-speech and the whole conversation end to end. and so this is where kind of speech-to-speech versus cas cascaded systems are important and then voice agents go in this loop and that's why real time latency is so important is because many iterations lead to this loop going back and forth. And so benchmarks are a really important part of the eval stack. So Coval itself we don't actually sell benchmarks. We make no money off of benchmarks. All of our benchmarks are completely open source and they're independent evaluations of how each provider is performing in the system. We don't have any financial incentive for any of the providers that we benchmark. And I think this is really important because a lot of benchmarks are either easy to game or they're one shot of exactly it's only one test and we actually have continuous eval.

On top of that, it's also really important to be able to be unbiased and so it doesn't matter which provider is doing the best on the benchmarks, but benchmarks are super important for your eval. So it's a important piece of first you start with benchmarks which are generalized just in the world like generalized benchmarks across all use cases for general data sets but then the next step is actually doing internal benchmarks. So this is something you can do in our platform or something you can run locally on your own machine. But it's like how do you run these same benchmarks of a bunch of different models for let's say three to five of the

Top models that you're looking to experiment with. And the advantage of doing your own internal benchmarks is that you can run it on your own personal data. And then finally, you do your finest grain evals, which are going to be task specific evals. and so these are kind of in order of The value of benchmarks.coval.ai is that you can see how well different things are performing in production without necessarily having to integrate it into your stack from day one. But obviously benchmarks have their flaws which is the data might not be representative of your data. it might be gamed we try really hard not to make our benchmarks gameable. and then also just eventually you're optimizing for the average and not for your specific use case. so yeah that's what we're going to talk like that's why I think the benchmarks we have today are so important.

If we go to the next slide, we can see what our benchmarks look like today. Here's the benchmarking page, with an overview of the different models. We evaluate speech-to-text, text-to-speech, and speech-to-speech systems across latency, word error rate, naturalness, and other metrics. Not everything that matters in a model is captured yet. In the future, we want to incorporate more languages and capabilities such as emotion, prosody, and relevance. We're constantly improving the benchmarks, so we want to hear what else you're looking for.

And now I'm gonna have Cale, one of our engineers on benchmarks, come and share a bit more about how do we actually run and measure these different models and how do we make this possible. Cale, thanks so much for joining us on this webinar.

Stoked. Hey, I'm Cale. I help build benchmarks at Coval. And yeah, something that we get asked a ton is what model should I use? And initially to answer this question we just set our favorite models. we also looked at public benchmarks and we very quickly found out that with public benchmarks there were some flaws in the they were being ben in the way voice models were being benchmarked. that made it hard to make any decisions based on those and that's why we made our benchmarks. And so what does that look like?

First of all, the public benchmarks that we've seen are incredible, but they're constrained to one-off tests of a happy path. And so the first part of that, the one-off test means that you only see the In many cases, for voice models. really latency is a first class citizen. So seeing best case scenario latency only does so much to tell you what's going to go on. really what we try and do what we found is that the server is just as important as the model in these instances.

And so a good example of that is if you have a very slow model that has very high quality and you run it on a really fast system, it looks like a fast model with high quality to your customer. on the flip side, we've seen really really fast models that have pretty good quality that for the engineers at your company, they're going to be all about this. But if you run that on a slow server system or something that isn't reliable to your customer, that looks like a really horrible model.

Yeah. And that's actually why I love the violin charts that we have on our benchmarks so much because it shows the continuous latency as well as the margin as well as the distribution of that latency because as you know you have many turns within a conversation. And so P95 latency is actually super important.

And so yeah, that's why I love those violin charts where you can actually see the range of even if the min latency is really really low. If the max latency is 1.5 seconds, like that conversation, you just assume that the agent is responding by the time you get everything else back. And like you said, that's why we run our benchmarks continuously. We don't want to do a one-off happy test. We actually run them every 30 minutes. And the reason we do that is we are trying to catch those spikes in latency that may not show up on a status page for a provider. But for a customer, a 15-second spike in latency feels exactly the same as if the entire service goes down. what you're looking at here is actually our speech-to-text benchmarking UI.

And one thing I want to highlight with this is the fact that we're not trying to encapsulate it all in a single number. we're giving you tons of little dials and buttons to click with the goal that you first look at the page and you see the most general form of benchmarks. But over time as you implement these models in your system, you're going to find certain edge cases that break down for you. And we want you to be able to dive in there and dial in exactly which models perform well on those edge cases you need to hit. So a good example of that is actually our word error rate scores. most benchmarks will have some global word error rate that is useful while choosing a model but when it begins to break on something like accents or clipping it's really impossible to tell what the best model is there for we break it down into these category components.

So that you can find the use cases that are hurting your customers the most. and really find a model to fix that for And I think I love this about there's no one good mo like the best model across everything. obviously you want to be better in all regards, but I think the industry is progressing so fast that a lot of these models are becoming really good at certain things and then becoming better overall certainly over time. but I love how this breaks it down into kind of what are you trading off by making it faster or cheaper or higher quality so that you can make those engineering decisions. Exactly.

And that's really where our benchmarks are focused is those engineering decisions like we think that you should go towards most benchmark providers when you're concerned about individual models and when you're thinking about the actual serving infrastructure and what is going to be hit from your endpoint. we want you to dial a bit more into looking at the whole stack. And so a good example of this is our TTS models have a latency that measures perceived time to first Mo most benchmarks they'll measure how long until audio comes back. But when we really dug down into it, what we're seeing with a lot of these is that the audio that come back wasn't the same. Some audio would start immediately while some would have 800 millisecond delays, just silence that they were sending back.

And when we added that into our latency benchmarks, the entire leaderboard flipped on us. And that's one of those little things where if you're just measuring like token throughput on a model, you're never going to catch that. Yeah, I think latency is so deceivingly hard actually to measure. We find this even in task end to end voice eval a lot of times people find out that telephony adds a ton of latency or network or any of these other pieces. so I think actually a lot of our iterations on latency have been really awesome because time to first transcription versus time to first correct transcription versus time to first byte. All of these are going to be really different. So I think Cale has done an awesome job kind of coming up with the right metrics so you have a suite to look at and make those trade-offs.

And now we're gonna have Madina who has been doing all of our speech-to-speech benchmarks come on and share a bit around how we're building out those parts of the

Hey, I'm Madina. I'm with the benchmarking team and first of all, thank you for having me and I really hope you guys enjoy your talk about latency because I love latency. Definitely. I personally think that that's the first thing that you go for when you benchmark. But I also want to just go with a conversation level where we can talk about the end goal because in the end for providers like us you isn't all about your customer calling your agent and getting an answer as natural as possible. And if we think of it that way, then for the user it might feel the

But for people that actually build the systems, we know that there's a difference between native speech models compared to the cascade. So if there is a difference, then we also know that there's a difference in how we dial and tune all those things to improve the agent performance for your user to have the best experience. And that is why I think we think about speech-to-speech slightly different than how we think about speech-to-text and text-to-speech. And on our page you can see that we benchmark latency for speech-to-speech as well which obviously everyone should have but we also also pay a lot of importance to the instruction adherence because yes you benchmarked latency but it's also important that your agent does not drift in the conversation. And to be really realistic 90% of the times your customer is probably not calling from a very clean studio environment like we have right now.

And you're gonna have a lot of background noise, different accents and that's what we're rolling up right now with speech-to-speech. This is a very MVP version of it and in the coming weeks I want you to guys stay tuned and look at that. Yeah. And one of the other things that we're trying with speech-to-speech is naturalness which is really hard to benchmark in natural way. So what we came up is we had so much debate around building arena and everything methodology was super tricky. but I think in the end what we concluded was that why don't you tell us what you find natural. So we built this arena where you can interact with it real time and actually see what models you find natural for your use case.

That's awesome. Yeah, I think the big questions we get all the time is why not use speech-to-speech models?

And I think the biggest question is there's so much more information that you get from understand when you listen to something versus text. That's one of the reasons why you just hop on the phone instead of it can be hard to express yourself over Slack versus if you send someone a voice note versus if you call versus if you video and it's because you have a lot more signal. but at the same time, the instruction following of the models is a lot harder because now the models have a lot more to do, they're more distracted with more parameters. and so it can degrade the intelligence of the models. so certainly that trade-off is something that we're really closely monitoring and constantly looking at when does it make sense to use speech-to-speech versus a cascaded system. Awesome. is this amazing? Thank you so much, Madina.

Yeah so I think really again benchmarking your data and really deciding these models part of it is based on the benchmarks and that's used to understand which models should I even start to investigate but beyond that then you need to benchmark on your data so that looks like actually taking a short list from the leaderboards and then taking your test cases and replaying against all of those Same model. So that's something you can do on a platform. You could do it with your own scripts. but again making those benchmarks contextual to your data is really going to give you higher signal. And then finally the highest signal is going to be scoring the full end task evaluation with those same metrics. So putting the model into production obviously is the highest effort but then you can actually benchmark are we doing better across the board. Amazing.

So now these are some questions actually from the registrations today but feel free to leave questions in the chat on which benchmarking questions model questions how to actually create data sets and evolve your agents as an organization so benchmarks are too generic how do I benchmark on my own data this is something I wish more people did because I think it's one of the best ways once you do select from public benchmarks to be able to understand which ones are going to perform best for you. So obviously I think there's a lot of controversy around benchmarks where benchmarks are either gamed or people don't have faith in benchmarks or they don't capture everything which I actually think is true but that's kind of advant the advantage of having public generalized benchmarks is that it's optimizing for the average case.

Definitely not gaming benchmarks and making them continuous. these are things that are really important to us. Also being incentivized to not have a single provider that we are pushing is also really important to us. But still benchmarking on your own data is a great way to understand how is this get a better sense of how is it actually going to perform in production. scraping those and then creating those data sets and then running just like a bench of the last two days of conversations or the last month of conversations is an excellent way to do this and you can actually do this all in our platform super easily as to speech cascade or hybrid and how do you evaluate without a transcript?

This is why I think the future is definitely speech-to-speech evaluations because not only even with a cascaded system, I think having text to text evals can be faster and certainly can capture a lot of the instruction following, but one of the hard things about text-to-text evals is that you have really clean turns. and so you're not capturing any of the turn-taking, which actually is one of the largest pieces of voice AI and voice AI engineering is how do you recover from interruptions? How do you recover from incomplete messy transcripts, how do you recover from transcript coming in late? All of these things like timing is super important. So, I highly recommend having speech-to-speech evals as part of your testing harness, even if it's not 100% of your tests.

Kind of having a pyramid of you have a lot of text evals, you have some voice evals and then you have some very realistic or load testing evals and then finally maybe you have a couple where the human calls and tests and then you can scale those with simulation. in terms of speech-to-speech cascade or hybrid with the way I think about this is that speech-to-speech is really good at taking in more signal and then creating a more natural conversation flow. and so there are certain applications that do much better that benefit a lot more from this than others. For example, companion applications or friends, therapy, being able to have I think outbound sales is a great one as well where if you don't capture someone's attention right away, you risk that they're hanging up.

For applications where there's higher compliance or instruction following is really important. or if the user is really motivated to get an answer like so customer service or you're filing a claim you're like booking a shipment these types of things those are probably you actually want a cascaded model because you have a lot more controllability over the models and I would say that's still true today. I'm really excited about the GPT live launch that is coming up. They announced it a while ago, but it's not yet public. so we'll see how that new model fares, but as of this moment, I think that still Cascade is if you want that controllability, the way to go.

And actually one other thing on the hybrid is I'm really excited to hear how people are I think speech-to-speech is probably going to look less like a drop in replacement and more like a hybrid model where you have speech-to-speech and some background models running or other kind of infrastructure around the speechtospech model. So I'm really curious to hear if people have been experimenting with this. would love to hear what you guys are doing. do we have a question in the chat? We can kind of switch between the two. All of our benchmarks are from the US, but this is on our road map to add European servers so that this is actually something Kell is working So stay tuned for our European benchmarks. Obviously, where you're calling from is really important for your latency numbers.

How do we keep ground truth current while we iterate? this is actually something that I'm really passionate about because I think creating refreshable test sets is super important. So something you can do with Coval is you can actually take production data and then feed that back into your system. for all of the calls that are coming in then you can create dynamic test sets that sampling from those calls. Maybe you say 1% of all the failed calls in the last week depending on your traffic what makes sense or maybe a 100 calls randomly sampled from the last month. This is something actually we did in self-driving and I actually built out all of our systems around how do you sample data sets to keep them unbiased and also actually in this case drift is a good thing where you're actually changing your data sets with the data that you're seeing in production.

So definitely I think whoever asked this question, your head is in the right place, which is creating data sets that evolve with your data is super important. obviously then keeping some data sets that are more very specific examples to make sure that you don't accidentally lose track of really important things. Yeah. So, we actually don't turn detection models. I feel like I'm just so excited about all the things that we want to add to benchmarks. So, we could go on for a long time on our road map, but turn VAD and turn detection is something else that we're going to be adding benchmarks for. we just launched our voice arena and speech-to-speech and that's next up.

But yeah, I think turn detection again is one of these things that feels really simple but actually is really hard and it significantly impacts your rate of resolution. I think someone just raised their hand. You're welcome to unmute and talk. never mind. How do you reproduce the live calls to use as a ground truth? awesome. So, the way that we do it is that we actually take calls from production and then re-simulate them. So, we use a combination of, audio reimulation plus the transcript.

But that's a great way again this is something we did a lot in self-driving was taking real logs from the road and then reimulating them in production with the new driver behavior and then comparing to how it worked with manual driving. So we do the same thing with call logs. You can take a transcript that you see in production and then re-simulate that and were we significantly faster or slower than humans. So, that's a great indication of if your agent solved it in one turn, and then the human solved it in 10 turns, that's probably a good indication that your agent did something weird. but if your agent took, 10 times as many turns, it probably meant your agent's looping or doing something strange. And then similarly, we also did it take the right steps to get there? can we distill? So what we do is we distill the steps that the human took and then we also distill the steps that your agent took and compare them.

So that's a great way if you already have traffic, those are like I love when customers already have traffic that they can share with us because then we can map that to test sets. when you benchmark a voice agent, what do you use as a human reference point? Do you ever measure how a human agent would have handled the same call? And if so, how do you capture that comparison? So I mentioned a little bit of how we do that with calls that we already have. The other thing that we do is when you launch a simulation, you can just put it against your own phone number. So that means you can say call me 10 times with all these different test cases and I'm going to call and test the agent 10 times.

And so then you can call and just kind of experiment of changing turns like saying interrupting the agent doing all the things that you would normally test but now those are in Coval and you can re-simulate all of those test cases with our agents. And so it makes it a lot easier to scale what you're able to manually test because now you can just take those 10 test cases you ran just now and then re-simulate them 10 50 times. awesome. And then as the agent answers differently, it's really hard to keep the same scenario as it drifts. Yeah, this is why simulation is super important. obviously I think you still have so that's something that's like the term that we use in self-driving around scenario decay is when you have a scenario that is supposed to measure one thing and then ceases to measure that thing.

So in self-driving that might be that you're trying to pass a car. you're trying to test that you can pass a car but now the driver accidentally ped the autonomous driving sped up so now it no longer has to pass that car at all. So now that scenario is no longer testing the thing that you were trying to test. The equivalent in voice AI is you're trying to test capturing the identity but your phone number no longer works and so you don't even get to the identity step because of either the scenario changed or now you don't have the right answers to the question.

So this is definitely a hard problem of some of this is just eval maintenance and this is where our agents come into play is the not only helping you to detect failures but also helping you to maintain evals. So being able to constantly update evals based on what you're seeing in production and then on the other hand simulation in general is just going to be a lot more resilient to changes in your agent behavior if you have auto updating test sets. again, I love auto updating test sets. So, my one true passion and other than voice AI in general, that's my one true passion. room. And

So being able to update test sets based on what you're seeing in production is a great way to maintain your evals. And finally hopefully if you can describe the scenario in sufficient abstract terms then our agents can actually continue to embody that scenario. So as long as we have the right information if we're trying to get a refund even if your agent behavior changes contrary to script behavior with our agents we can actually find new ways to try and get a refund or ways. So harnessing the powers of non-determinism can be helpful there. how should a r replicate your results? so we have So you can always run our open source benchmarks. we do have our own data sets that we use behind the scenes in addition to the data sets that we have publicly available.

So it's kind of a tricky balance here because on one hand we want to make it really clear how we're benchmarking everything but we don't want people to game the system entirely. So we do a mix there where we actually make everything open source on exactly how we're measuring everything where we're measuring it from is all in our documentation. and then part of our data sets are public. And so that's like the way that we tried to make it so that people are really aware of how we got to these results. And then also making it so that it's not gameable and we're able to still keep some data sets behind the scenes that allow us to understand true performance in production. And we have an API for all of our benchmarks as well, which is really cool because then you can put your agents on it.

So you could have your agents like ping our benchmarks every day to get some updates on what models are doing the best in production. You can have a downtime tracker. So whenever a model that you're using, you can watch our benchmarks to see if the model goes down. you can like So any way that you guys are using our APIs and our MCPs in your agents, we would love to hear how that's useful with benchmarks. What should we track in production? So I think that this question is kind of getting at what metrics should we be evaluating in production.

Definitely a lot of the one same I think this is the beauty of having simulation and observability in the same platform is that a lot of the things that you're measuring in observ in simulation you also want to measure in observability which can be helpful for understanding when are the results different and how can we make our eval representative with higher coverage and then on top of that being able to measure latency all the standard things that We're measuring for our benchmarks. You definitely want to be measuring in production as well.

So, I think those were the questions that were submitted. but if anyone has any other questions, we can stick around. I know we're about 3 minutes over, but yeah, happy I have extra time after this, so I'm happy to answer any questions or any thoughts. How do you create different user behavior scenarios? what we do with our user behavior scenarios is that if you want to share our app, we can share kind of how that works in Coval. so the way it works is so if we start with personas, personas are going to be how you create different simulated environments.

So this is going to be your persona like you're calling you're an elderly patient this is how you're talking and then we can do different background noises etc. And so again this is something that we're doing in our benchmarks as well we're creating all these simulated environments and then from there you can also configure how interruptive they are or you can create your own background noises etc. And then the this is your simulated environment and then the test set is going to be how you actually configure each one of these test cases. So this might be a happy path of all the different prescription refills that your pharmacy needs to do. so you're going to call and get a routine refill. You're going to ask about an upcoming one. You're going to ask about an old one. etc.

And so you can cover all of the main trajectories that your agent might take. these are all scenarios but if you click on scenario you can see sorry under type you can then see that we have lots of other types. So you can have a transcript or a script. So everything from you can actually give us audio and we'll replay it or we'll re-simulate audio or a transcript all the way to this very more totally synthetic scenario. And then you can also add in different checks or different variables to vary that conversation. Cool. let's see.

I think what was the last question? How do you prevent eval data sets from drifting as you iterate on the voice agent? the person who asked that if they could clarify what they mean by drift because I think with eval drift can both be a good thing and a bad thing. Non-determinism can be both a good thing and a bad thing where sometimes you want to re-simulate the exact same thing over and over and something sometimes you actually want to have that variance because if you run the same scenario 10 times, you want to see if I run I want to get a refund 10 times or hundred times, that's actually great because now you can see what is the probability of success. Whereas I think sometimes I want to evaluate I want to evaluate this exact thing when it says this into my agent.

And so that's a script versus a totally synthetic scenario can be helpful. and then in terms of drift of the data set entirely, that's again where I think refreshable test sets are awesome and setting up scenario these loops where not only is your agent becoming better by feeding that creating self-improving agents by taking the evals from production and feeding it back into your agents, but also creating self-improving evals.

So every new persona that you see creating a new simulated environment for that creating test sets based on things that you're seeing in production. And this is a lot of what we do with our enterprise customers is setting up the evaluation strategy across the enterprise. Are you seeing teams standardize around one model or voice provider or moving towards multi-provider stacks where they're constantly evaluating which model performs for a given task? And how do you see that Coval fitting into that world? I would love this because I think that this is really good for the industry. I think that choosing teams that you really trust and can work with definitely is a really big part of this. We're seeing that enterprises work with teams where they're constantly evolving with their models. But I think certainly being multi-provider can be helpful especially when you have lots of different use cases that might require different types of models.

So what you need for an outbound sales versus handling your customer support might be two different use cases. the hard part about enterprise is that obviously you have to go through procurement and so procuring these models can take a long time for large enterprises and I think this is a question for how can prises set up procurement cycles that allow them to be really fast and nimble and adapt to new AI trends while also not thrashing. I've actually been talking to a lot of enterprise leaders this week about this where you have the problem of on one hand procurement can sometimes actually be good because it keeps the grounding force from adopting every new AI tool that comes around the corner but on the other hand it can lead you to be really behind very quickly because as soon as a new model comes out but I wouldn't say procurement is the only blocking thing it's also how easy is it to add in a new model and this is where I think teams eval are

Only important for making sure your agent works before you put it in production, but it's also really important for making sure that you can evolve your agent in the future. it might be fine that you can brute force your agent now and just call it a bunch of times. But then two years down the line if you want to change anything about that agent it's going to be really really hard to do near impossible and so you'll have rock to your agent and so that's why I think eval are really critical piece of being if there's having lots of different models provided I think is going to make the ecosystem better overall and so that's where I see cobalt fitting into it is that we are actually empowering you to be able to evaluate

Those agents and evolve your stack. there's actually I think having really good routing too when you set up your agent so that you can have a couple maybe you pick three models that you have and you're using those for different use cases. and this is also where choosing a model family is really good too that you trust. So to reproduce the previous calls, do you have something more like unit testing a specific turn of the dialogue? Because otherwise it's really hard to reproduce that whole call. it depends. So I would say this is actually similar to how we did it in self-driving is you have a kind of a suite of different types of simulation tools that you have at your disposal.

So for one, it's totally synthetic tests where they're really specific and highly curated all the way to large scale tests where you just take a bunch of calls and then re-simulate them in bulk. and that's kind of the spectrum and obviously reimulating in bulk is going to be a lot less maintenance whereas these highly curated scenarios are a lot harder to maintain but going to be more accurate. And so there I think it's like what are you trying to test? One of the things is that you can actually have simulation set up before the simulation launches and then once it does launch simulation setup before the simulation. So you might say I want to create this persona with an order from two weeks ago. and then you can recreate those traces based on things that you saw in production. setting up a really good test harness is more of like a traditional software engineering problem where it's like how do you set up good mocks?

We also have simulated tool calls where you can hook it up to our endpoint and then it will just simulate back and forth bas given a certain JSON it will return that JSON. but then you're obviously not testing things end to end. so really it's like I would say in a lot of ways similar to traditional software engineering where you have some options where there's trade-offs of how much work does it take to set up versus how much signal do you get out of it. So during evals, have you guys ever seen one of the agents do anything crazy? yeah, totally. I think that we have a channel internally of every time we see one of our test agents do something or stuff we come across

Even when we're just messing around. I do think one of the cool things that we've found recently is there was a squatter on the line of one of our customers where that was actually on a Reply All podcast that I saw me that I listened to many years ago. Reply All is an awesome show that was from 2014. It's one of the first podcasts I ever listened to. It got me into podcasts and it was called a show about the internet and they found this example where there were squatters on phone lines and then people were calling in and it was just a totally random it was called America's hottest talk line and it was completely you'd be calling about insurance and then it would call in so then we actually found something like that happening in production where it transferred the phone line to what they

Thought was their internal team and it transferred to that. So that was really cool to see in production. we've also seen definitely emojis being said out loud or numbers being said out loud which is always I think funny because agents always punctuation poorly which is fun. and then vocal hallucinations never cease to be funny. the agent will, maybe become breathless or start screaming. And we found those and those are Probably less funny for if you're on the other side of the agent, though. How can you prevent voice agents from leaking system prompts or wait, can you go back to the question?

How can you prevent voice agents from leaking system prompts or non-con conversational output tool calls? Are there any deterministic ways to prevent this? Yeah, I think this is certainly a challenging topic. So, we do a lot of adversarial testing. we actually have some docs on this. We have cookbooks and we've launched a sub agent. So, we have a Coval sub-agent that you can set up in Claude Code so that it will call our sub-agent. And you might be like, what is the difference between a sub agent and just an API? But our sub agent actually has all of this information of things that we've seen across hundreds of different voice agents and how we've seen people successfully deploy voice agents in the enterprise. And so one of the things that we have is it's both in our Coval agent skills as well as our Coval agent.

So some things also are also in our skills but you can ask Sophia to help you create adversarial tests and that takes from what are common things that we've seen in terms of adversarial tests of people trying to break your system prompt etc. the other thing is simple heuristics. this is where controllability of the model is really nice. Regex is, vintage Python. can be useful and you don't always need an LLM for it. but you can even just have regex checks before you return the message to the user. that's a great way making sure that there's no JSON tool calls etc.

Do you evaluate different voices across the same models and what is the threshold and what is the threshold when there's a lot of them out there? yes actually Madina I would love for Madina to share a bit about this because Madina actually has she's actually implementing a lot of our voice comparisons here. Just see the question again. I think it was about So, this was pretty nice finding and I would like to hook it up to the area that we're talking about. We realized that we were benchmarking these different models. At first, we just randomly chose a voice and then it occurred to us that Yeah.

It occurred to us that male and female voices, they may have certain differences in latency. And it's actually pretty funny because you would never think about it, but then that will prompt you to think more about all the small nuances about how to choose a voice, which voice is better for which customer case. And right now, we're just trying to start doing that. And in our benchmarks, every time benchmark models, we like separate half of it we do in the male voice, half we do in the female voice and we get the metrics from there. And to be very fair, we actually also talk with providers a lot on what kind of metrics do they want to get answers from and everything.

And it's on the road map, but I think it's pretty exciting that we're also starting on the arena for the voice cloning because that's going to get us so much more deeper into this voices theme where we could actually test how different voices affect different metrics but regarding naturalness. So that's going to be super fun.

Amazing. Answer the question. amazing. So, stay tuned for some of the work that Madina is doing on research of how different models perform across the same voice. and we're also doing a lot more naturalness studies across the board. and finally, would love for you to participate in our voice arena. I think having more naturalness data is also helpful kind of from the community because it can represent a really wide swath of what does naturalness look like?

So cool. yeah, we'll drop the link in the chat for benchmarks.coval.ai and if you want to put that in the chat, Henry, you can put benchmarks.coval.ai and we come talk to us if we're happy to always talk about what we're seeing in production with different models. We're always happy to talk about different evaluation strategies and of course if you are looking to implement evals for your voice agent obviously you should use Coval which is the best. So thank you so much guys. I am super excited that everyone joined today. Thanks.

Thank you everyone.