How a Voice Agent Learns the Rhythm of Conversation — Shawn Wen

12,562 words · about 50 min to read

Voice AI requires real-time adaptation over pure reasoning

0:00 They want the agent to be super powerful like a human being, but they want the agent to be super controllable like a robot. Voice AI is very important because that's the native way that human talking to each other and it's the most natural interfaces. Computers have not been at the same level as human in terms of voice conversations yet, but we do actually foresee that this should be the future in the next couple of years.

0:25 Do you think that speech tech has become commoditized or is that just completely wrong? It becomes a lot more commoditized in a way, but also in a lot of different ways is still not. But modeling the real time conversation, it is still hard and it's not quite there yet. The bottleneck has already shifted from producing content to actually validating and auditing the content.

0:51 This show is made in partnership with PolyAI. People think that, you know, voice assistants are a solved problem, but it's actually much harder than people think. Why is that? When ChatGPT came along, we thought that, yeah, voice is probably going to be solved for the very, you know, very soon. And I think a lot of people have the same idea and then that's where you see a lot of wrapper companies come along, just wrap along, ChatGPT, you know, piping those ASR and TTS together and then try to solve the problem. The reality is that it's actually harder than we think.

1:22 And we kind of like, you know, settle on this idea that actually voice is... If you think about voice, the voice is actually one step to the real world, where if you think about all the current innovation on AI front is predominantly on the text and the digital world, where solving a PhD grade physics problem seems to be not a problem anymore because you have a really good documentations, you have a lot of online papers you can research on, while in the voice scenario where you actually have...

1:56 You have additional, like, axis, which is time, right? And then time is a very critical matter. I think we have not trained the model to be able to better understand that concept yet. And I think... And because of that one step into the physical world, we started to realize, okay, well, there's actually a lot more complexity when you add that, you know, time element into it. And also, like, with the humans is about...

2:22 The intelligence here is not really about reasoning, it's really about adaptation. When we spoke last time, you said that, you know, doing doing voice technology, it's a little bit like robotics. Right? So it's incredibly easy for humans. You know, there's the famous Wozniak coffee cup challenge, which is I want a robot to make me a cup of coffee. That's right. Yeah. Just tell me a little bit more about that. I mean, why is it so difficult? Again, get get get back to the physical world situation.

2:48 Right? Because in the physical world, for humans, we're collecting data in the physical world very naturally, and we have a lot of data because we have visual data. We have, like, you know, the body movements, your hands can reach out to things in the world. However, computers don't have that yet. Computer, all they have, you know, all the all the tools, it's all, you know, about how to use the computers, right? So I think now we have trained the model to really good at understanding text and understand how to use tools in that virtual world.

3:21 But in the physical world, the data is still very, very little. And that applies to robotics, that's why a lot of the robotic companies right now, as you say, is trying to grab the cup of coffee from one place to the other. I remember one of my really dear friend and also a really good researcher from Cambridge, he he was originally at Poly and he now left to to start his own B2BP business. And he wanted to do a robotic company.

3:48 I I was like, what is the first application you are trying to do? He was like, I want the bot to be able to pour beer for the... For me. That's the test number one. I was like, oh, wow. Okay. Well, because it's just really, really hard. And then you look at self driving car, you know, Elon has been part of... Promising the world, you know, this is going to be, you know, fully commoditized. I think we started to see the sign of the... Of it now right now. I think voice is the very similar problem, just that we have so much...

4:19 So many conversations between, you know, between each other, but machines just never be able to observe them. And what do enterprises actually want from a voice agent? Enterprises want a branded voice.

Enterprises demand both human capability and robot control

4:34 They wanted to be able to pick their identity. They wanted to... You know, the agent to exactly say their branded name exactly the way they wanted to. So that's number one. Number 2, they want the agent to be super powerful like a human being, but they want the agent to be super controllable like a robot. So it's actually a very contradictory addiction. And I think that's the difficulty of deploying AI agents into enterprises because the expectation of people towards AI is they just need to be as good as human and actually as good as my best human in the team.

5:12 Right? And I think that's the... That's always the adoption curve is, like, you know, harder to justify. Until... Unless you can find a particular area, you're saying that, hey, here is more about efficiency play, and then, know, people started to adopting it and started to feel, okay. Well, actually, agent is actually better than I thought. That that... That's what we are seeing from the enterprise adoption right now. And why did you guys decide to build your own custom model rather than using some of

Cascaded voice pipelines break in production environments

1 Highlight · Why cascaded voice models fail in contact centers

5:40 the existing ones out there? We have been doing research and building our own models for a very long time. We built our own speech recognition model previously. We built our own larger language models as well. Around one year ago, we decided that cascaded is going to be outdated in the future and the end to end model will be the future. And then we started to, you know, to do a lot of experiment and try out and we decided not to invest in our speech recognition models anymore because of that radical shift into the future.

6:11 And when we started to look into the market, the reality is that all these latest speech to speech models, they are really good for the demos because I can show you really natural conversation, you'll be like, wow, that's singularity already. But once we deploy into the contact centers, it breaks in so many different places. The interruptions sometimes is a bit too aggressive. So if you are on the phone with an old lady and the old lady is the first time talking to the bot and she was just constantly confused and, you know, stopped, and then the agent just keep, you know, speaking over her. That is like a level of frustration, you know?

6:52 So we see a lot of these scenario, we just say that, hey, we probably have to really train our own model for these use cases because I think the Frontier Labs are not optimizing for what we care about.

7:07 It's a real conversation turn taking for the business use cases. I think consumer use cases, I think you can build voice agents already. For for enterprises, there's additional governance there. There's additional, like, you know, enterprise requirements that we have to satisfy. And how adaptable could that be? You know, for example, if I'm talking to an old lady on the phone, I I would adapt to my speech, I would slow down a little bit, I would I would give them give her more thinking time.

7:35 Could the models be adaptive in that way? Yes. So that's why you have to collapse all these modules together into an end to end model or some sort of end to end, in a different fashion in a way. Because the traditional cascaded system, you have a speech recogniser, you have the larger language model, and you also you have the turn taking on top, and that is actually making the end of the sentence detection. These 3 models don't really work with each other very well, and you have the tweed, the parameters.

8:05 In a traditional way, you can have another detector to say, oh, this is an old lady, so I have to tweak the parameter here and there, but it's all very clunky. And then reality is that it doesn't really work like that and it doesn't really work very well as well. So therefore, it's almost necessary to collapse all these pipeline together. So you have a large language model directly perceive the audio, and the audio can be slower past speech, it could be background noises, it could be crosstalks, it could be a lot of these scenarios you will run into when you actually have a real conversation with a human over the phone.

8:47 Then you can collect those data so the model can adapt to that. So I think collapsing it is the first step. And then the initial model needs to be good enough so you can continue to iterate on that because you use it to collect more data as well. I suppose it all comes back to Rich Sutton's The Bitter Lesson, which is that, you know, in in principle, maybe we could write the code to do this. But there are just... There's an exponential number of failure modes, like combinations of bad things that might happen at the same time.

9:17 Like, you know, in this situation, we need to do this or here, there's some cross talk or here, there's some noise in the background. And just training a neural model end to end, you you can you can cover a lot of those failure modes. But I I suppose though it still needs to be a cascade in some sense because it's not possible to embed frontier level intelligence in these voice models. But you you could potentially draw a boundary around the voice intelligence model, and then it could call out to an agent when it needs

Audio-native models process streaming audio while outputting text

2 Highlight · Architecture of an audio-native text LLM

9:40 more more intelligence. But I wanted to ask, can you just sort of sketch out that the high level architecture for this voice model? Yeah. So the high level architecture is that this is basically audio native model. So the base model itself is a larger language model, so it actually output text. So we are not training models to do end to end speech yet. The reason is because enterprises want a level of control, so text is a lot easier to actually apply guardrail on top, while speech is a lot fiddly to deal with. And so we train the model to output text, but then at the same time, we allow the model to be able to collapse ASR into its input channel. So that means that the model itself natively understands audio. The audio you pipe into the model is that you are streaming audio chunks into the model. Then the way that we are structuring the model is that the model is

10:40 doing multiple things at once. So first of all, the model would be based on the streaming in audio, it would predicting a first token. The first token is basically the turn taking signal. The turn taking is basically saying that, hey, is the speech just getting started Or is it still... The user is still speaking? Or it's already end of the speech? Right? So that initial token is very cheap to generate.

11:08 It's just like the first token in the other output. As audio comes in, we keep generating that token until the model generates, okay, this is end of the speech, then we started to have the LLM doing the product processing. But at that point, all the audio has been already pre processed because as we encoding them and then predicting the 1st... The whole token, it's already in the embedding space. So you don't need to encode that sentence from scratch in that case.

11:36 So then the next the next job of the model is to directly predict an output. So the output is a text, as we say. So it could be a response or it could be a tool call, for example. And that is all streamed like the typical ChatGPT API. So then we are streaming that response back. And then at the end of that response, we then continue to predict a few things later. A few things that we will also generate citations.

12:08 So for example, if you have a knowledge base with a lot of different content, the model will then generate a citation and say, hey, actually for this response, I'm using this fact and that fact to actually make the prediction. So enterprise knows that, okay, it actually has a red reference there. Then finally, it also stream out the transcription.

12:33 So we're kind of reverting... Reversing the order order because the model's job is not to predict the transcription, but we still wanted to predict the transcription for, you know, auditing purposes, debugging purposes, and things like that. But it comes at a last because you don't have to wait for the transcript to actually start responding back to the user. Even that that's quite a nice interesting trade off to talk about because,

12:56 you know, many folks would have used like the GPT live model and that is a bidirectional waveform to waveform model. And, you know, it's it's really interesting. It it just makes all of these like these weird pauses and sounds. It sort of breaks the uncanny valley. But actually, that might be great for me at home. But in the enterprise, you need to have constraints because you need to say, no, you're not allowed talk about this, you're not allowed to talk about that. And conceivably, I suppose you could run separate ASR on the output and then you could do the retrieval augmented generation and linking to the knowledge base and stuff like that. But there are just all of these trade offs that, you know, actually an enterprise is not necessarily what you wanna do.

13:30 Retrieval itself... I mean, you can also do retrieval with these native... Audio native models as well because the audio... But the retrieval mechanism is slightly different. You cannot use a traditional RAG anymore because a traditional RAG is that you embed the user transcription and then you do a vector search in your vector DB. Right? But because now you don't have that transcription, what we need to do is that we will prompt the LM to generate a tool.

13:58 The tool is basically a search query. So let's say, hey, now I think that in order to answer this question, I need to generate a search tool and that search tool needs to come with that query. It's pretty much like local style search, which is like, you know, people call it agentic search, which is just another boss word, but it's basically just like a tool call, really. The important thing is that our innovation is not in the model itself, We are innovating at the IO contract.

14:29 So it means that, how do we actually prepare the data and how do we prepare the input and the mechanism of the input... The data feeding into the model, but also how does the model output that data? And then we also... How do we train the model based on these input and output pairs to actually behave exactly what we want? So to be honest, to underlying model, almost doesn't really matter all that much because all our research team is doing is that because we know in this market, open source model come up every month, even every week as well.

Contact center interactions provide a massive training data moat

15:06 So we kind of wanted to maintain our data, our moat, and then we have these training scripts that, you know, always getting ready whenever there's new model coming, we will train on top of it and to see which one has a better performance. And voice data is much harder to collect than than text data. How are you doing that? For us, we don't have to collect them because we... Our platform generates a lot of voice data.

15:32 So because we we have been deployed to enterprise contact center centers, we have banking clients, utility clients, logistic clients, restaurants hotels a lot of these retail outbound sale use cases. So we have already a lot of those data. And those data is exactly the kind of data we wanted to solve the problem for because we serve predominantly enterprise at the moment.

16:00 So therefore, we kind of use leveraging a lot of the listed data. Obviously, getting agreements with enterprise clients to train on those models is nontrivial because all these data privacy, GDPR, API, data things a lot. But kudos to our legal team, we actually get quite a lot of our customers who agree to that because they know that if they agree, the model can continue to actually help to improve for them.

16:26 And the way that we are training the model is that we will redact all the PII so that none of the customer data is actually exportable post. And we will produce a lot of the fake PII data as well. So that model can still learn how to identify names and addresses and things a lot. But really, the data is actually just to let the model learn the mechanics of the conversation rather than the actual user data.

16:55 Very cool. And I wonder, does the model currently take in pure waveform data or does it take in featurized data? I just remember I was doing audio stuff for my PhD. And and back in those days, would have been unimaginable to put in waveform data. And and now now it's simply a case of the model's big enough and you train for long enough, you can. I think currently, the standard audio encoder is that you feed in a mel spectrogram. Right? So, like, you know, it's still like this, you know, image. You can... You remember, like, you know, there's, different spectrums and then you have different frequencies and things like that. It is still that. But then to the model, you you are feeding that segment by segment. Right? And then to the model, it feels more like an image recognition task.

17:37 I remember... You're from the speech perspective background, so I remember there was like a MIT PH... Professor. I think his name is Victor Zue or something. He was a... He was... Back at that time, like in 1990s or something, he was like, you will be able to identify what the human speech is by just looking at the spectrogram. And he was like, this is, you know, exactly that particular, you know, syllable, whatever.

18:05 People were like, you know, what do you mean? There's not... Nothing you can identify there. So in fact, machine can learn those patterns really well from it. I I love looking at spectrograms because there there are so many things that you you can just immediately see speech or if it's like a violin, for example, that you just see these horizontal lines and you can see that there's a harmonic series. And if it's percussion, it looks like a sort of vertical bar in the spectrogram.

18:32 But yeah, a while as a human, you know, that that you've been looking at these for a while, you can just read like a book. Fascinating. And and noise, course, just looks like, you know, salt and pepper. That's right. But on the subject of noise, this is quite interesting. So how how do you guys deal with that? Is is this a kind of thing where you you deliberately sample noisy data or do you kind of like augment your data and add noise to it? Yeah. So it's a very deliberate decision to add a lot of different noise into that training dataset. I mean, to be honest, our training data set by itself contains a lot of different noise already because it's a real conversation data.

19:10 But we typically would apply another generative model on top of it. So say that, you know, now I wanted the model to pronounce my Mandarin name, Tsung-Hsien Wen. And then by doing that, I also asked these TTS models to actually add these additional audio data we are seeing in the real production into it. So the generation or the outcome of that generated dataset could be a robot pronouncing my name, while there's a background, you know, baby crying in the background, for example.

19:43 So this can be kind of like simulate in a way so that you can have a lot of different variety of that because, you know, thanks to the modern generative AI models that you can do a lot of, like, very realistic data on that. So the answer to that is yes, we do a lot of this and it's a very deliberate approach because we want a model to be robust against all these different environments. But even in the training the data ourselves, it's already contained a lot of that.

20:11 So the funny thing is that because we, in the previous technology, we kind of doing a lot of denoising or doing a lot of these pipeline sanitizations beforehand. All those additional components to this new Dialog Reason model, it's actually making it worse just because, you know, like it's too sanitized environment. It has not seen before. I remember for many, many years that there was this phenomenon called the cocktail party problem, which is that, you you know, like when when we were at a cocktail party and there's many people talking at the same time, we can focus our attention on one person and our brain can just filter out the the other people.

20:50 And ASR models could not do this. But but now increasingly they can do it because maybe internally what they're doing is is they're learning, well, this is speaker 1 and this is speaker 2 and they kind of focus in on on speaker 1. So so yeah, crosstalk has always been a problem. But you know, what... How how are you doing with with those failure modes? Crosstalk in our model currently, it doesn't actually have a specialized mechanism for that, but the architecture is suited to actually tackle that problem just because... I mean, the speaker diarization technology was actually previously predominantly applied to speech recognition and is always assuming in a cascaded pipeline solution. Then you you you tagging this speaker 1, speaker 2, speaker 3. But it's very...

21:29 It's not a very natural way because as a human, you... If you are in a party, you're listening to so many people listening, you will not actively label this is speaker 1, this is speaker 2, this is speaker 3. You'll be like, oh, this is another person. That's another person I shouldn't pay attention to because there are so many of them. Right? And then some of them would stop speaking and then you will lose them for a while. I think the traditional speaker like dialogueization is trying to solve a very different problem.

21:57 But with a dialogue system, what you really need to do is that the agent focus on the right kind of subject that it is talking to. So in that case, we just... Because now you combine the LM with the speech recognition together, so the LM can natively perceive there's a background, like, you know, cross talk somehow. I mean, to be honest, this LLM is also promptable.

22:23 Right? So you can prompt it to say, hey, don't respond to background cross talk apart from the major speaker you are speaking to. You can also say whenever there's a cross talk in the background, take an active participation in that conversation as well. Technically, you can do all this. The major difference is that how do you train the model? So how do you collect the data?

22:50 What kind of data you collect? And what is the instruction you instruct the model to do? So like, you know, for example, I think that you you you watched the GPT, you know, live demos where you have a lot of these, like, you know, old ladies coming along and then there's like, you know, model will be able to identify a new speaker. I'm pretty sure that in their prompts, they are saying that, hey, now you are in this party like conversations, and then you have to gradually identify speakers as they come along.

23:20 So the model... Once you fuse these models together, it has the ability to do that. And then the major difference is that how do you train it, what do you train it to do. The turn taking thing in particular, as I spoke about earlier, it's really, really difficult. And I suppose we should imagine in in an ideal world in 10 years time when this technology has reached the utmost maturity, maybe we can have microphones in every single meeting room. And when there's anything private discussed, it it will know not to pay attention to that.

23:49 And there might be multiple people involved in the conversation. And you can address the agent and the agent can get involved and the agent knows who's talking. But so at the moment, we're kind of solving it on in the one on one case, And we're solving this turn taking problem. But is is that a reasonable estimation of of what the future might look like? I think the technology would be complete. I think I think you were... Prediction is right. I think the technology at that point, 10 years from now is a lot of time, you know, at the speed of all these technology is improving right now, I think it's completely reasonable for the future.

24:24 I think though, the only thing is that the voice modality... I mean, people say that it's the most natural modality to interact with the computer. The reality is that we kind of have several iteration of these kind of personal assistants in the past to start with like, you know, Siri, and then you have Google Assistant, Alexa, you have the in car voice assistant whatever. Maybe the technology was not really mature yet, but then also there's just a little bit of this like voice is a lot more personal and private to you.

24:55 And so then I think I don't know whether the consumer behavior will be there already yet because that means that you have to surrender a lot of your control to

Natural turn-taking and latency budgeting eliminate awkward pauses

25:06 the agent, which, you know, a lot of people will say that that's a great thing, you know, because, you know, the agent can do a lot for you. But then I think there will be some consumer adoption curve that we still have have to climb, you know, even though the technology is here. And can you tell me about how you guys did latency engineering? Now, you know, we're gonna talk about this uncanny valley and all the things that we need to do to make it kind of, you know, for people to suspend their disbelief.

25:31 Because they're talking to a voice agent and it's suddenly really good, but latency is probably one of the most important things. And when you go lower latency, there's always a bit of a trade off. You might be trading off some of the intelligence, and you guys have best in class latency. So how have you done that? Latency 1st of all, the major thing is you need...

3 Highlight · Collapsing ASR and LLM solves turn-taking latency

25:48 To be honest, all these model processings, if you are using the right size of model, the model processing itself latency is not really a problem. Historically, the cascaded system is struggling in waiting for... To decide when is the end of the speech. Because there's like this very coarse, you know, the parameter you can tune is like how long you want to wait until the end of the speech.

26:13 And then if you are tuning to be too short, it could be too aggressive and you are going to interrupt a lot of older people, for example. If you tune it to be too long, you will be waiting for too long and a lot of people will be impatient. So everyone has a different threshold in terms of like, expecting the... How fast the conversation should be. So I think that's the difficult one because you are trying to set a programmatic metrics for every single user that is speaking to your assistant.

26:44 Now with this kind of model, you have a lot more flexibility on that because the model itself is now predicting based on the audio frame. And the audio frame contains the how fast you speak as well, and it contains it contains a semantic information. So if I'm pausing in a mid sentence, the semantics has not been completed yet, the agent knows, okay, I have to continue to wait until the person finished the full sentence. So that is the major bit by collapsing the ASR and the LLM together, so then you have this more natural turn taking built in into the model. So the model is now making that prediction by itself.

27:27 So that's one way. A lot of the pre cached, processing, as we talked about earlier, is another thing to actually how to actually optimize it. The fact that... I mean, to be honest, the fact that we host our own models with the

27:41 rest of our platform is another benefit because you don't have the network travel latency usually impacting the most at a long tail use cases where the P95 is the major problem for these net... Network latencies. So a lot of this we optimizing in a way so that the agent can

4 Highlight · Training voice agents in latency-budgeted reasoning

28:02 just respond very fully and back to you. However, now we are all my team is adding more reasoning to the model as well. So, you know, the good thing about it is that once you save a lot of time, you can use that time to do more things. So now we are... Our team is looking to adding auto reasoning to the model. So the model can decide, hey, based on this particular user, I stop my reasoning right now here, how likely I can give a good answer?

28:36 And so we train our models auto reasoning in a way that we always challenge you to say, hey, like, you know, do you actually need to think that much time? Can you, you know, cap it to, like, one second and then just respond back to the user? So the reasoning we are training for the model to do is very latency budgeted reason. It's almost like, you know, you are you are going to go go go go go go into, like, a competition.

29:04 Someone give you a question and you started counting down.

29:08 It's so interesting because sometimes when I interview people and I ask them a really difficult question, even though you're thinking, you still feel that you need to say something. So quite often, 1 of the difficult questions. Sometimes the 1st sentence of their response is just filler because they're obviously thinking and then they actually answer the question. And it's a similar thing here, right? So this auto adaptive reasoning is really interesting because like from a user experience point of view, how do we solve that problem?

Mitigating user panic with audio tricks

29:31 Do we kind of make the model say, that's interesting and and kind of indicate that they're thinking. That's right. But, you know, because it might actually be a significant amount of thinking, maybe they need to delegate to an agent and they need to do some stuff and then they come back to you. There are a lot of tricks you can do in that scenario. It's like, you know, in our real productions, you can add a lot of sentences, obviously. But if you are repeating it too many times, it becomes very repetitive, and the user will know that, you know, you are just, you know, bluffing them.

29:58 Right? We have another way is that you can put in the typing sound as well. So, you know, there's, like, computer typing in a background, try to mimic that actual like scenarios. The re... The real thing that we are trying to solve in that scenario is that, you know, because you always have the API latency you have to deal with anyway, and some of the enterprise backend is not the fattest backend by default. So you have to bake in a lot of those things to actually make sure user don't feel... Because the reality is that user has been talking to a lot of these automated systems, and if the AI go dark on them over the phone, even 3 second or 5 second, you will panic. You'll be like, is the AI actually working?

30:44 Right? So in order to have that trust, you kind of have to provide this constant feedback. The good thing is that... And the contact center has a lot of way to deal with this. Agent will put you on a on a mute... Music to wait while I'm think... You know, maybe in a in a in a old world, human agent actually, you know, going to different places, talking to their colleagues how to solve the problem. The AI agent will be put you you on a whole music and is doing the reading reasoning. So so I think there are a lot of tricks you can do there, but, you know, architecting well is very important because the...

31:23 I think the voice agent has the biggest issue in a contact center is that user that don't have the trust in them just because they got burned many times in the past. That's a really cool idea. So play elevator music while it's while it's thinking. But I suppose that this gets on neatly to the next thing,

Regional accents outperform generic robotic voices

31:39 which is, what is the uncanny valley for a voice agent? And is it because we talk a lot about anthropomorphization and the risks of that. Is that something that we actually want in the enterprise? Do we want it to sound indistinguishable from a human? Different businesses would tell you different answers, but I think the majority of the businesses will probably hold the same position as Apple and Google, for example. So if you think about Alexa is probably the consumer personal assistant that goes the furthest to in terms of personalization, right?

32:20 They give the Alexa identity, there's a very unique voice there. But that's as far as they wanted to go because if this personal assistant is representing a particular company brand, and it's very hard to know how successful it would be. So a lot of times, companies still try not to attach too much of a personality to it. And the goal is to just make the assistant solve the solve the problem first, basically. So, yeah, I think majority of the enterprise is still more on the conservative side. We do actually have some customers, you know, as... I mean, for example, there's this cut cut...

33:01 There's this casino from the South America. They like the local, very southern American voices. You know, they like the agent to say howdy to open the conversation, for example. To be honest, in in the end, they didn't launch that agent because there's... They will be like, oh, this sound... That feels a bit too much still. So they are like person... People would think about that once they feel more comfortable about the technology.

33:31 But I think we are still in the in the phase where people are just building that trust under the technology still. Yeah. I guess it's interesting how much we should or shouldn't chase the personality thing. Because the reason sometimes it sounds egregious is because that that might be normal in Texas or in a particular location, but to me, it's not normal. But do you think in principle there should be just a kind of generic voice? Or should the agent get to know you and when it kind of knows how you talk and what you like, should it adapt to you?

34:02 I can speak for what is happening right now, but I cannot really predict in the future what people would like. But currently, the majority of our customers don't like the generic voices. And also, the generic voices are in general perform a lot worse for the consumers as well, just because the consumers would identify it's a robot. And once they identify it's a robot immediately, they don't have a trust in it because we have these traditional IVR systems that has been deployed into...

34:38 Especially in the America, you know, a few words, you know, like if you have a payment issue, say payment. So if you have... You you wanted to, you know, transfer money, say transfer. And you say many times, you still didn't recognise you. And those voice are always very generic and also a little bit robotic just because the previous generation of technology cannot really make such a natural sounding voice. So the generic voice right now for consumers, they immediately connect it to that previous system they talked to.

35:12 So then they immediately hang up or they just try to bypass the agent as much as possible. We found the majority success is that you don't want it to be generic voices, you want it to have a little bit of local tech, local regional tech. So like, for example, some like in UK, for example, a lot of the contact center or a lot of consumers like the new ChatGPT has a voice for obvious reasons because a lot a lot of agents are from there.

35:43 So when they hear the Newcastle voice, know it's a human agent. I can actually say my problem to them. They can solve it for the... For for me. So I think that kind of voice choice is actually quite nuanced but quite important. So it depends on which brand it represents and then what people you expect the voice to be. But the best performing voices is usually the voice comes with a little bit of regional take.

36:09 The funny thing is that in the early stage of us, you know, shipping those voice agents, we found that all the English voice was actually not natural enough. The most natural voice is a French voice, and then you force the French voice to speak in English. So it take a little bit of French accent. It's a imperfectness of that voice and make it real.

Multi-axis evaluation for voice agents

36:32 Oh, that's fascinating. We should talk about benchmarks. So, you know, for years, there's there's already been benchmark blindness and many of the ASR companies, they they kind of embellish their benchmarks and they all use different internal tests and different public tests and and so on. But now in the regime of voice agents, it's it's even more difficult to test this. So how how are you guys doing that? Benchmark for the voice agent... I mean, benchmark for ASR, benchmark for LM, you...

36:57 We have a lot of us... The tabular standard. These are all, like, you know, more of a particular module, and you evaluate that. The voice is really difficult to evaluate because it's a voice agent has the latency considerations, you have to reply fast enough. There's also quality of the response. There's also a reasoning task, how difficult this agent can handle the task and how accurate you can identify personal names and information like that. So it has too many accesses. I think majority of the benchmarks that is publicly available out there doesn't really serve that purpose.

37:41 And also, another problem is that a lot of these voice agent benchmark on the market are synthetic to a degree because voice is a lot more private, releasing those voice better than dataset for evaluation and then make it public, it's nontrivial as well. So therefore, we have a lot of these benchmarks on the market, but not really sufficiently represents what a good voice agent should look like. And I think the industry as a whole is still figuring out how to best evaluate it. And the reality is that voice agent moving towards so much into the real world application, sometimes it's also hard to say there's a really good benchmark and solve all the problems.

5 Highlight · Building production benchmarks for voice agents

38:27 So we are... Our benchmark is built on the real data set we have in the conversations with our consumers. So we build it in a way, so it measure a lot of different important axes. We wanted to make sure the agent response time is factored into the benchmark, because a benchmark...

38:53 A voice agent benchmark without latency consideration is not a real one because user cannot wait on the phone for 60 seconds for you to come back with an answer. So that is a very important element. We evaluate the understanding, like, would you be able to actually understand what the user say and then transcribing it accurately? So the ASR transcription is still one of them.

39:19 And then the tool and the traditional LLM, you know, the evaluations, instruction following, the quality of the response. Are you using citations? Are you using tool calls? And things like that. So there's a lot of different evaluation we run on. So currently, our model is evaluate on that benchmark. It is the best models. Of course, it's internal benchmark right now.

39:46 But our plan is to, in the next month or 2, releasing that benchmark for the market and for the research community to be able to also evaluate on it as

39:59 well. Because we do believe that having a more realistic benchmark to evaluate on would be a good thing for the overall industry to continue to move forward. 1 thing that I'm thinking of is that this might be an example of subjectivity. So there are some objective things, right? So, you know, just quality and trust and tonality and stuff like that. But for example, it might be cultural specific, task specific.

40:26 So for example, you know, in this particular application, someone in Newcastle, they might like it, someone in France might not like it. So what would you do there? Would you sort of like, you know, statistically aggregate over lots of different situations? Or I mean, how would you approach that? Currently, the benchmark is actually very objectively look at the quality of the read response and more at a factual level, right? The actual end user experience level, I think that's an even harder problem to measure. I think for measuring that, we can argue that we need a benchmark dataset to evaluate that.

41:04 But the reality is that if you're building an agent or assistant towards that level, it's almost at a production grade already. The production... The production agents, you have the end user to tell you already. So the way that we are also measuring this in the real production is, hey, does the user want to engage with agent? And does the user want to be able to...

41:35 Does the user actually finish the task in the end? So there's a lot of other production related metric you can measure there. But that one is a lot harder to measure because it's not just about the model performance. At that level, you have to couple the model with the harnessing together. So it's that entire agent evaluation.

Enterprises own harnesses while outsourcing models

41:58 And that is even harder task. Can you talk me through the complexity of harness engineering? I mean, the reason this is interesting is so many enterprises, they're adopting AI. And the main form of adaptation is harness engineering. And it's an incredibly good thing and it's an incredibly bad thing because everyone is building their own harnesses and they're adapting their skill surfaces and prompts. And essentially, everyone is reinventing the wheel and and doing different things. So, you know, so there is an argument to have some kind of, like, platform and standardized ways of of doing this.

42:28 I mean, how how do you deal with that problem? This is the market demands. Right? Because I think enterprises enterprises acknowledge that artificial intelligence is going to be a very important technology for them to own, not just buy. They wanted to own them. They obviously wanted to own the entire brand, the entire, you know, like, ideally models as well, but they know they cannot own the models because they don't know how to tune the model and it costs a lot, and they have... They need to have a lot of GPUs, they don't have the budget for

6 Highlight · Why enterprises own harnessing instead of models

43:00 it. So they gave up on the models. They don't want to own own the models anymore. Right? They was like, hey, we can switch it to different model APIs, that will be okay. But they wanted... They still wanted to own the intelligence themselves. The only way for them to own is to own the harnessing. So that's why I think a lot of enterprises opt in to build their own harnessing because everyone is considered that this is such an important technology they need to in house and the harnessing is the only thing they can do to in house currently.

43:33 So I think that's kind of like the market sentiment right now. I do actually think not harnessing... Not every harnessing you should own yourself because if you are building... It depends on how critical the task is. And to be honest, I think voice agents and the voice facing the customer... I mean, we we talk about, like, voice is actually a harder product problem than just general re re reasoning because the adaptation itself is very important and is very, very critical. I don't think enterprise value that part as much.

44:07 They value the reasoning part a lot more. So we'd like to talk to our customer is that, hey, you don't actually need to own the voice agents or the voice agent harnessing itself. You can actually have this voice agent as the front front front door to your customers, and you can own your brand in the background. So then you have the voice agent that can respond very quickly, but then it can delegate a task to your mastermind behind the scene.

44:37 So I think this is probably what... Where we are going to see in the next couple of years is that enterprise will realize that they don't have to own everything, but they have to figure out like, hey, they have to draw the bottom the boundary. This is how I feel comfortable giving away.

44:53 Some people will probably always stay at, oh, I I really need to own the harnessing because I wanted to do a lot of customization ourselves myself, and then I'm comfortable to use the model. And that... That's okay as well, but I don't think there will be the majority. I love thinking of software engineering as a specification acquisition problem. Apparently, a whole theory around this, which is software engineering as theory building. But many folks at home would have experienced this, that when do vibe coding with agents, there's a there's a kind of Rubicon moment where your software is sufficiently well specified that an an autonomous agent, when it sees problems, can just fix things.

45:32 So it's almost like if you can get it to the point where it's well enough specified, then it just kind of converges from there. And that... That's the magical moment. Exactly. The beauty of of working for these enterprise conversations or pre business conversations, there's kind of a a good look like. Right? It's actually relatively well specifies because in enterprise conversation, the major thing is that does the customer's problem got solved in the end? And then does the customer feel good in the end?

46:04 Is it a good conversation? And how fast you can solve the customer's problem? So the good thing is that, you know, the... Despite that in our human to human natural conversation, there's no gold standard because a chitchat... If it's a good chitchat, it will actually change the relationship of the 2. But in the business conversation, there's no good chitchat. You know? It's just like, did you solve my problem or or not?

46:30 So it's a much more well defined problem. So that's why I think we see a lot of success in applying coding agents on top to be able to actually automatically optimize it. Very cool. And I guess one of the issues with agents, I mean, this has been a dream of mine for ages. Know, what if we could almost build an artificial organization with some agents and some humans, and then we have to solve the coordination problem between them? So we have some agents that are specialized in front office and some that specialized in back office and coding, and they're all just sending messages to each other, fixing things. That's very exciting, but it's quite problematic as

Enterprise demand for auditable AI workflows

47:06 well because there are all sorts of failure modes. How would you approach that? It is. And I think the enterprises don't feel comfortable about it yet. Despite that, we have a lot of white coder coders and you have a lot a lot of AI native companies is, like, rushing to build all these video feature. And to be honest, a lot of these products are really, really cool. But I think enterprises are feeling very threatened by that. Yesterday, we actually have one of our largest utility customers talking to my team. My team is really freaking out because they say, oh, in the future, all the PRs got merged into the repository.

47:39 We need to have a review, and the review cycle is 2 weeks. And then we have to make sure that everything got merged into production is auditable and then reliable. And in the past, it wasn't really the case because we were just get... Extending out the agent and making the agent really, really good for their use cases. But now they are applying that control on top. And I think they have been holding for a very long time, and now they feel it's a time to do it because the agent is really good right, right now. But the thing is that just because enterprises wanted to have their visibility, their guardrails, their their...

48:15 You know, that everything is auditable, so you kind of have to build around that as well. So a coding agent just directly spit out PRs and auto merge would not fly with with them. That's why in our platform, we also build a lot of visualization components with all... Of the agent. It's like, hey. This is what the flow look like. So if the agent enter here, this is what the processes look like.

48:42 So they still have a general sense of, oh, agent is actually executing these processes one by one. There's a bunch of tools and these tools are exactly what the agents can do. So then anything that is outside of that toolset, the agent doesn't have the ability to do that. There's a citation module of the model so that every response comes with a citation so they can the trail that.

49:08 And I don't know how long we will have to have this because we kind of live in a world where you have a lot of vibe coding and coding agent can do things very fast, but we are still dragging this traditional SaaS enterprise, you know, like no code no code flow with with us. I think now the good thing is that this no code flows is no longer the primary interface for people to build.

49:33 It's just for you to audit the the actual processes of the agent. So I think we are kind of shifting towards that way. But that UI component and visualization is still very important for the enterprises. Adaptation is intelligent. So these models, they they are being adaptive. But at the moment, we've been talking about kind of skill surface adaptation and code adaptation. The great thing about weight adaptation is that you can, in principle, share this model in your organization.

50:01 You know? Because if you're adapting code and skill surface and you've got agents everywhere, it's a little bit brittle in the sense that, you know, I can't really take my code here or my skill here and share it with an agent somewhere else in your organization. So like how do you think about this adaptation problem in general? There's always this debate is like where do you want it to, you know, change the behavior and then where does the customization should go to?

50:26 I mean, to a certain degree, we agree with that principle because we do actually train our own models as well because we agree that there's just like inherent behavior. Like, of course, you can try to solve from the agent harnessing side, but even if you try to solve it there, you're probably going to put a lot of red red tapes around it. It's probably not going to be good enough for your use cases still. And if you have the ability to alter the weight, then you have a lot more control over what you're ability building. So I think it's a more of a field of... Philosophical questions, like, you know, to what degree you think you should have a control because there's definitely some danger in terms of going that deep.

51:09 Said, yeah, you can control the weights, but maybe the... You don't know how to do do it the best way, you end up with a suboptimal solution. So for the expert like them, I'm pretty sure that that, you know, that they know what they are doing. But I would say that for a lot of the enterprises, enterprises, they they probably... That's probably not there, but the primary area of expertise. So I think that will require some help in terms of, like, you know, even if they wanted to fine tune a model or I...

51:39 Usually, what I was suggesting for a lot of our customers that start with the harnessing and if you really cannot get around that, then you can think think think about, you know, using different models or tune your own models. But there should be a second resort. Know, that should be like you're you are number one by the priority immediately. Another thing I wanted to talk about is this notion of cognitive debt. So I've I've been using OpenAI.

52:06 You know, they've they've... You you can now link your phone to it and I can control all of my coding agents. And it's amazing. I I can I can go down the shops? I can take a walk along the road. And in some ways, it's amazing. In in other ways, it's not. And the reason it's not is when I go back to my computer, I realized that it it sometimes the agent was doing the thinking when I thought, you know, one of the, you know, the voice agent was doing the thinking. One of the underlying coding model should have been doing the thinking. Sometimes it wasn't using my established agents, it was just creating new ones and

Combating cognitive debt and AI slop

52:34 it was just, you know, it was a little bit messy. And and also the cognitive debt is... It's really important sometimes just to see the visual information in front of you. So how how you address that problem? It's a huge problem. I mean, it's a huge problem not just for the product development and our customers, it's also a huge problem for us organizationally because all our teams are adopting Claude Code right now.

53:02 And we have a lot of people integrating to all the Gmail, G Suite and integrating to Slack channels. People now started to, you know, sending Slack channel calls at the sections with each other. If people are not careful, they're sending a wall of text to the other person. And the other person will be like, oh, like, I have to spend so much time reading it. And over over time, we see more people just say, okay.

53:28 Now, you know, have have Claude read all the text and then summarize it back to me. It's hard. It's the problem, you know, like you... We humans don't have that much... I mean, the bottleneck has already shifted from producing content to actually validating and auditing the content. And that has been the problem my engineer has been telling me since the beginning of the year, like, we have so many so many codes to review, and I don't trust the agent to do the code reviews yet because, you know, I don't feel comfortable for them to just merge the code.

54:03 So I think we are definitely entering that stage. And I think we we probably need a lot of education for people is that, hey, especially in the coworking scenario, you have to educate people that, hey, you have to help people to summarize your point. And then also don't send people the AI slop that you have not read yet. You have to read it yourself and then really understand and agree with what you send, not just, like, you know, throwing that out out to people.

54:35 So I think it's a real product for the problem. I think encoding is particularly acute because you're just dealing with something of very high complexity. There's lots of background knowledge. Because it feels like in customer service agent or something like that, both parties understand enough about what's being spoken about.

7 Highlight · Why voice agents cause less cognitive debt than text

54:50 Yeah. And is is that part of it? Is is it something like if if we both understand it, then it's fine because the voice agent, it's a little bit lower resolution, but it's pointing to things that we already know. But in the domain where we don't know something, then it... Then there's that cognitive debt problem. Voice agent has a less of the problem because by nature, those voice conversations are lot shorter. Right? Like, it doesn't actually generate a wall of text. Like, our average output token size in the production agent is probably about 50 tokens. It's not a lot. Right?

55:22 So consuming those is very... Is is relatively simple. It's the is the reasoning behind it. It's like if you are querying a a larger agent to do the reasoning task where it generates a lot of reasoning tokens, that's actually more of a problem. Usually, voice agent by itself is is not a problem. Voice agent has a different problem is that it doesn't have the sufficient bandwidth in that communication channel because the voice require very ping pong y, very short that that kind of conversation.

55:54 So it has a slightly different problem. I think you... The the task you are doing is actually very interesting because I have been thinking about, you know, using voice agent to do vibe code... Code coding as well. That would be really cool. I think that would probably require a little bit of harnessing into it to actually make sure that, hey, voice agent always service the necessary, like, high level bullet points rather than just pass on a simple

56:22 sentence per per per se. But I don't know whether it will be easy enough to really for you to understand the full picture still because that's a lot. What I really love about voice agents or just agents in general is that I can do insane amounts of context switching. So I can be doing my email triage in one. I could be doing video editing in another. I could be doing coding in another. And it's really made me change my workflow. So now I have a breadth first search workflow, where I'm just kind of like popping off one, popping off another 1.

56:53 And there's good and bad because in a sense I'm not going really, really deep on things. I'm just kind of doing many things at once. But the beauty of the agents are, as we were saying earlier, these domains have become well enough specified that the agent can have some autonomy and I just sort of like feed information to every single one. So it's it's it's kind of changing how I work. I mean, you have you noticed that? I build a lot of these agents as well. I mean, currently, my workflow every day is...

57:19 There's only once... I mean, probably 2 software, 3 software I open, Claude, Slack, and Chrome, you know. So Chrome... Sometimes because I have to open Chrome so I can log into some software so my agent can do a computer use on top of it. So it's not really I'm using it myself, it's an agent so the agent can access to it as well. So it's already changed my workflow completely.

57:46 Yeah. Yeah. Do you think this is a mindset thing? So obviously, like, we are engineers.

Social friction in voice modality adoption

57:50 And what I did as quickly as possible was transform all the work I do into an agentic representation, basically. And because the million dollar question is we want to get to a point where voice agents are used for basically all work that can be done. And I mean, how do you see that transition happening? Voice is a very good input modality, whether the voice agent itself is the best at walking, you know, in intermediate layer.

58:20 I'm not actually sure, to be honest, because, you know, I'm being... Give you example. I've been using with Wispr myself. I I... I'm pretty sure you use it as well. It's a very convenient ASR tool. You Oh, Wispr Flow. Yeah. Yeah. I love it. It's amazing. Yeah. Right? I'm in the top 1%. Yeah. Exactly. Because it's so convenient. You can actually just talk to the agent. And the agent... I mean, despite that, you know, I'm not a native speaker, so then the transcription is not always accurate.

58:47 But, you know, those agents understand those, you know, errors still. So then, you know, it's not like, you know, I send a voice memo to someone and then they see that there's a lot of mistranscribed or they get confused. Agent don't have that problem. Agent would do the reasoning and say, oh, this word should mean that, this word should mean that. Okay. Well, I'm going to do it that way. However, I am a user that heavily using it. I use it even at work because I was like, at work, you know, you put your headphone on and people are sitting next to you and there are people already taking calls on, you know, on their desk already.

59:26 What's the problem of putting head on talking to an agent? I got judged by so many people that were like, what are you doing? That's so weird. I was like, no, I'm just working. So I think the voice, as I say, voice that... Voice modality has these like privacy layer where some people just don't feel natural doing that. Even me talking in the office just talking to my agent, people feel weird. That's the same argument as the Meta glass.

59:57 Right? You you can wear it, and then you can actually talk to your agent talk to agent. I got one as well. I started using it. People were like, what's on? Because I was like, hey, meta. Play the music. Take a big picture. And they was just like, people feel it weird. So I think there's this user adoption issue. So I I would say we are rare because, you know, we are really agent built.

1:00:22 We really like to actually work with that new modality, but I think a lot of people probably still not quite there yet. We are engineers. And what I do is I build tools to help tasks become agent enabled. So for example, video editing, You can't use an agent to edit something in DaVinci Resolve or sometimes I make 3 d animations in Blender. You you can't really... You can do that with an agent, but it it's always gonna go wrong.

1:00:49 So what I do is I build a tool. I build a mental model. So an abstract model of the domain, you know, and then I build an agentic CLI and I put a skill prompt in there. And and then my agent can do it. So I'm I'm always in this modality of tool building to make an application amenable to agents. And that feels like the gap at the moment because like my mom wouldn't be able to do that. So we want to get to the point where they can almost demonstrate. So they say this is something that I do and they can put a demonstration in there and they can describe abstractly what their thought process is and then they can build an agent interface. And then from there, the agent can do it and you can correct it when when it makes mistakes.

1:01:25 I have been building quite a lot of tools for my internal teams as well because people don't know how to do them. I think one of the most useful tool I build for the team is building these internal wiki stuff to track all the product features, functionalities, sales, like how do we pitch to the customers, that kind of thing, and then continue to iterate on on that.

1:01:50 And also build another agent that goes online research to search, you know, suppliers, vendors, competitors, you know, market trends, and then just building that into the Wiki. So then that is the context engineering. I'm trying to actually do that for them. There's also a lot of tools you can build around it as well, like access to your email, your Slack channels, and then, you know, I'm going deep... You know, because Claude has a lot of these default connectors, I almost throw out all those connectors because I don't think they are, first of all, fast enough. second of all, it's actually not powerful enough.

1:02:30 I want to go straight to the to to to the source. So now it's like really powerful. So, yeah, I think this just require a lot of that thing and then I think people... A lot of people just wanted to use it. You know, they they don't necessarily have the mindset of building something and use it yourself. They're kind of waiting for people to build for them. And what do think the next decade of voice research is gonna look like?

End-to-end turn taking in voice research

1:02:54 The voice agent need to be increasingly more end to end. And in that end to end, I mean, number one, turn taking must be end to end and collapse into the entire pipeline because that is the single most annoying bit when it comes to engineering and the most difficult bit to actually get it right when it comes to engineering. So that is almost... I mean, this is... It's already happening with a lot of these models, right?

1:03:23 And then we will start to see in the voice modality, we'll probably have to start to branch out. There should be a consumer voice for consumer entertainment day to day use cases, Alexa, Google Assistant of the world. There's enterprise voice use cases, starting with the enterprise contact center, more of a customer support and professional use cases.

1:03:48 It could be also... I mean, it could be also some like meeting note takers and then also like, an agent help you to coordinate these meetings, could be in that scenario as well. And then it could be also becoming the internal productivity layer of that voice between your codic cortex. I don't know what they will look like yet, but I do actually think that the voice channel that require for consumer and enterprises are very different.

1:04:18 One requires a lot more professional, a lot more like regulatory or auditable. The other one is really about fun, because conversation is also supposed to have fun sometimes. So I think there should be these 2 different branches coming out. And then we will see there are more hardware devices to be able to carry these voice agents because the voice is a natural modality for these IoT devices.

1:04:47 We probably have see a second wave of IoT, you know, coming along. How successful of that would be? I don't know. But I think I'm really excited about that. I actually buy open source robot recently, just like, you know, more like a voice speaker robot, but, you know, it's not... You can program a lot of it, but, you know, it's still like very crunk... Crunky. But I think that all these things would become a lot more prominent in the next 10 years.

1:05:17 I can imagine a future where the technology becomes so invisible that we can train the system. It's almost like teaching a child. And then I suppose a really important thing certainly in terms of organizations is how we share those learnings. Currently, it does actually feel like the technology is still very visible because there's still a lot of learning curve. And then AI... I mean, our... In our organization, people have been talking about AI slop because it's very, really visible. More and more people becoming, like, sending AI stuff and message to to each other. So it is still very, very visible.

1:05:52 I do actually agree that eventually it should just becoming, you know, invisible or in a way that you just blend it into your organization. I think that will probably still require a bit more iteration in terms of sharing the context, sharing, like, who you are and what the o- organization is about, I think dis- despite that we have done a lot of internal knowledge, uh, you know, ar- ar- ar- architecting, there's still a lot of principles that we are not writing into those MD files yet.

1:06:23 So I think there's still a lot to do. And some... You know, we are probably running out of this low hanging fruit now because some of the some of the things that they they are actually so personal to ourselves or to the organization is more, you know, spiritual, you know, it's about more like, you know, this kind of constitution or some principles that you probably don't print it out, you know, everywhere. Because the funny thing is that if you have the agent to ingest these cultural values of your company right now, sometimes the agent just come up with random ideas and then just overly focus on one thing. But actually, it's not about that.

1:07:04 Despite we write a culture values down like this, there's a reason why this come about. And those reason, a lot of time, is not injected into the context. So agent oftentimes, focus on the wrong things and just because in their weight, somehow they think that out of these 6, this 1 is the most important one, but that's actually not true for the organization. So we kind of have to share a lot more in order to get to that, you know, invisible state that you are talking about.

1:07:38 The slop thing is very interesting. And so my definition of slop is generation without competence. I'm using memory systems. You know, I have lots of kind of like adaptive specialization, skill adaption, and so on. So it always has significantly more context. Increasingly, it knows what I, you know, what my preferences are. I've grounded it with with the correct technical referenceable materials and so on. So the fact of the matter is that there's 2 modes of AI.

1:08:05 There's AI that has sufficient context and it knows what to do. And that AI does not produce slop. I think slop, unfortunately, is probably a little bit more subjective in a way that I engineer and context engineer my agent so that the agent doesn't feel like slop to me, but it may feel slop to the other person who is actually reading my message. Because in his context, these things I care about probably doesn't really matter all that much to him. Or maybe he's not aware of this is important yet.

1:08:37 So then when they read it, they thought it's a slop because it doesn't actually see the context behind the reasoning. So that is a bit subjective problem. So I think we have this internal debate or argument is that should I care about whether I generating slop and then just sharing with the colleagues or well, the fundamental debate is that should a human being reading those things anyway or should a human being just ask agent to read it for them and then summarize in a way that that is better suited for their for their for their focus.

1:09:13 You know? So so, like, we are... I I almost landed in a way that, oh, I should probably build a skill for everyone's agent to... Based on their personal context, summarising whatever message that people send to them, so then they can extract the useful information for them. So in that case, you wanted to encourage the output side to be as rich as possible, so then you send a wall of text, that's okay, because the other side, there's an agent summarized it for that particular individual. But I kind of feel like this is running in circles.

1:09:53 It's like, we human communication now need agent in between to help us to cherry pick the useful information for us. Shawn, it's amazing having you on the show. Thank you so much for joining us. Thank you. Yeah.