The Future of Audio AI: Insights from Mati Staniszewski of ElevenLabs

13,115 words · about 52 min to read · YouTube’s automatic captions

0:00 what you would try to do is effectively map set of characteristics of that voice in the model so try to predict what's the gender of the voice what's the potential age group of the voice what's the emotionality of the voice and you would hardcode all those components and then the model will try to predict those components and then recreate that voice based on those components you're listening to the AI native Dev brought to you by Tesla hello everyone welcome back to the AI

Founding ElevenLabs and Finding Product-Market Fit

0:37 native Dev I'm really excited today to have with us mati staneski if I got that right that's perfect that's perfect uh who is the CEO and co-founder of 11 Labs super excited I know mati from from the sou the London scene for a little bit here and really happy to have you here thanks for taking the time guy thanks for having me and a pleasure to be here so if you don't know 11 Labs first I'm a surprise second is I think uh really established itself quickly as the leader in AI audio right in generating uh anywhere from just get some text audio we'll talk about that some more to much more elaborate and dubbing processes just a little bit of Vital Signs uh now have recently raised a series C a couple hundred million bucks or 180 in the press release and the stats are first of all interesting in the choice of metrics but also impressive that you've recorded you have millions of users and they've generated a thousand years of audio content a fun stat on it and clearly it's widely adopted by well over half of the Fortune 500 companies so first of all amazing

1:37 well done thank you no it's the right moment for these additional funds as well I'm sure every company says that but it feels like it's still at the very beginning of what's possible across audio space and the combination that we are trying to achieve both building research and the product work and the application side requires a lot of those resources to be able to do both at the same time so building the models as high amount of gpus infra that's actually needed to get to get that of the ground and we still think there's like number of Innovations coming that we want to be at the Forefront as we think about multimodal models making it more interactive more emotional and then the application side still so many products as we wereing about AI agent space and bringing voice dabbing content internationally or just producing the content at scale yeah no amazing and yeah there's a lot to build and I guess on top of that the AI world moves at a lightning pace which I guess I didn't point out that the company is less than three years old right that's right we started at the beginning of 2022 so very both my cofounder and I left our previous companies coach used to work at at Google before I used to work at paler and then we swapped over at the

2:38 beginning of 2022 for the first few months worked deeply to understand whether the problem that we want to solve actually exists out there and second whether we can actually achieve anything from the product research side and then after a few months we we rised our first round as we started scaling the team and infra to bring a little bit more of those costs outside of our own Pockets into few others and then started hiring so at the end of 2022 we brought few PE first people in first three people in so it was team of five at the end of 2022 and then we publicly launched at the beginning of 2023 so the product is out there for two years the company is out there for for three years and the idea I know my co-founder for 15 years we've been best friends since since high school took all the same classes then traveled studied worked together for the years and are still best friends so yeah that's amazing and that's so important being a co-founder is it's a marriage and it's hard to find them on the spot it happens and sometimes successful but it's much better when it's someone you know so well we are so lacking that perspective like we in high school we are the best

3:38 friends through the through the years and through the years we always wanted to potentially find a problem that we are both excited about and can work on together for after and then everything aligned which we had a good and and comfortable setup at the previous companies then we had the idea that we were excited about and we wanted to work together so was perfect and have a a clear things we enjoy in our work together so it's clear split of responsibilities and some things that we like working together which was some of those parts so very lucky sounds like a

Pivoting from Full Dubbing to Text-to-Speech

4:05 very good uh beginning to it so let's indeed maybe talk about the early days and maybe the sort of the technical breakthroughs right so I I know and people like it's easy to find when you write look for a write ups about 11 Labs origin story how you were uh frustrated with the dubbing uh experience you and your co-founder uh Peter right yes of of Polish movies of it I grew up in Israel I'm familiar Israel fortunately has some brace that nobody's going to bother dubbing to uh to Hebrew not enough sort of Hebrew speakers and therefore you see things in in English but dubbing in in Polish was underwhelming and you wanted to fix that but I think that's a nice succinct easy thing to say uh but from there to actually technically execute that it's not one thing and I know you you had to break through a bunch of walls maybe in those months that you were exploring the ideas I guess I'm curious what were the sort of the core original Innovations breakthroughs or even just problem spaces you had to to do to deliver on this hey we want to do AI power dubbing and you are right it's even worse than just poor dubbing where

5:07 if you watch a foreign movie in Polish you not only usually you would have dabbing with low quality but you have one voice narrating all the characters so whether it's a male or female voice seriously you have like one one one person narrating everyone no emotions no intonation they on purpose keep it even flat so you can infer yourself the emotions on what's Happening and the crazy thing is of course we knew it when when we grew up but it's still happening for majority of content today and and something that we know will change as the technology evolves so maybe I a quick background there so my my cunder worked a lot through the years and he is a brilliant researcher he worked doing that at University initially spending time on image space then working Google on Knowledge Graph uh so closer to a text space um and he hasn't worked on audio but but a lot of the ideas that that we've seen in those other sister fields or other fields across AI we're potentially applicable in audio and we know that this will be something that is possible and fixable and on the other side from what I was seeing is that there's no great product for the space

6:09 to really do anything with audio AI space and then very clear problem set of people that we were out out bounding to of like would it actually help in your process when we started the company what was the sort of state-of-the-art for audio at the time like three years ago the what would have been like the best product out there at the time there were some early indications through 2022 of some open source projects that were a bit better but like most of the voice quality was what we would experien with Alexa or Siri like something that you can immediately tell is a robot it's robotic exactly but so we started and when we speak about this initial problem we started actually with building dabbing solution so in the early 22 the first prototype we've built was effectively a stitch up of few of those different steps so the first step is understanding the speech to text part who is saying what and what they are saying and already in that step the process isn't perfect because most of the models weren't able to detect different speakers or overlaps accurately the speech to text was okay for English but the element of when

7:10 things are being said was hard and the second was hard was the time stamps of of like exactly knowing when is that being said so that was the first speech to right it's more than just transcription which maybe was a little bit more evolved at least for English but it's a lot more of that sort of exactly segregation of the conversation and the time stamps of it exactly exactly and that's where we've seen already the gaps that we might need to address so spe Spees exper the first one and maybe just to give a like wider overview then you have the translation part of changing from one language to another of course in dabbing you need to keep the length rightly the same given you'll be dabbing in the other language in audio unless you extend the video but that's something that most platforms wouldn't support so there is a translation step and then there is the last one which is the text to speech step so speech to text we knew that there are some faults there were good open source models for the transcription side like Nemo from Nvidia but but really the other parts were were like and we spend a lot of time okay can we build that additional speaker diarization to detect the speakers can we build some better understanding of time step and how we Chun that together the translation of course something that

8:11 is a massive problem that we know we won't be able to fix ourselves so here we would rely on anything best that exists at the time yeah and this is three years ago is pre-chat GPT but it's not pre-m gpt2 is around were you already using those types of Technologies exactly like through 202 2022 like you could see some of that momentum so both gpt2 there were like other just cable apis that we would try and we try all of them and the truth is like already then you had 80% of the quality of the translation well and then it really depended on type of content if you had like voiceover documentary translation was pretty good the moment it started being like short sentences with a lot of emotions that when it would fall through but we would use any external API or starting to use the models which is also good for us because it gave us the signal what's happening in the field yeah and then the first step text to speech here I think in the very first prototypes we just used whatever it was best there's the famous open source project for some people in

9:11 the field called tortoise which was incredible where you could effectively the the first uh step change from those Seri Alexa voices into something that sounds human the problem with that model was that it only performed well on the very short segments so when you had a longer it would Terry it wouldn't be stable you couldn't create that voice stability over time so we started looking into like how we could potentially modify it to to our own setup uh using the the torti as a base but that was early 2022 and you're you're my kind of lay view a little bit think about it there's also the notion of capturing emotion right transcription doesn't capture emotions doing it so you'd have to somehow annotate that or maybe the Polish background music Started wait not expressing any emotions like the dubbers there right then is that correct so like the main thing like you want to capture in that process of course is what is being said how it's being said and who says it and the how hasn't been solved before and the way we thought about this in those early 202 days was could we

10:12 effectively take the original voice clone or reproduce that if we do it in a short enough sample in that short enough sample there will be enough of the emotions that we can effectively take from the original and then when we recreate that in the other language that short sample will carry some of that emot tionality across but we' built this prototype it was like this end to end dabing with those three steps and it was okay it was like not perfect especially like looking now if what's possible and we knew that all those components aren't good enough to produce the dabbing solution also as you think about dabbing it's like combining those three Tech like three different technology Stacks which were underdeveloped at the time and if you combine them it's in the some of the parts wasn't greater of the parts and at the same time we were starting to ask a lot of the potential users creators would you be interested in dabbing trying to prototype based on our work and it was very clear quickly that the quality is slightly below where they would actually be interested in like pushing ahead at the time we could deliver and then we took a step back and

11:13 that's to your second part of the question which was to be able to solve this problem truly instead of relying on the technologies that exist what we really need to build is our own models so not rely on any of the the ones that do exist for audio but actually go a step step deeper and try to create our own starting with text to speech model and voice cloning or voice creation model to fix that last step right which at the same time coincided with understanding the pain point of when we outbound it about dubbing was like yes dubbing is a problem out there in the future but really if I could just do the voiceovers more easily without my voice if I could correct and pre-process or postprocess and understand how the script can sound or change the corrections after I've done it that would be so much more immensely helpful so we had those the two very clear signals of there's this big other problem that is on the way to solve what

Breaking Down the Audio Stack into Modular Models

12:00 we set out with and then the research isn't good enough so we actually need to start take a step back and build our own models and in those models did you still break it up like when I think about building a company in the space and do have kind of a lot of developer audience so if you think about how do you chunk how do you modularize this problem did it require thinking about dubbing as as one thing in which you'll train some sort of brilliant model to just take voice in one language and and spit out correct kind of Alternatives in another or do did you still find kind of a need and value in breaking it up into these different stages that you talked about and training a model for each of them definitely breaking it up and training each of them it's a it's an interesting question because this is very true for our work over last three years like breaking it up and working on a separate models but interestingly as we think about the future we actually think it will be more like a joint one model that does all those actions and we can talk about that too but but in that space it was like let's break down the model and let's focus on where we see the biggest

13:00 value that we can create both in terms of the Innovation the Delta to the space and then in terms of value to the user so the first one was specifically text to speech model can we create a model that understands the text uh better than the models that exist and based on that understanding delivers uh a better emotions and intonation so to make it like more specific on an example if you say what a wonderful day the model should infer that okay this is likely a positive sentence and pronounce that in that positive way now if I said what a wonderful day and it's let's say a fragment of a book what a wonderful day I said ironically the model should know that and pronounce that very differently and then if you have like a dialogue sentence the model should once again know this and be able to reflect this dialogue between two two interactions so we knew that the first and that was like the first Innovation we brought into the space was this much better context understanding of of the text itself to bring that now into speech so that was in the text to speech model but it is still not there was no product or I

14:03 don't remember a stage with 11 Labs where you were offering a sort of V like the transcription piece of it this is still the text to speech piece so you're still reliant on existing on existing models in the first phase so that that's yeah that's a great point so here after we've realized how dabbing is like still complex problem where the quality isn't good enough but at the same time realize in that text to speech space or narration space there's such much bigger opportunity let's take a step back let's not do dabbing right initially let's still keep it as our North Star in terms of technical problem but initially let's start on the thing that we know we can solve so we took a step back put dabbing on aide let's start with text to speech and then we'll get to dabbing in a company as we develop sense big Vision small steps like you have to it's not a very small step but you still have to exactly aspire to that big technical problem but there's real value in just the text of The Voice piece exactly and then like we knew that even if we fix

15:03 one of those components best in the hopefully in the world then we knew that other components are also improving based on what we see in the field so there is this great help of the field just in generally doing better where at some point translation will hopefully be fixed to the extent we can adopt it instead of building something ourselves which would be such a different and hard problem so Tex speech was first um where we've done that Innovation and that was like the first product around that and the second was how you can create or clone voices how you can create synthetic voices or clone existing voices and here also that was like a a second big innovation that we brought where traditionally when you try to

Letting Models Learn Voice Characteristics Abstractly

1 Highlight · Replacing Hardcoded Voice Traits With Abstract Vectors

15:42 recreate a voice what you would try to do is effectively map set of characteristics of that voice in the model so try to predict what's the gender of the voice what's the potential age group of the voice what's the emotionality of the voice and you would hard code all those components and then the model would try to predict those components and then recreate that voice based on those components we took a slightly different approach where instead of us hardcoding those features that we didn't felt represented the voices in the right way let model decide what those components should be so keep it abstract to the human and model will effectively Pi pick its own Vector numbers for each of those characteristics and then we found a way where we can have a lot more of them than usually any model would do and reconstruct that in a better way which is very very very aligned with in general kind of the Transformer and LM World type models in which instead of trying to encode it across a thousand or 2,000 or 10,000 different properties now you can have millions and billions in the world over exactly how many parameters if that's the right term to

16:43 use did you end up creating for that the initial models were pretty small it was like I think the first model was 100 million yeah so it's pretty small for a model but there's no way you could have defined 100 million different sort of

16:57 distinct attributes of a voice of the no it was 100% it's both the Transformer Innovation and the diffusion work was a lot of the the ideas and the initial bases for that that models that we've created came from so both like great thing which is exactly what hasn't been happening in the audio space most of those didn't yet bring Transformer or diffusion models into this space and we could leog a little bit of that space and start T to speech inv exciting so you had to combine um I don't know if it's really two things but I think if I back the journey you you started with sort of big aspiration of dubbing you broke it down into these different pieces of it as you should any problem you started with the state-of-the-art in the moment got to a proof of concept shall we say right or sort of some some offering but then customer conversations got you to appreciate that actually this middle piece there of Tex to speech is actually very valuable in its own right and so you can really focus on making that amazing and you identify the pieces that are maybe don't have to be the things that you excel at like translation where you can say okay I can

17:58 rely on the ecosystem for that and then you just said about making the text to speech piece be incredible and ahead of the competition and Le into that yeah exactly and it's it's good timing that this conversation is happening today and like through of course through the years we then adopted a lot of those components from the field into the models but the speech to text interestingly while the transcription is incredible with what whisper did Gemini and few other Solutions it still is relatively poor on those components that were lacking like speaker diarization time stamps time stamps are still not there and we are as we speak today or over the next few days we'll be releasing our own speech detex model which will solve all those components that traditionally weren't good enough so it feels like all those components are finally three years now on are getting into that place where technically are solvable amazing first of all congrats and and very excited I get my hands on it and uh and try it out I think we've had an exchange as I was building an app with my son creating

18:58 sort of small YouTube shorts from some facts and things like that as he was playing with that at the time and we're talking about getting the timings and the subtitles the understandings of it at the time so I'm very much Keen to we ended up creating it it's like specific very simple text that calls out facts in YouTube shorts so yeah there's actually like a channel for it right now and it get some sort of some tens of thousands of views of it and I think this is I think you were just a few months into the like the product was out for a few months when I was early user so very excited but ke to do that and also I love the expansion element of as well are you still a user do you know I am still a user yeah I've used it so if you check your inbox we've done this thing as part of series C where we wanted to figure out how like like we wouldn't exist without our users and our community which really propelled this set of possibilities both to tell the world what's possible but tell us what's possible too so we did give everyone a small all the early users I think the first thousand users an early gift of appreciation where they will get same swag that we have as a team so I think you should check your email I will check my email I don't if I made the cut to

19:59 the first thousand maybe I don't I don't know what the count for and they need to be active still so it's maybe it's should I am still a user including some some workshops for the kids so I I love the I just want to get a little bit into sort of the product side so I I love the journey and I love the both the sort of the growth aspect of it as well focusing on it and I think there's a lot of like entrepreneurial learnings from it that build out I do like I want to talk a little bit about sort of the the the way you package the product and the developer experience uh but I just want to hone in a little bit on just like when you got started you had to innovate and do these big things in dedon like big research breakthroughs but now you have a product you have millions of users and you probably have this tension between how much do you swing for the fences and try to think about some brand new thing that is amazing versus where is it that you probably have a million incremental things on how in Polish these words don't get pronounced correctly or whatever it is right making it up how do you think about balancing the more incremental aspects of research versus trying to rein invent the foundation because it is a fast

Balancing Fundamental Research with Product Optimization

21:00 moving world you probably can't do zero in the lad bucket yeah the I think the EAS there's an easy answer here in the harder truth around it but easy answer we still think you you just not do Innovation or new models or to make it more concrete we actually think that there will be one or two new model breakthroughs that are required in the audio space to really get it to the another level if you were to rely on current infrastructure and just improve some of the pieces it's it's not going to solve the problems you to solve especially as we think about agentic workflows where you need to add voice in like interactions and customer support call centers or education where you like learn uh something that's going to require very different type of experience so the thing will happen over this year and hopefully will be one of the first to do it well is going to be multimodel approach which effectively combines the llms the reasoning side and and the audio state-of-the-art here what we would do is effectively use our state-of-the-art models use the open source llms fuse them together fusing

22:00 isn't an easy task and then create a multimodal experience that is effectively what people will refer to as voice too you don't go through those steps in the in the middle which should help for everything from understanding the emotion better to doing dabbing better to doing that interactive experiences better and this is so important that if we didn't do it what we think would happen is that we would create an okay company that will be there for the next five years do set of use cases but not company that we would love to create which is like still leading on the research side hopefully is here for a few more decades and is leading from that research perspective in terms of what's possible now the so that's the easy answer that like you need to and that's where we actually took an active choice to like overweight towards doing the innovation in the models and potentially sacrifice some of our lead with the current generation of models because of how much we believe that the new generation will be important would be important although that is very Ed Case specific voice to voice I I can I know intuitively see how that would be beneficial it feels like any intermediary representation like

23:02 getting it down to text probably loses so much information uh you have to be able to get at end end but also we just spoke about how text to speech is also on its own very valuable so I guess just ask as the the software kind of geek in me might say is there a format in which you can represent something as an intermediate so that we can people can create text like that does it have to be the sort of the uh blackbox brain that just gets trained and sort of all knowing I am I'm being a little bit fous here does it or can you represent something in the middle it's a tricky question but it's like still even with that fused approach you will definitely have the same component so even if the the backbone is a model that has combination of audio and LMS you can still effectively create like a fine tune version which is text to speech model and it's better than the existing ones and maybe to make it like an a specific example let's take a a narration of of a book today you give a text it understands somewhat the context and can deliver the audio pretty good to

24:03 the human level quality in some cases of narration but what it couldn't do is for example detect the different speakers in that book and assign different voices or create sounds and that's what we think the multimodal approach will bring where if you listen to the book it will be like more like a a movie or immersive experience where all the voices are created and assigned that you can have that that different set of narration or narrators when you hear a scene about say thunderstorm you hear raindrops in the background so there's a much bigger brain behind that model so yes you'll definitely have those separate models with higher brain power than what was possible before but your point earlier on is is spot on which is it will take a while before those models are stable they are as good off the quality in audio and as you think about a lot of our big set of clients on the Enterprise side we work with over 500 different companies of of of bigger size it will likely not be at the level of quality they require for the next year maybe

25:04 next two years and that's to your second and the harder part of this question is initially we overweighted took almost no we need to do all on the research and innovate on the next model right let's not do anything on the current and then we realized after pretty quickly but after a few months maybe a bit slowly once a good customers yeah you cannot just abandon a lot of the the work

2 Highlight · Splitting Research and Research Engineering Teams

25:23 that's required to make that one or two% differences to get it a little bit better so now we use we have our research core research team builds the new models and then we have research engineering team that tries to improve the current generation of models like how we can really cut it to close that Gap and make sure that the it's whether it's 95% today and we can get it to 96 or recently one of the things we've done was we we had a model which was text to speech model which was great for interactive use cases had like roughly 200 milliseconds of the time to first bite and then another model got created that from a competitor that was roughly 100 milliseconds and it felt okay what how much time do we invest and and still wanted to show to our customers that yes we are the leading Labs so our research engineering team brought it back to 75 milliseconds we have a great researchers Max and Tony that pushed it across and now we have the fastest model and that's interesting I'm I'm going to go back a little bit to organization later but but you described these not just as a percentage investment but actually

26:24 literally different teams so you have one team there might be multiple under them but one part of the orc focusing on the more sort of big swings try to reinvent what is the status quo and then another that is more on optimization and is it a different what would be the

Structuring Teams for Breakthroughs Versus Iteration

26:39 difference between the profiles of the individuals you'd have in one team versus the other is it just allocation or is it actual different skills there's some definitely there's some overlap where you could imagine one person being closer to the other but roughly research our research team is extremely small and here these are mostly the people that we believe can create net new ideas net new work new breakthroughs for the world so they could create a paper that we believe could be one of the top papers on Europe orml or any of the other conferences so they need to be really well vered in this space and I think we've been able to accomplish when with my co-founder who I think is a genius level and and pushing all of the research and one of the best researchers that we have was able to bring a lot of the other researchers that love working in that space and are at that level so few world-class researchers that are the best in the world and then research Engineers are usually more uh people that we think you could give them any paper and they could Implement that paper it would be a bit harder for them to create those new net new ideas or

27:39 create the papers themselves but if you give them any piece of work they could likely replicate it and make it slightly better or tweak it and for research Engineers to also make it a little bit more concretely on how we frequently stuff those people researchers we just get from wherever we see the piece of achievement on open source or paper or through recommendation these are like the top people we think there's maybe hundred in the world that the pool of c does what we know research Engineers we usually start with people that join as software Engineers maybe with some ml experience and then if they are proving themselves in a core product we can bring them into research engineering space got it yeah very interesting on it and numbers wise probably that first team the research team extremely selective in so of the skills that you bring into that but probably numbers-wise a lot smaller than the research engineering uh both still pretty small I'm impressive what they can do but it's it's a part of the funding part of getting the funding we have five researchers and and five to 10 research Engineers oh yeah very tiny yeah yeah amazing how much how much

28:39 output they they 100% 100% so let's talk a little bit about you've been on IBM right I sadly I don't have sadly I really actually when I was at IBM which I got acquired into with my company the IBM research was actually quite interesting to me but I do not have a degree and so the one primary thing that was well out of my reach was I research got it got it I've recently read about this concept that they didn't know of course this applies across the organization and some of the ideas we try to steal but I think they had was one which when IBM became this Big Bam of like how do you keep that Innovation going and they created this concept of wild duck so you had uh wild ducks were eight people selected and Reporting directly to the the the the founders which just had a free mandate on just going after any idea that they thought will push the business forward and sometimes I feel about our research is

Expanding from building blocks to end-to-end conversational AI

29:27 it's kept a little separate so they can really innovate something they can do something well yeah yeah that's interesting I've not heard of it when I was at IBM and who knows maybe I would have stayed at IBM a little bit longer if I get to be part of those uh maybe switching a little bit back more to interacting with your platform and I want to talk about developer experience in apis I just mentioned how I have used your apis on it it's a good pleasant experience using those and I was very much using the modular pieces like literally here's a piece of text get me give me back some audio here's more parameters more sort of tweaks not AI parameters and and I I think for a while that was your only API type building blocks a platform a model and I think we'll talk in general about the evolution towards app but I think on the API front you now if you open up the 11 Labs API page you'll see conversationally AI you'd see pieces that are more end to end of sorts like they provide a more complete functionality so I guess my question is one is just I'm interested a little bit on the journey from one to the other what made you do it and and maybe even when you think about someone building on

30:27 top of the platform what are you seeking or optimizing for is it more about the full unit is it more about the pieces how do you think about it that's a big question and definitely our approach evolved over over time or expanded over time but let start with the story just like how did this conversational AI kind of a piece came to be and how do you think about it so starting from there it's as you've said initially we had text to speech voices and we didn't have much more of the other components in the conversational stack but what we've seen happen is both developers and and our Enterprise clients effectively what they were doing is using that component and building a conversational experience and now it's not a simple thing to do it takes usually months of implementation time to do it well if you like we are speaking now you will adjust your emotions you want to make it relatively smooth and quick when you respond to each other maybe you want to interrupt each other and then get responses responses back I wouldn't interrupt you in a minut sorry come but this is exactly what we've seen being tried to solve across across so many of of the clients and it wasn't

31:29 easy everybody had a slightly different approach it was not always great experience for the end user and and it was a complex problem which was like great one thing we know we should and want to provide is help provide that end

3 Highlight · Solving turn-taking and interruptions in conversational AI

31:41 to end solution so people don't have to build from scratch but something that we can bring to the table and then two as we buildt so closely to the research pieces we can actually optimize it even better for how to optimize for turn-taking or or interruptions to to make it smoother because we can embeds a few uh predictors on both sides of like when will the speech when I will stop speaking or will the content will go to the pause and and that's how kind of the element came to be where we decided let's meet our customers where they are let's try to build solutions that are actually solving their entire problem versus part of their problem right and and the conversational AI was one of the big steps in doing that yeah but but

32:19 conversational search interrupt I'm going to be self-conscious every time I interruped talk about interruptions as you as you do the work to allow your customers to handle interruptions more often like why pick that right you probably have you have these millions of users you have conversational AI as one common use case but there are probably many others and I don't know exact prevalence but I don't even know if conversational AI is any sort of like majority of the use cases it's probably some single digit maybe sub 20% usage no the so it's low double digits no maybe not low uh medium double digit percent but the interesting thing is that like in general we think voice will be the future of interactions the digital interactions of how you interact with interfaces of of the digital world and it can carry so much more emotion so much understanding than text and the pleora of use cases is wide and happy to speak about some of those too but maybe to give two quick examples like from Healthcare space where you can work with nurses we work with company like Hippocratic which will try to automate the process where nurses call patients

33:21 to remind them about taking medicine or asking about how they are feeling people that frequently cannot use computer at the same level of proficiency such a great use case so much of potential of reaching the people and then completely different side company like dis gen which helps train 911 responders so they can speak with the potential person understand how to respond in that emotional situation um and similarly massive scale then customer support call centers new ways of digital media so we see it even if today it's a little bit smaller although it's starting to get some momentum in the next five years it will be one of the key models and then the second it was definitely one of the places where we knew we can provide that that solution so conversational AI is like probably one of the categories which we believe will be one of the the biggest the flip side of that question is going to be more interesting of we youve probably seen so many different companies trying to build a gentic workflows it's clearly becoming a buz wart so there will be an interesting competition of competing for talent competing for attention right that that

34:22 will be harder but going back to your to your other part of the question as we think about our work we have have on one side creators that use the platform and the entirety of our audio tools across effectively the interface right then we have a big group of developers that will use and deploy the API or try new use cases and from the start one thing we are trying to do was bring the interface and the API roughly at the same time so people can easily built with it can show that what's possible to the world and then of course third group the big one is Enterprises where you do need end to solution and that's where like our focus is is being built so at the core we are providing horizontal layer for everything AIO AI whether it's text to speech voices text to sound effects in the future speech to text and then for our business parts we are hoping and and are building endtoend solutions for specific things that we believe are important conversational one for specific sectors and the second one is is Media entertainment where we provide

35:22 a dubbing solution for end to end work yeah interesting as a complete path and I guess can you share today indeed you can use 11 Labs through the through the UI and create a bunch of of audio pieces and I've done the the amateur version of it but I'm sure there's full-on business use cases for it and you can do the developers what would you say is the rough split today in terms of platform usage is it is it mostly are you mostly a platform or mostly an offering today like mostly a a product yeah the probably the most frustrating answer which is it depends on the user group given we have so many but starting with like high level numbers it's mostly platform for selfer subscription users so like the creators of the space where we have millions of people coming for our platform are going to be using the platform as the interface of combining those components and then as you shift further up to to either small medium businesses or Enterprises it will be majority of of API users now within those groups there are like very clear distinctions that we will see so for example gaming companies that we work

36:23 with Will mostly use the API now any media Entertainment Company that's creating ads voice overs dabs them will use the UI conversational AI this is an interesting one because conversational AI is at the end of the day you configure things in the interface but you are an API user like you need to configure that in your application connect it to your knowas it is a solution though it is not it's not a building block it is you might need to build on top of it but it's done the the The Lion Share of your conversational AI functionality 100% 100% but then you do need that link at the end of connecting to your CRM or knowledge bases to to go through through the API pieces but with this like end to end solution where you can configure all those components through the platform and what are the set of constraints that you need to build in place how do you do the end tooling what are some of the security parameters you want to set voices latency configuration so there's a pretty wide set that that we provide which is like common across across the problem set that that we see and of course op so much on Incredible use

Navigating the tension between platform provider and competitor

37:24 cases too yeah and and I guess maybe this's a slightly tricky question but there are a lot of AI applications today that are being built all sorts of startups Innovative teams and if they need an audio component of it they might go to 11 Labs as as their solution for it uh and oftentimes for companies building like that it's there's a bit of a a scary relationship with the foundation models including maybe a little bit less scary in audio I don't know maybe just less discussed but there's a bit of a concern okay if I'm building something that is audio related will 11 Labs or I'm just rusing us example of any of the foundation models expand to cover this space and how do I how do I kind of balance or manage that risk do you have any guidance people must ask you that question of hey I'm building this on top of your platform are you gonna expand into my space how what type of answer do you give them the so it's interesting because we also have seen the opposite of that problem and I will answer your question too but we've seen exactly the opposite in the approach we've taken which probably some of the hardest moments in the in 2023 at

38:24 the end of 2023 were due to that where we kept our a is extremely open and we wanted to solve set of use cases and we've seen people solve the use cases that we wanted to solve before us and take all the take all the users Market opportunity exactly the closest one and probably like one of the mo moments

4 Highlight · The risk of clients beating you to your own product vision

38:41 which was emotionally frustrating was with dabbing where we started with dabbing we created Tex to speed we created all those components and then in 2023 we we were offering all the apis openly and we were planning to launch our own new dabbing solution or the first like true dabbing package solution with our apis in the product in like roughly Q3 of 2023 we told it to one of our clients maybe it was unrelated and this client released their own dabbing two weeks before us and took all the users that we wanted took our vision away from us which was like okay not happy especially how close this was in our origin of course we double down and are continuing to work on that but the guidance in general for us is like we will keep the horizontal um solution up there we are really focusing on on very small subset of the work um the one is within conversational AI space how we can create more immersive media experiences more interactive media

39:37 experiences how we can work in call centers and build that that audio element there and potentially uh uh potentially some of the close to call center elements but not entirely a customer support um then the rest we are a partner we want to effectively wherever the research allows us to provide you an advantage from the product perspective that's where we'll go the rest we will keep it open and on the media entertainment side the the the thing that we started with building like an entire process for creating dabbing is going to be important and maybe to take a another element what we want do is focus on on video we focus on text manipulation to a large extent so any of the use cases building with that have the natural advantage and similarly what we've seen is that if you work with both with the companies or users know this better than than I do it's like there's so much n integration that will need to happen for most most of these cases apart from the on mentioned we are not planning to cover and that's where a lot of startups or companies building on top

40:38 of 11 UPS go yeah that's good I I love the clarity I think probably when you're building on top of a platform there's no guarantees in life and Company strategies change and opportunities arise but fundamentally you want to the challenge I think that happens in the foundation world is more when there just there isn't declarative statement of these are the domains if you're going to be in these and you're building on 11 Labs you might get yourself in trouble and so you can do that you might be competing on the platform with the platform that you are building on top of 100% although we saw interesting sorry now I interrupted you but we saw the same we we you know it's it's something that we think about too it's like we want to allow so many companies and individuals to build how do we make sure that they are they feel that transparency too and when we were entering conversational space for some of the companies that were closer to that we let them know before that this is something that we happening we had a ation and an interesting thing happened where then for most of them they effectively saw this as a potential Advantage where they can now provide that conversational components inside

41:38 their platform and focus on the additional parts to bring the End customer with the specific integration for healthcare and how do you connect to the systems on the hospital so that was like an interesting when the ocean Rises all the BS yeah you want to know which parts I guess one aspect of it is where is it that you might compete with me and the other other aspect is how do I know that you will continue investing in this space and so the Declaration helps on both fronts so I think maybe just

Targeting real-time multilingual translation and call center automation

42:05 continuing a little bit on the sort of app and product uh path and and we'll talk a little bit about org I guess if you cast your eyes five years out and you think about where is the the value that you are providing how does sort of the company look I think today I would characterize you as it's still like predominantly a a a core model you are able to do this text of speech now spe to text better than than anyone and you're you're building that out and people build on top of you and you have those Solutions do you first of all do you agree with that characterization today and when you cast your eyes three five years out which is you know more than double the company's lifespan right right do you do you think that shifts do you think the foundation model give you like a right to play and and becomes a critical feature but no longer a defining feature yeah there world will be interesting in the next two years so on the characterization I like what we started the company with and still so much believe this is that you need both you need to innovate if you are to create like a company of the scale we want you need to do research side then

43:05 you need to do the product side and I think we've been able to build some of the major Innovations on the research side we haven't yet built some of the major Innovations on the product side although I think the conversational AI solution is the first one that brings some of that to reality where you can really have the full package workflow and the dabbing is similarly that product aspect but I think it's still still early in terms of the adoption of that product as we think about the world so maybe a spin on that first characterization pieces I think from company perspective we've built incredible research we've built great set of products in terms of the adoption the research is already heavily adopted as we think about our work the product is just getting started of where we will will go now to the second question as we think about the five years from now I think the things we would love to to continue being characterized at is that we were able to push that envelope of the for research forward the multimodel AI audio experiences and really being the ones that fixed speech as a category

44:05 audio intelligence as a category where you can understand who is saying what you can recreate that to the human or better performance and whether that's that interactive experience or a multilingual experience we can do that across all languages so we want to be like known as one of the research apps if you want to join this is the place to go but the second one is have the full-fledged audio platform form where regardless of the workflow or use case you want to build on top you have the components and the solutions ready for that and I I hope it's going to be much quicker than next five years we think it's going to be that next year or two is exactly like we spoke a bit earlier have those few cases where we go super deep let's take dabbing as an example we want to be able for you to create a dab on our platform we want to connect it to the live streaming event so you could dab it on live with the work so like next time so we worked with Lex Freedman for example who dabed his conversation when he was with president vadir zinsky of course we dab that in the process afterwards but what if in the future you could have that conversation between

45:06 them where they speak original languages yeah and that is generally like a holy grail of be able to communicate cross languages live in real time 100% like the cultural educational barus is is incredible and then the like how can we help in the conversational space go the Deep solution for call centers where if you are building that that application like recently as part of CC we brought in do Telecom on one of the backers how would that look in the future where you're are taking those calls at scale where it's not like it is today where you're trying call calls any customer support usually you're waiting for a long time you're frustrated that they don't understand you they don't have the right details and and swapping that that like immediate great experience right on the flight that you and that's a good example where You' need to think do you go all the way to also knowing what it is that you're supposed to say so understanding the support cases and the needs or are you the enabler the conversation aspect of it but someone else comes along you're plug in with whatever solution of choice uh it is that sort of provides the the actual knowledge base interesting and and I so

46:08 I I take from that you say okay we still think that the fundamentally it would be a defining feature that we're The Cutting Edge but we will also continue to invest in making it more accessible and aligned for the specific use cases that we focus on 100% And you think like when you envision the sort of this world of foundation models and everybody's throwing multiple billion and many kind of resources at it of all types do you still Envision the audio focus even with the multimodel as in audio to audio the different definitions maybe than than maybe some others would have come to mind when they say multimodal you're how do you think about that as a discrete

Leveraging audio-specific research and specialized data labeling

46:44 slice of the foundational model World versus the all knowing brain maybe that some sort of AGI ASI type path that we're trying to get to that the open eyes and the likes are trying to reach it's a complex one we do at the core we know and believe that the set of use cases where having the highest audio quality and the highest audio controlability is going to continue to to both grow and be the slice that we want to solve and there's so many other companies that of course are building but they usually falter in terms of the specific quality or building the solution for a specific use case so almost the if you think about those two components is like research will allow you set of breakthroughs there still the ones optimized for audio will be where we can build that Advantage with the specific data sets that we create and how we create them and set of breakthroughs but then the second part will be increasingly important in the next five years it will be like how do you provide that solution that platform

47:45 uh element to use the research at scale and in a good way now what is interesting like the llm space now is so much about scale and and getting more data getting more compute in audio space we still think it's a little bit earlier in terms of the breakr that you need to create to to compress or decompress audio so that's a good Advantage for us where we can focus the research resources and now what will be complex and that's the part which will be interesting is today we'll have growing size of our Audio models where we see those breakthroughs um and we as you think about the multimodel experience will use the LMS that are out there and fuse them in yeah what would happen if the open source sto that would be a potential problem where we don't have the access to the LMS that are state-of-the-art in that same way but of course you have the you like in audio I think you are still earlier than llms in terms of diminishing returns in terms of use cases that are possible so hopefully by the time potentially companies stop doing the open source the if they ever stop it that's unlikely that's a

48:46 different sort of conversation exactly but that would I think for what we can bring to the table is audio expertise that that none of those companies ever focus on and Bling That with LMS will help and there's so many components in their research work so we talked about breakthroughs but even we now have a team a big team of data labelers that have voice experience we have voice coaches that tell them how to understand audio uh how to annotate the emotions in the right way so we can then use that into I think that's the virtual cycle is around you get the product you get more users but that in turn gets you more data which in turns allows you to do it and I think the more uh specific the domain is the more you can accumulate data that that is unique so very cool I would love to see the sort of the Innovations on it have all sorts of like deep thoughts about the uh open weights model even open sources is like a little I guess some of them in audio some of them are actually truly open source but uh but a lot of them and that's its own ratle here yeah they are pretty pretty bad conversations and also the yeah and the Strategic kind of aspect of it one of the key distinctions making them not to Rattle too much here but like key distinctions making them my beefs with

49:48 them is that they're not really open even the open weights they're more freeware but because in part you can't the community can Fork it and continue it if they behind them chooses not to not to continue releasing but some of them indeed in the audio space are trully open source true the audio ones are so there are some recent good ones especially on the architecture that that we've seen that we created in 2024 there is like now good examples but and they are truly open but there's always that lag that and that lag is an an interesting one even as we think about our approaches like what is the lag which you can still build the advantage on and we think we can especially because we are at the Forefront of audio and like our goal is to to roughly be a year in front of anything that's possible out there so to make this to continue making this TR we need to release our model in this quarter the new model which I think is possible that's the kind of The Guiding Light for the research work but it'll be interesting five years is a tricky question in general yeah it's hard it's uh in this world it's uh in this at this space it's hard to imagine it yeah hey

50:48 time flies and there's like I've got a million more sort of subsequent

Operating flat organizations with no corporate titles

50:51 questions but I want to spend maybe the last sort of few minutes here talking a little bit about organiz ation and so you you already described the sort of focused research organization on it the tiny and high impact team and the product you've published a controversial decision that you've that you've made just unusual uh on it in which you've shoed titles you have no titles in at 11 Labs yeah I guess just open up like can you tell us a bit more about what it is that you decided to do there and what was what drove it so we are now team of over 100 soon approaching the the magic number of the 150

5 Highlight · Eliminating titles to maximize individual contributor impact

51:27 but is behind the scenes really it's more like a team of 20 teams of five or like 10 teams of five to 10 depending on the specific period of time and that's what we try to do it's like extremely small high degree of ownership High degree of Independence and they can just run which helps us move with building on one side of course the research but very specific solutions that go deep uh very specific markets where we try to get through the goto Market part and and that responsibility is there that high degree of both impact and owners is there now because it's and that's how we think about effectively nurturing that flat organization where almost everybody is in ic like you are expected to to do a lot of that IC IC work and and where we are today is down to ic's pushing the barrier was possible so for IC is being individual contributor if anybody's off the acronym here sorry yes that's right and the thing that we wanted to create is an environment where people feel that when they join this organization especially at this stage where things are moving so quickly they can have any

52:28 size of impact in any amount of short period of time where they join they can have that impact and the title shouldn't be a limiting piece in terms of how much or how you are what's the perspective on you from the others in the organization so that was the first one like impact shouldn't be defined by the title it

52:44 should be defined by individuals no titles helps that you can get anywhere in the organization and and and bring those ideas forward second there's this concept of the uh and I discussed some questions in social media that we are trying to keep the best idea wins where you have new people they are expected and also they we know they are great people that can contribute on on on day one and and we want to keep that that that idea that if you join others are there that you can that you can compete on those ideas of course if there's like a really no no decision making we have team leads for those each five team five people teams that will make that end decision and then so they are identified leads even though they are yeah they they don't have a t but everybody knows who's the sort of the lead exactly and the structure that we have is as you think about those teams that structure is heavily evolving still so we wanted to keep a little bit of the of the dynamism on that higher level of like how we are thinking about restructuring and that is all to say that as we think

53:44 about the future those structures will solidify in some ways and maybe there will be a good point to introduce those titles that give that that that that element of additional recognition and understanding for new joiners of who is where and who is doing what but at this stage impact matters ownership matters the best idea wins and and and is a distraction for climbing a ladder that that we don't want to nurture where now you are building for the entire company not building for a specific component you probably heard about all those ways U of like how the frequently when the companies build themselves they they have a team and the individuals build for their team rather than for the company and of course those should be align but not always are so that that hopefully helps avoid us but then on the other side which is the tricky thing of of your question of like on how that's on on some of the thinking is that you do want to keep the some form of recognition and we are now investing in different ways of recognizing the individuals by being much more open in terms of our releases of the work we do on like who created that release giving

54:45 them credit and giving that element out there and then there's always this interesting thing of how the titles can inflate the or the inflation of titles in the organization there many problems that might exist with that titles but I think like behind the scenes not everybody gets paid the same and there's an element of seniority whether within 11 Labs or uh past experience on it that you would put different type of sort of expectations right if you're paying someone 100 Grand you'd have different expectations than someone you're paying 200 Grands from them so you know you want if the person getting the lower salary there has an amazing idea you want to allow them to unleash it and drive it but at the same time you're not really looking at them the same as an employer or as a boss I guess how do you think about because the risk with the step of change which I I absolutely see the appeal of it although I am um I have doubts around its ability to scale and so maybe it's also right for now and not right for later um but I think the risk really is that there's a still seniority it's just hidden instead of putting it

55:46 on people's uh titles I guess how do you I assume I'm not wrong about the salary comment and so how do you control for kind of hidden seniority versus explicit one yeah and you mentioned also like a line which I will echo which is like right for now maybe not right for later and that's exactly how we approach many of those decisions is here we think it's the right moment for current state where that organization is evolving as a whole and likely we'll need some other component or structure in the future second on your structure piece our structure is actually very clear and transparent so we know who the teams are who is the person responsible How They Roll Up further up most of those teams are small so they are those equal levels but it starts to now create that so we still have a clear and identified leads that everybody knows there's no you don't have the title attribute that is going to stay and be there forever because we expect that it's more an assignment than it is a uh than it is a title you're not a manager you are currently managing this team exactly exactly um so that that structure um is

56:49 still very transparent um and the well the compensation is an interesting one because given that the nature of our work which we do have the research engineering some of the individuals in those teams or even in the product teams are acceptional and wants to do those more of the individual contributing work of course you you do pay a lot for that caliber of talent so maybe where I'm going with this is standard organization like if you're a manager you are paid usually more as you think about the progression than the individual contributor path and in our case we see there's like some of the people that are I's can be paid more or as much as any of the other people and want to make sure that this is something that people know that if you are a brilliant mind you can have that in them in any part of course it's harder because you need to be truly exceptional but definitely possible I think look at I think it's very interesting I've toyed with it as a thought in the past never as bold a statement as as what you've you've done here and I think I do think the sort of the stage question is is an interesting

57:50 one so it's interesting to evolve I have only I guess the one more immediate sort of challenge I have for you is just as you think if you hearken back to the product versus research type lens on it or what's the center as you build up a bigger sales organization a bigger support organization product organizations it are you hoping I I appreciate the sort of humility of maybe not right for later maybe yes but is your anticipation that those would also fit here or do you think hey this works as long as we as our sort of Center is this type of activity and that's also an interesting one and like one thing that I think is also valuable for us is that what it one we did have a number of people that really wanted us well that hey we don't think titles are useful there's a a good internal support for making the move but then also externally as we think about hiring Talent we want people that will join for for that impact that they can bring to organization which allows us actually to hire a lot more

58:50 people that traditionally are in the senior level some of the organizations because they're really joining to right to execute for the company rather for they don't need a title of it exactly which actually helped and like we have now number of people incredible people that are reaching with that expectation that if they are doing well and rightly they will have high degree of the ownership in the company you did ask the right thing which is of course there's a lot of external roles where the title itself can be helpful as people interact with the clients as they get the sale there are some markets where title is the way of gaining and and understanding the respect and here we we are avoiding this as a so you don't still have a title because then everybody would would have a high title but in a transient exchanges if you think that the title can help you in a specific business activity that you do you are allowed to use it you just we have like roughly of pre-agreed of what is the right what is legitimate but but like good example is Japanese Market where people really need to know who you

59:50 are interacting with where you do use that in in some of the business contexts or if you are out bounding to a client we do potentially allow that we've actually not seen any of the positive of if you have a title it like leads to any better out bounding conversion right yeah um in presentations too if you are saying you're leading a region that's as good enough as I don't know VP or director head of yeah and I think that part I wholeheartedly agree with and we do the same we don't have any sort of VP or director or such we have head of we have lead of a team on it here at Tes uh

Envisioning personalized AI tutors with massive context windows

1:00:22 it was harder to do it 12200 people that sneak it's interesting hey we're really coming on time here just one kind of final question Beyond The Amazing future that 11 Labs I'm sure has ahead of it and you'll build a lot what other aspect of AI excites you the most when you cast your eyes out I think the I I feel like this is a common answer you get but it is extremely exciting of how the LM or the multimodal space will evolve and like how can you have even higher degree of understanding like effectively that infinite context window or like pretty large context window and as I think about education space as a whole and how Ai and education space can come together I think this would be incredible if somebody knew all the knowledge I gained over the last since I was born can be my personal tutor through that through those years be my like matatics voice when I'm solving mathematical problem and then help me on this sales deal that that they haven't known about the process which will be an interesting innovation in terms of can the llm space

1:01:22 or multimodal space carry that that very large context across my life and provide that personalized experience so I think what what companies like on Tropic X openi are doing on the on that side or Google is this is incredible for the world and for what's possible yeah oh exciting to see that as well Ms is amazing I really wish you all the best on it and I really love the insights that you shared over here so huge thanks for coming on to the show thank you guys thanks for having me it was a pleasure speaking together and thanks everybody for tuning in and I hope you join us for the next one you're listening to the AI native Dev brought to you by Tesla