Google DeepMind's Logan Kilpatrick on Why AGI Won't Be a Model

7,727 words · about 31 min to read · YouTube’s automatic captions

0:00 As the amount of software in the world increases, there's going to be lots of problems. There's going to be lots of things that don't work. There's going to be lots of edge cases. And like the difference between those two things is where I think traditional software engineering developers will add a huge amount of value because that gap is going to change. The way to use AI tools today, 3 months ago was different and 3 months before that was different. It's mostly if you want to get the like frontier level productivity gains, you need to sort of continue to evolve. The rules are being rewritten under our feet and you can either sort of like ride the wave or you're not going to get these like frontier level productivity gains.

0:33 A couple of years ago, you asked how long until AGI and Musk replied, "Next year." That didn't happen. Where do you feel like we are today versus a year ago on the path to AGI?

0:45 We're sort of like close to what folks will feel as this like narrow superintelligence. I think it's like actually more close to the way of describing it. Like if you could have a model or a system that could like build anything with code, I think about that as like narrow superintelligence. AGI is not going to be a model. It's going to be a product that somebody creates.

1:05 Who's down for building that then? Is that one company, one individual, or is that more a combination from an ecosystem of products?

1:12 That's a good question. I think

1:24 Back in November, we hosted the first ever in-person AI native dev con in New York. This June 1st and 2nd, we're bringing it to London. It's 2 days built for AI native developers and engineering teams. One day full of hands-on workshops and one day full of practical talks on agent skills, contact engineering, agent orchestration and enablement platforms, and how teams are actually shipping AI in production.

1:49 Join us at the brewery in London near the Barbican for all of that plus networking, parties, giveaways, and a room full of people building the future of AI native development. You can also join us from anywhere in the world by the live stream. As you're listening to this podcast, you get 30% off your ticket with code pod30. Just head to ainativedevcon.io and we'll see you in London.

2:17 Hello and welcome to another episode of the AI native dev. And we have a special episode today. We actually have Logan Kilpatrick from Google DeepMind joining us today. Um Logan, a massive thank you and welcome to the episode. How are you?

2:29 I'm hanging in there. Lots of uh it's another week of chaos in the AI ecosystem. So uh trying to trying to stay on top of it all.

2:37 There have been wildfires all over Twitter and and the social spaces already this week, haven't they? It's been It's been quite the quite the ride. And uh for those listening, it's it's April 1st today. Uh but uh but this is the this is the the the the week in which uh Anthropic uh knowingly or unknowingly open sourced uh Claude code and and and things like that. So lot lots of fun things happening happening in the world right now. Uh Logan, for those of you who don't know Logan, Logan was uh previously a leader of the developer relations team at OpenAI and currently a member of the technical staff at Google DeepMind. So Logan, why don't you tell us a little bit about I guess some of your time at OpenAI and and moving over to Google uh and what your day-to-day looks like?

3:19 Yeah, it's uh there's lots of parallels. I do think the ecosystem has changed so much. I've been at Google um actually today's my 2-year anniversary at Google, which is crazy.

3:30 Yeah, thank you. I joined OpenAI uh end of 2022 a week after ChatGPT had launched and sort of got to launch GPT-4 and ship a bunch of, you know, the chat completions API, the assistants API, plugins, GPTs, a bunch of other stuff across the ecosystem. Um and just seeing how how fast things have changed um and how much progress we've made. I think like my most interesting reaction is like, you know, all the agent stuff 12 months ago, there was like lots of conversations about agents in the developer ecosystem and and more broadly and like to see them actually work now and like people are building agentic products, people are using agents to like do the things that it was sort of like the promise. And it's it's interesting to see this and we should talk more about it because like it was kind of like a joke 12 months ago. It was like, oh, you're doing agents like hahaha, like that doesn't really work.

4:21 It's kind of science fiction. I think today it's like very real. And so I'm always thinking about like what are those next set of things and and so spend a lot of time like with our team thinking about like what are those next set of things that we need to be thinking about that are sort of a little bit science fiction today. My like uh, we have a bunch of these like research to reality uh, monikers internally that we sort of like think about of how we it's our team's responsibility to bridge the research to reality um, because we sit in a research org inside of DeepMind and our job is to build products for for builders and developers. So, um, the day-to-day looks like shipping products and shipping features and building a platform so that we can make that research to reality flywheel happen.

5:01 We'll start off with the today and then we'll kind of like lean into the tomorrow at towards the end of the episode. Um, so so one of the super interesting pieces that have come out of

AI Studio streamlines the path from prompt to production

5:09 Google recently is is AI Studio. Um, and of course many of already familiar with you know, more of the the terminal UI style of using using uh, Gemini and things like this. So, first of all, talk us through a little bit about what AI Studio is and why people would maybe lean into AI Studio more than Gemini and when people would kind of like more lean into the into the terminal versus the UI.

5:33 Yeah, it's a great question. I think one of the one of the beauties and one of the challenges of Google is that like we have a very wide ecosystem. We're sort of touching every part of the developer life cycle. We're touching every part of uh, the different ecosystems across like web and mobile and cloud etc. etc. AI Studio I think the most simple way to frame it is is a platform, a set of things that help you go from prompt to prototype to production and do that really fast using all of our AI infrastructure.

6:02 So historically AI Studio was like a UI playground that you could sort of test all of Google DeepMind's latest models and experiment with them and then actually take them into the API and like go and build products and and tools and apps on top of the Gemini API. And I think what's happened in the last 6 months is we've sort of continued to evolve the platform. We still have that playground functionality so you can test all the latest models on day one and and sort of experiment with them. We also now have an entire vibe coding experience and I think part of this is like it's a natural extension of the work that we were doing with the playground which is like help people understand how to actually get these models into products and like bring their ideas to life. I think the main thing that's also shifted is like there's just way more people now. It's not just developers, it's this sort of like next gen developer builder persona who historically couldn't use these tools who now can and I think it's part of our mission in AI Studio and probably inside of DeepMind is to like bring this technology to as many people as

7:03 possible. So now you can go to ai.studio/build you can like vibe code entire apps that are built on top of you know Firebase which is our sort of like storage solution and parts of Google you can deploy using Cloud Run, you can sort of use Google Search, you can have grounding data with Google Maps and sort of we're exposing all these bits of the Google ecosystem and and lots more cool stuff coming on that front in a way that's just simple. You can click a couple of buttons, you don't need to sign up for eight different accounts and we're sort of like bringing together the ecosystem and building this platform for people to build on top of which has It's really really exciting.

7:41 So lots of lots of cool things. To answer the question specifically, like Gemini the app versus AI Studio and like Gemini CLI versus AI Studio and then even anti-gravity versus AI Studio, um Gemini app is sort of like a personal assistant for your everyday use. Um so, like, you know, I was asking a bunch of medical questions to the Gemini app this morning. Um the Gemini CLI is like a a form factor of the CLI. So, if you're a developer, you use CLIs and sort of that feels native to you. Great product experience as a developer to go and use that. Um and then anti-gravity is sort of this like full-stack IDE agentic developer platform uh where you can actually go and and similar to what you would do inside of like Cursor or Cloud Code, etc., etc., you can build apps. So, if you're a developer, it's sort of your daily driver uh to go and write code in like really complex code bases, etc., etc.

8:33 AI Studio sort of has a batteries-included app builder and then a bunch of like underlying infrastructure that folks can build on top of.

8:40 And do you think that, you know, a a lot of this is very kind of um dependent on how the the the engineer or the developer wants to work? Some people prefer to like terminal UI, some people prefer the more the the the the IDE approach. Um there's a version that people believe where sometimes, you know, vibe coding is maybe for more for non-developers and actually, you know, uh senior engineers don't really need to change the way they work and things like that. Do you feel like we will actually converge into into an an approach whereby if you look at development today, 99% of develop Well, not today, sorry, 5 years ago, 99% of developers were using an IDE to develop.

9:17 Today, that's very much more split. Do you feel like when we get to a stage where agentic development is just the norm, we will have a single typical way where people will develop code? Um or do you still feel like there will be the more technical folks that maybe want to lean into the terminal, other people will maybe more want a visual aspect. Are we in a moment in time or do you think that's going to persist?

9:42 My assumption is that it will persist and part of this is like back to the threat of like personal preference. Like this has always been the case. Like there's been developers pre-AI era who were using IDEs. Like I've I've always been sort of an IDE user myself. I was using, you know, VS Code and whatever the one that was built by GitHub back in the day that there's another they had their own GitHub at one point had their own IDE before like pre-Microsoft acquisition that I I used a bunch that I really liked. And at the same time there's always been like Vim and Emacs and a bunch of these like more terminal CLI centric tools. And I think this is just like a developer comfort thing.

10:18 There's like trade-offs from an ergonomic perspective, there's trade-offs from a productivity perspective. I think that will actually increase with all this stuff. And I think part of this part of the world view of this is like the ability to create new software, the like cost is going down. So I think there'll be like even more exploration. There'll be like even more weird stuff. Like you could imagine like each developer actually has this like very customized experience of like how they like to build software, how their mental model like the way I think about software may be different because we learned in different languages and we sort of, you know, you know, come from a different place or whatever it is. So you can sort of really customize that.

10:56 And I think I think [clears throat] developers will actually probably end up building a lot of this stuff for themselves. I also think there'll be like platforms that have this level of extensibility where like, you know, you could imagine ever in the future every developer has like a fork of VS Code for example and it's like, you know, there's the editor part of that but then there's an agentic part of that as well that they sort of build themselves and customize. That's maybe like an extreme world view but I think there's like the answer is probably something in the middle and I think you'll see a lot of this like developer choice, meet people where they are. All these different ecosystems require different stuff. People's level of like comfort and familiarity with AI and these tooling is also different. So like it warrants a different product experience and it's actually hard to build that like highly extensible customized product experience at least today if you're like one of the teams that are

Vibe coding requires asking for multiple changes at once

11:48 building this product.

11:49 Absolutely. I'd love to kind of go a little bit deeper into the kind of like the the how we can vibe code well. So one thing that you mentioned was if you want 30 things as part of your, you know, vibe coding task, ask for 30 things in the in the in the first prompt. The model is now smart enough to handle that logic. And and that was kind of like a little bit different to the original form where we had to essentially ask for something small and build up because giving too much information, too much context essentially, would would overwhelm that model. Do we need to rethink the way we were kind of originally taught to develop with AI?

12:23 Yeah, 100%. I think it's not um but my my framing of this is like this phenomena is going to continue. So like you need to have this sort of it's it's part of the challenge of this moment and and I fall into this, you know, the the bucket of the folks who have a difficulty with this as well. So I'm not saying that I've figured it out, but like you need to have the mental plasticity to sort of like change. The ecosystem, like the way to use AI tools today, 3 months ago was different and 3 months before that was different. It's mostly if you want to get the like frontier level productivity gains, you need to sort of continue to evolve. And that's hard. It's just like difficult as a human to continue to do that, but like

1 Highlight · Asking for more in prompts unlocks productivity gains

13:03 that is the reality of what needs to happen. And this like ask for 30 things um example is like a very acute way of of uh feeling the difference. Like truly for me, 12 months ago I was like, let me ask for the bare minimum thing possible because otherwise the model will sort of and the agent will fumble over itself, not be able to actually do what I ask. And now I'm like constantly kicking myself to be like, maybe I should ask for three extra things um or four extra things or five extra things or all 30 things that I want and the rate limit is now like how quickly can I ask for things and that's just a very different world to be in and I think that it's it's been like literally in the last 6 months that this shift has happened. I think it what this was not the case a year ago. So yeah, you kind of need to the rules are being rewritten under our feet and you you can either sort of like ride the wave or um yeah, or you're not going to get these like frontier level productivity gains.

13:58 Yeah, another I guess kind of thing that has really shown its face here is context and I think prompt engineering was a thing which we discussed and talked about, you know, a couple of years ago but seems to have died off now in terms of the importance compared to something like context engineering and in particular I guess skills over the last 6 months or so or less skills have kind of like you know, been built up and and used so heavily by people for productivity gains. How much do you use or how much do you hold context and skills as a part of a real efficiency gain in the in your work in in AI studio and in Gemini?

14:38 Yeah, we have a ton of skills that the like when I do like engineering work and use anti-gravity internally to sort of like build AI studio. We have a ton of our engineering teams build like tons of skills which is great for me because I don't know how every I'm not a sort of day-to-day engineer at Google. So I don't know how a lot of the systems work. So they can bring that

Prompt engineering is a bug replaced by dynamic context

2 Highlight · Prompt engineering is a bug; dynamic context is the future

14:55 context into the skill and sort of influence like the architecture decisions and stuff like that which is really helpful. My world view has always been that like prompt engineering was a bug. Like if you go and talk to users they don't want to prompt engineer and actually like the things that you're asking them to add in already exist somewhere else. So like your job as the human using AI systems and sort of like the LLM or AI app 1.0 era was like to do the context engineering. That was the value add and as like what you were providing as the human in that experience was going and finding all these disparate sources, bringing it into a little chat box, and then sending it off to the model so it could do something useful. I think my worldview sort of shifted when deep research came out, which was you could sort of like take this really ill-formed question or idea or hypothesis, and you could sort of send it off to deep research, um and Gemini would sort of like go and browse the web and find all the different data sources and like do this context engineering on the fly to really sort of

15:55 like bring all the information in, and it was a I had this magic aha moment where the deep research UI would like show you all the different like sites it was visiting and all that you'd see like hundreds of sites and thousands of sites even in some cases being visited, and it just like was the epiphany to me that like this is clearly the way that the products are going to end up going, which is people do the thing that humans do, which is like we ask our very like context-thin questions, comments, requests, um and then the system actually goes and does the work in order to go and find that, and you see that now actually with coding tools, which is like I can ask for some change, I'm in a million-line code base um somewhere in like the model will then go like grep all the files and sort of look through and try to find you know, pull in the right pieces of context. I don't need to say like and I think it's sitting in this random HTML file in this folder here, like that is a complete bug, you shouldn't have to do that. Um you should be able to with like pretty minimal context go and do these things, and the model should be able to figure it out. And so I'm I'm very happy

16:56 that on the coding side we've like seen that as the direction of travel, and I think hopefully we'll see that in like a bunch of other domains as well. Um but this is the unlock uh is the model on the fly doing context

17:08 engineering. It's an interesting thread of skills because they're obviously very helpful today. Um I think it will be a similar direction of travel, which is like the models will probably learn how to make skills on the fly and thus um not and actually the the reason why this will end up being the case is because like it's a token saving efficiency thing, which is like if you can get some like really solid skill that has the right context, it just makes it so that the the model can figure a bunch of this stuff out. It is just going to like fumble its way through. It's going to like send 100 requests to the Google Drive API and none of them will work and it'll sort of like self-correct a bunch of times and read 20 web pages and look at 70 examples. It'll figure it out, but like it just takes a lot of time and it wastes your it wastes your tokens.

17:56 Um so skills I think in the short term are like a helpful way to just get around that, but I would expect over time that the model just like pre-writes a bunch of this stuff or pulls from some repository of like domain authority skills, whatever it is, um and then solves this problem so that like humans aren't handcrafting skills in the way that I think they are today.

18:16 Yeah, it's super interesting and I and I think I I I love the I love the fact that you called prompt engineering a bug there and I think that it that's absolutely the right way of thinking about it because there really was that that that disconnect really between human and and and LLM, whereby typically it's the human not asking for something in a way that the LLM really expected, uh you know, to be asked about a certain thing. Context is very interesting cuz I guess a particular with skills because the model will maybe not understand or know exactly how that user or that developer wants something to be presented. There could be the average how the industry works, but it's more how a company maybe its policies, maybe it's in you know, very specific things in a skill. I guess the model can find that out locally, um but I I really like your approach there of saying, you know, it could do all that research, but if the skills are just there, they could be handcrafted once or they could be they could be built by the LLM, but once they're there, you might as well kind of like just have that in some repository and and just pull that as needed. So, I I love that approach.

19:15 I'd I'd love to ask a question actually about about when you talked about in a previous quote where you said software volume's going to be a million times what it is now in 10 years. That's both exciting, amazing, and scary at the same time. What what do you feel that's going to do to the value of development or a developer? How how's that going to change when we have to look after and deal with that much code?

19:45 Yeah, it's a it's an interesting question cuz I think like the landscape is shifting so quickly and so dynamically. I think there's a few things like A, in 10 years like what we consider to be development will likely look significantly different. There will still be things that are similar. Like people will still be sort of like fixing problems with software. I don't think that's going to go away. I think the way in which the tools are wielded sort of like the the scope, the level of detail. I think a lot of those things are sort of like up in the air in my head as far as how it's going to shake out.

Exploding software volume elevates engineering problem-solving

20:17 The interesting thing is um why I'm like relatively bullish for software engineering or just like engineering as a discipline in general is because like as the amount of software in the world increases and as the like number of people who are creating software and having software created on their behalf increases, um there's going to be lots of problems. There's going to be lots of things that don't work. There's going to be lots of edge cases. There's going to be There's always going to be a frontier of like what can the tools do that like the average person wielding them can't do. And like the difference between those two things is where I think engineering sort of like traditional software engineering developers will add a huge amount of value because there will that gap is going to change, but like the gap is also is going to be ever present. Um even if the even if the models and the tools get get really really good. I also think it's like I think people get caught up [snorts] in a lot of the like

21:17 pedagogy of like this this conversation, which is like what I think about software engineering, I think about like a a way of solving problems, a way of thinking about the world, a way of like um yeah, a a way of approaching problems and I don't think about it like I'm not at least personally myself like attached to this like I type keys and characters show up on the screen and then those things represent, you know, some formal structured programming language like Python or JavaScript or whatever. Um I think it's like more general and actually if you look at like computer science education like in a lot of cases there's like there's a there's a difference in the way that it's taught different places, but like there's lots of computer science education which is like what I'm talking about which is just like very problem-solving. It's a way of thinking about the world. Um AI doesn't minimize the value of that. I think it actually like accelerates the value of it and that value of like the way of thinking about the world,

22:18 the [clears throat] way of problem-solving I think is going to be like super super valuable. Again, I think the value of like typing a typing keys uh that make characters render is probably going to go down though I still think there'll be reasons to do that and there'll be value of it. Um but this like way of solving problems I think is going to persist and like I'm grateful that I like spent the time to think about those things and like it manifests in the way that I build stuff today.

22:44 Um yeah.

22:46 Yeah, do you think there's a mechanical sympathy there essentially in terms of us understanding how things have been built to allow us to actually architect and and and make applications you know, more reliable, more more robust going forward?

22:59 100%. Yeah, yeah. I think you you someone needs to have the level of depth to understand all these things. Um I think it's like does everyone need that level of depth? Probably not. But like you want to have like experts to go in uh go deep in these different areas and like think really deeply about the systems and know the right know the right questions to ask. Also a lot of it is like

23:20 [clears throat]

23:20 as the as the means to build software is like dramatically increased like lots of life and lots of these like technical decisions don't have a real answer. They have somebody who has a strong opinion. Um and so I think you still need somebody to like have the opinion based on their own experiences or their sense of the direction that they're that you want to go to like make a bunch of these technical questions. Like there's often not like a single right technical direction. There's many possible technical directions and it's like based on people's own lived experiences and intuition and understanding of technical constraints they make a bunch of decisions. Um so I think the same thing will all of that will continue on in the future.

23:59 Interesting. Let's continue looking into the future a little bit and talk a little bit about uh the wonderful topic of AGI. Um now a couple of years ago um you asked uh how this is on Twitter on X, how long until AGI? And uh and Musk replied, "Next year." That didn't happen. And I actually I really like your kind of your your underneath the sitting underneath the hype approach to this. And you said, "Someone is going to weave together the right components at the product level with a model that's really smart and people going to call that AGI." And I and I and I think that's really interesting because obviously the AGI timeline predictions that you know many have kind of thrown around haven't really aged that well. So I guess where do you feel like first of all we are today versus a year ago on the on the path to AGI?

24:50 I think this conversation has gotten even more complex over time cuz I think when a A of these conversations started a few years ago and like even like 10 years ago like you didn't have tools that could do any of these things. Um it was sort of like very it was very like academic, very philosophical and I think a lot of the definitions and sort of like preconceived notions are grounded in that initial stage where there wasn't like we didn't have tools that had any of these capabilities. Um and or like the most advanced version was like, you know, AI playing games um but like it didn't generalize to a bunch of other things yet. I think the the like product adoption and the tooling and the way that this is impacting our lives has changed so dramatically in the last 3 years that I've almost like decoupled myself from some of these like AGI conversations because uh I think it's important that somebody has these conversations and like thinks about it and and sort of like has this uh more sort of like academically

25:51 rigorous point of view. I don't have that point of view. I think like my world view is very grounded in I think

AGI will emerge as an integrated product experience

3 Highlight · AGI will be experienced as a product, not a single model

25:58 how the world how like the average person is going to interact with this stuff. If you took the technology we had today and you brought it 3 years ago, they'd be like, "Holy crap, you know, that's the future. That is, you know, these systems are so smart and can do everything." So like in some sense the goal post keeps moving. Um in some sense I think we are like, you know, AI coding is an example of like so much value being created all these like research tools, all these other things like it feels like we're sort of like close to what folks will feel as this like narrow um this like narrow superintelligence. I think is like actually more close to the way of describing it. Like if you could have a model or a system that could like build anything with code, I think about that as like narrow superintelligence.

26:50 Humans can't compete on the same level in that respect. Um I think from like the like academically rigorous point of view, this like AGI question is grounded in whether or not the models have the generality. And so I think they still don't have generality. Like, you know, at the same time that I can basically build anything I want with software today using AI coding tools, I can trip the models up with all these like goofy things that humans are able to really easily do. And so I think the academically rigorous argument is like, well, we don't have this general intelligence um until we can not trip until the models don't get tripped up with these things. Like, you know, it's pretty easy to beat the models at poker, chess, or any of these things that like humans can be relatively good at.

27:33 My worldview is like for the average person it doesn't matter. Um for the average person these tools are super impactful. They're like already creating tons of economic value. Um I think like this like system from an AGI perspective is my worldview of like AGI is not going to be a model, it's going to be a a product that somebody creates.

27:53 Um and I think that's already the like I think folks would probably agree with this worldview, even those who have this like very academically rigorous point of view, just given the way that the models are, you know, now to do coding as an example, you need sort of an agent harness, and you need a a product, you know, whether it's a CLI or an IDE to like bring the model to life. You need all these things. So like, I think this worldview is like um likely tracking the progress that we're making. My last quick comment is I think it's a realization that like there's this capability overhang. And so I think that's part of my worldview of like why we'll see like it's likely that it won't be like, and this model lab just dropped a new model, and then everyone thinks it's AGI. I think it's going to be that like there was a model that came out 3 months ago, some really smart product and engineering team found a way to like do something really interesting with it, and put it into a system that I think people generally think is AGI because there's this huge capability overhang that exists today.

28:49 And that resonates so strongly with me because if you think about what's actually unlocked you know, development today with AI, it's not a model that has you know, got significantly better than any other model. It's It's the fact that agents has been has been layered on top of that and it's that interaction between agents and and well well as an LLM itself and agent and a and a and a back-end LLM that has made agentic development so much power more powerful. I think that when you say it's AGI will probably feel like a product release. It's that environmental how you actually build your system that has a various products in versus a a single LLM that that kind of like says, "Okay, ask me anything you want. I'm AGI." It's It's For me that I That does make sense what you're saying there as a as a product or a a feeling really with a set of products in an environment makes makes a ton more sense.

29:43 If AGI again does arrive kind of more as a product experience than a single model, who's who's down for building that then? Is that Is that one company, one individual or is that kind of more a combination from an ecosystem of products?

29:59 Yeah, it's a good question. I think this is one of the interesting threads about like, you know, I'm grateful to be at Google because I think we have product distribution across all these things um across like so many different verticals and and places that uh you might expect need to be good if we really were to have this like general level of intelligence. Um So, yeah, I don't know. It'll It'll be interesting to see. I don't know if I have the answer. Um but I would expect like your you know, the Gemini app as like a as like a personal assistant sort of like to help you in your everyday life. I think should if it's like artificially uh generally intelligent, um it should be able to like dispatch and work with all these different tools and ecosystems to like help me complete any task that I want. Um, and I think we're like, you know, we're trending in that direction with tool use and all these other things uh that are like coming to the to the app. And I think that that paradigm will continue as you'll likely have this like

31:01 um again that product experience where like one system maybe like relies on many other systems. Um, but I think it will be very I think I think it's under appreciated probably the level of breadth and depth and the like complexity of solving this like artificial general intelligence thing. Like I would be very surprised if it's like we end up with this just like chat UI where you go in and you just ask your question and everyone's like this is AGI cuz it can do anything. I think it's going to probably be this like very orchestrated like lots of different stuff. Um, a ton of UI complexity and all these different things. I I think it's going to be like much more verbose than I think this like simple AGI idea that that um that folks have to That's at least my personal feeling of of how things will shake out.

31:50 Hey everyone. Hope you're enjoying the episode so far. Our team is working really hard behind the scenes to bring you the best guests so we can have the most informative conversations about agentic development. Whether that's talking about the latest tools, the most efficient workflows, or defining best practices. But for whatever reason, many of you have yet to subscribe to the channel. If you're enjoying the podcast and want us to continue to bring you the very best content, please do us a favor and hit that subscribe button. It really does make a difference

32:19 and lets us continue to improve the quality of our guests and build an even better product for you.

32:25 All right, back to the episode. Cool. Logan, this has been awesome. I want to jump into some quick fire questions to to to wrap up this episode. Um, first of all, and honestly Luke Logan, OpenAI or Google, where did you have more fun?

32:39 I'm having more fun at Google these days. Um I think it's uh yeah, it's an incredible place to to build products. So.

32:46 Awesome. When you were at Open AI, what was one thing Sam Altman got right that no one really gives him credit for?

32:52 I think people give him Sam a lot of credit for a lot of things. Um I think he was right about the level of compute. I think Sam was like maniacally focused on making sure that they were resource resourced for the level of like AI consumption. I think he was he was right to make that the focus even like you know, basically the first meeting I was ever in with Sam, he was talking about that. So, um he was definitely right about it.

33:15 Amazing. There are a lot of big labs out there now all jostling for position. If you had to bet on one lab not existing in 5 years, which one would it be?

33:25 That's a good question. I think maybe the the bet would more so be that like I think the way that we see them as labs today, I think will be different. So, I think like one of these labs will likely evolve into something that like probably doesn't look like the like lab of today's era. Uh but I think all of them have my sense is like there'll be many winners in the ecosystem, but I think people will like pivot into different and there's like a good actually I don't know if the social media analogy is perfect, but like a lot of the early social media products looked very similar and over time they ended up like it's very clear to most people that like Snapchat's a completely different business and product than Instagram than you know, X and Twitter is etc. etc. So, I think we end up in that kind of setup um a little bit.

34:09 Awesome. What would you say is most overrated today? Agents, rag, or prompt engineering?

34:15 I think people have moved on from rag and prompt engineering. Um so, they they perhaps were overrated before, but I feel like they're probably adequately rated now which is that people don't put a lot of stock in them. Obviously still important both from like a conceptual point of view, but um I think the frontier has moved on and people are focused on other things.

34:33 Sounds good. What is the most overhyped AI benchmark today?

34:37 I like SweetBench a lot. Um, I think that my my only comment on this is like and there's better versions now and actually the folks who built SweetBench have done a great job of like continuing the a bunch of the different versions of SweetBench, but some of the original versions of SweetBench is like completely out of distribution of like how I think most people do development work. It's like tested. I think there's it's like 40 this could be it's like I'm off on this. I don't remember the specific numbers, but it's like 40% of the original SweetBench is like the model setting up Django. Um, which is like fine and it's great that it can measure that, but like probably out of distribution from like how developers spend 40% of their time.

35:15 Um, so they've done a great job of continuing to evolve. So I think maybe not they're overhyped, but uh, they've done a great job of like meeting the moment with some of the new SweetBench stuff that they've that they've done.

35:25 Yeah. In 10 years time, will there be more or fewer software developers than today?

35:31 I think the absolute number will probably be very similar and I think they'll be like new role profiles where they'll be like maybe 10 to 100 x more people who are touching code on a daily basis. Um, they're probably also just like this like member of the technical staff profile is a good example of this. They're probably just doing other things. So absolute number of developers same number of people touching code, I would expect to be like dramatically more um, than what it is today.

35:58 What would you say there are a lot of amazing research papers over the last few years. What would you say is the most important AI paper of the last couple of years?

36:07 It's probably scaling laws or the original transformer paper just sort of as um, yeah, setting up this this industry that we're in today and those two things have really held true and had a huge impact.

36:21 Which AI company would you say is most underestimated right now?

36:26 I think people underestimate DeepMind actually. Um, I don't I think they're you know, I'm the I every 6 months I think to myself is Google the best place in the world and to be doing this work and every 6 months the answer is the same which it is. I think there's just like so many great things about being at Google and um the the like talent and the way sort of like Demis runs DeepMind I think is all deeply resonates with me. So um it's it's hard that Google's a big ship but when you steer it in the right direction things go really well. So I'm I'm excited.

37:03 Amazing. And the last question Logan, what would what's the one thing you wish developers would stop asking you?

37:09 I wish they would stop asking you know, a rate rate limiting stuff and and I think there's like a there's a threat of this is not because I feel I've a lot of empathy for folks asking this question. I think it is and we're doing a lot on the product side both on AI studio and across Google and other products to like solve this problem like the future version of the world that I want is like abundant compute developers can just do the things that they want they're not worried about rate limits and they're not worried about quota stuff. I think we need to do a bunch of stuff from a product perspective to make that possible but so I don't hope I selfishly hope they stop asking because we solve this problem not because I'm annoyed by them asking.

37:48 That sounds amazing. That's a that's a future we can definitely live in. Logan this has been absolutely great fun. Thank you so so much taking the time out to to speak with us and I appreciate it.

37:57 Thank you for all the thoughtful questions. This was this was a ton of fun.

38:00 Awesome. Thanks everyone for listening in. Tune in to the next episode soon. Bye for now.