No More Screenshots. Web MCP Lets Agents Talk to Your Website Directly

10,546 words · about 42 min to read · YouTube’s automatic captions

0:00 We need to stop thinking on LLMs as the only solution. Apple released an open source model to do some image detection. It's a model of 200 megabytes and works client side. So, now you can have an OCR system that can detect objects completely local. It opens a lot opportunities for a lot of apps and web apps that can run even without internet, just offline, and you don't need LLMs.

0:25 Tell us a little bit about your recent work and focus on Web MCP, like what it is and what problem it's trying to solve.

0:32 Right now, ChatGPT agent mode or Claude agent mode, okay, they're browsing the website typically in just taking a screenshots and then saying, "Oh, I'm analyzing with an image model." But then they trigger the click on that portion of the screen. But what happens if in those 5 to 10 seconds the JavaScript moved that button? That is inefficient both in time and in cost. And that's where Web MCP appears as a solution. Web MCP is kind of an API. It's just JavaScript functions that you expose to the agent. Instead of browsing the website as a human, it will just execute those functions.

1:22 Back in November, we hosted the first ever in-person AI Native Dev Con in New York. This June 1st and 2nd, we're bringing it to London. It's 2 days built for AI Native developers and engineering teams. One day full of hands-on workshops and one day full of practical talks on agent skills, context engineering, agent orchestration and enablement platforms, and how teams are actually shipping AI in production.

1:47 Join us at The Brewery in London near the Barbican for all of that plus

1:51 networking, parties, giveaways, and a room full of people building the future of AI Native development. You can also join us from anywhere in the world via the live stream. As you're listening to this podcast, you get 30% off your ticket with code pod30. Just head to ainativedevcon.io and we'll see you in London. Hello, everyone. Welcome back to the AI Native Dev. Uh today we will be jumping into the world of the web, which uh is is actually a fun space because my background in sort of the entrepreneurial world has been in um making websites faster, especially with the rise of the mobile web. And that's where I got to meet Max, who is kind of been building up uh you know, a ton in the web at the time, was optimizing for mobile performance, for responsive websites, you know, for many aspects of the the rise of the mobile web, and today is in AI world. So, Max, thanks for coming on to the show here. I'm a pleasure. Thank you. I can kind of wax poetic a little bit about your background. Probably best to for you to

2:52 describe the journey, uh but it's so tell us a little bit of just for people to have the the mindset of where is your information coming from, a bit about your uh your experience. Yeah, sure. So, I've been a a web developer for 30 years. So, that's a lot. 30 years now. So, I have started doing websites in 1995, actually.

3:13 Yeah, I was actually So, my first website was created in the edit of DOS. So, we are talking in text mode. And then I was getting into Windows 3.1, opening Netscape, and opening manually the HTML files to see how it looks like. So, that was the the idea at the time. And yeah, in my journey I I So, I've seen everything. So, from all the different design patterns, all the different languages, like ASP, PHP, then then the the move to the front, the the web 2.0 stuff, HTML5.

3:49 And on on mobiles, I'm also a mobile developer, so native native iOS, native Android. I did Objective-C, Kotlin then or Java for Android, then Kotlin. Uh and I I was like the like the the the merge of the two worlds, the mobile and the web. So, I I authored a couple of books on the mobile web, then moving into the performance side of that. So, I I authored a couple of books on on the mobile web as well.

4:16 So, I have books on JavaScript. So, I actually I'm I'm authoring right now, even right now in the AI era, uh like an old-fashioned book. So, then that's going to be my 15th book. And which are you using AI in uh in the writing of this book? Actually, no. And that's part of the part of the deal. So, I'm writing a book on vanilla web. So, how to to write websites without the libraries that actually is useful on the AI world because it's better for context and and things like that. So, it's better for agentic AI tools to code some web apps without a lot of dependencies. So, that's kind of my journey. And of course, as every developer, because I'm a developer, I'm in AI era right now.

5:00 So, there is no other way, no other option. So, that's why I've started doing even some courses on on working with ChatGPT 3 years ago. So, as soon as the API was released from Open AI, I started doing articles and content and courses on how to integrate your websites, your apps with uh Open AI APIs, how to do prompt engineering for for your apps, things like that. And as soon as um also ChatGPT, because I'm telling I'm saying ChatGPT because it was the first one.

5:34 When they created the first browsing plugin, I started doing a research on how uh is that working. So, how is ChatGPT rendering your website? How it how does work? So, if you want to optimize your content for that, how to do that? So, I started writing some articles on that. And technology is evolving every day, actually. You We know that. So, there is a fever right now on that. So, I'm trying to keep updated on every every new technology, design pattern, tool uh that will help developers to to merge into this AI world. Yeah. No, amazing. And I yeah, it is, you know, fever is a good is a good word for it. There's definitely, you know, some intensity around the pace of change of it that is exhilarating and daunting at the uh at the same time. So, um what triggered this conversation is your recent work and focus on Web MCP.

6:31 Uh and I found that super interesting as I as I think about your the problem at a high level resonated with me that uh it felt like agents keep needing to reverse engineer uh uh a web page and they arrive there and they, you know, they try to kind of find the signals on it. They're doing it impressively well, uh but it still feels somewhat inefficient. So, it really kind of caught my attention when I when I saw the the title and I think you had some a

Web MCP replaces fragile agent screenshots with direct JS functions

1 Highlight · Why Web MCP beats agent screenshots

6:57 big tweet that got some uh some big uh big reach around it. So, maybe let's start the conversation by just digging into Web MCP. Tell us a little bit about first of all, like what it is and what problem it's trying to solve. Okay, yeah, sure. So, Web MCP is first is an experimental API that is still not stable. It's uh it was presented by the Chrome team. So, it's right now working in Google Chrome under a flag.

7:22 And the idea is to offer AI agents another way to use the web app services. Because right now, when when I'm when we're talking about AI agents, we are talking about ChatGPT agent mode or uh Claude agent mode, okay, or Gemini. But also we are talking about Open Claw that is maybe browsing the website and also agentic browsers.

7:48 So, that's ChatGPT Atlas, uh Perplexity browser, even now Google Chrome or Microsoft Edge. They have an AI mode or an agent mode that they can browse the website for you. But right now, they're browsing the website typically in two ways. The most common way is just taking a screenshots and then saying, "Oh, I'm analyzing with an image model, analyzing the image, and saying, 'Oh, I need to click here in those coordinates.'" And then that takes time, couple of seconds.

8:20 But then they trigger the click on that portion of the screen. But what happens if in those 5 to 10 seconds the JavaScript moved that button? Okay? So, it doesn't work. So, then another screenshot is taken and then the agent says, "Oh, maybe the website was scrolled or the content changed or whatever." So, that is inefficient both in time and in cost because that's in in tokens, okay? We need to pay for that. How uh like why

8:49 are they doing images versus understanding the web page itself? Well, the other the other way to do that, that's the most common way. The other way is to kind of analyzing the DOM. Okay? Um the thing is that the DOM, you know, we we've been mostly after the React world, the when when you look at the DOM that we are shipping to the user, it's not semantic.

9:15 So, it's just a a list of 100 divs. Divs is just a generic tag that you have on a website that has no semantics. So, actually understanding what's there with just the DOM might not be useful on every on every website. Right. So, on for example, on mobile apps, on native mobile apps, for example, if you want an agent to use an iOS application in your iPhone, so, you can also do screenshots, but also they have some kind of an accessibility tree that is actually really useful because it's the same tree that accessibility tools such as a screen readers are using to understanding your app. So, I think that's um that's that's pretty cool. And on the web you can do something like that, but it seems like uh on some test, the the results are not really good. So, that's why most of them are still using the old way of taking a screenshot. Okay?

10:15 So, which works, but it's com- completely efficient. And I think that makes sense, I guess when you when you describe it like if you try to build an application today, it's so dynamic, you know, it is uh you know, sort of changes all the time on it, but it's also really hard to decipher. Like even as an as an expert developer opening up that DOM, figuring out what's what without looking at the visual is actually quite hard.

10:38 Well, it's optimized for a human brain actually, right? So, it's optimized for for that. So, that's kind that kind of

Web MCP functions execute locally and bridge client architecture

10:43 the agent is still um it can do that, but on a screenshot. And that's where WebMCP appears as a solution. The idea is that WebMCP is kind of an API. So, the the web app developer will expose an API to the agent. Saying, "Hey agent, if you want to like talk with to me to my website, here you have a couple of services that I can provide." So, you register a list of services or tools that you make available to the agent.

11:15 And it's just JavaScript functions that you expose to the agent. So, then the agent when when the agent is browsing a WebMCP capable website it will check all the services available and it will try to see if one of those services is uh suitable for the goal that it has. And instead of browsing the website as a human with the eye and the hand touching on the screen, it will just execute those functions. In this context, the the service provider of these functions is the web page. It's not necessarily the thing behind the screen. So, services can be anything that can that local execution can satisfy. You know, things like click a button, give me some piece of content on the page. Like what are examples of services that that you see people starting to build? The service executes on the client, so in in JavaScript, so on the web page. But that service can also use normal web APIs to connect to the cloud, to connect to a hardware using Bluetooth or whatever. So,

12:17 actually, the simplest example is an airline website. So, you get into an airline website and you are searching for flights. So, you need you need to go to the one text field that says uh your origin, then your destination, then you need to open the calendar or type the date. And you know that every airline is different, so the calendar is different. So, the agent has to understand how the calendar works. Well, in instead of that you offer just a tool, a function that is search flights. You uh specify the schema of the input data. So, you're going to specify to the agent, "Hey, I need origin, destination, and two dates."

13:00 And and if it's one way or round trip whatever you want to define there, and a function. And and you actually pass a JavaScript function to the WebMCP API and say, "When the agent wants to execute this, this is the function." On that function, you can do local stuff or you can go to your own server, your own back end, or whatever you are used to do on on the website. Actually, you can share the functions with your uh standard UI.

13:30 There is a flag that you can check if if currently an agent executing this or a user executing this in case you want to do things differently, but you can you can um take advantage of your current architecture and just call that. And you will return asynchronously. So, that's why the agent will wait for you. And in in that wait, you can go to the cloud, you can go to a hardware, a sensor, or or even you can ask the user.

13:56 So, there are some some for example, if you are uh ordering or you are confirming something that will have a cost you as a web app developer, you can say, "Hey agent, you know what? I would need to ask the user." So, the API includes a way for you to interrupt the agent and call the user. Yeah, you don't you you you tell the agent that it should ask the user or do you literally uh No, you you you ask the browser. You ask the browser and the browser will say, "Agent, you know what? You will wait because I will ask the user." So, in that it can be a like um a dialogue on the screen asking the user, "Hey, do you confirm to actually buy this this flight?" And if the user say yes, then you go back to the agent and and confirm the operation.

14:41 And this is relevant I guess when the user is actually actively behind the browser. So, like these agentic browsers for instance, it would work for for them. But if I asked ChatGPT or Claude some question and it went on and it uh the stuff, then the browser is not interacting with me as a as a user, right? Yeah, I guess that the standard will still need to figure that out how So, how that's going to work with uh It is important WebMCP as it is right now, remember experimental feature, works only on visible browsers. Okay? So, it's not working on headless browsers.

15:14 Got it. Okay, so that's an important distinction for now. It's not the intent of the protocol, it's just the original kind of the initial implementation of it. Focus on that.

15:23 And so, how So, if it's the server-side actions, um how is this different than having an API? Like why wouldn't I just have a you know, an open API spec, you know, that I make available and properly link from my side and have the agent just programmatically interact with my system? Yeah, well, that's actually a very good reason. A very good question because even the same question happens on the MCP world without the web. So, MCP is another protocol that we use to connect AI tools with servers.

15:51 True. And right now, even Open Claw or other agents among uh Claw Co-work for example, they're I mean, you can connect MCP tools, but they can just go and execute uh CLI tools in in the in the terminal. So, they don't need MCP. So, something like that happens here on the web. So, um if you have a RESTful API for example or another kind of API or an MCP API or a CLI, then you don't need to use your website. Yeah, that's and that's um that's a fair question. I think it has to do with with the kind of project that you have for example. Some tools need authentication.

16:31 Uh they have that they have a lot of uh maybe local database that you have on the on the client. So, that tool works actually better in the browser's context and from for security from security purposes. So, you don't want to open the API to the to the outside. So, it's kind of a different way to do that. But it's important to understand that WebMCP is not working automatically for your website. So, you need to actually go to your JavaScript code and implement the API. So, it's not going to be applied automatically. Yeah, and and and I think that sense it is the same as building an API. But I I get that.

17:08 There's increasingly functionality. That's why I perceived it initially as uh something in which the service provider is the page. The API call, the WebMCP call, the recipient of it is a JavaScript function, right? That is um there's code on the on the page and then it can do whatever wonders. So, at the very least when you have rich functionality on the page, and as you pointed out, you don't want the agents to just kind of do the heavy lifting of uncovering it within the system, but you want to give them an agent-friendly programmatic interface to it to your client side. So, at the very least for that, this is this is novel. I guess you could have you could have hoped the agent understands this with some sort of LLM sticks to whatever it is, but this is a standardized way to uh to do that. Um and then there's some overlap between for server-side activities, there's some overlap between that and uh and an open API. Yeah, there is.

17:59 Definitely. The but for example, one one quick example of why WebMCP might be better than than server-side only. Uh Apple Pay or Android Pay. So, if you want to finish the the payment process and you want to use Apple Pay it has to do on the client. So, that works on the client only. So, you're not sending the credit card details over over over the like a RESTful API.

18:25 So, um that's one example on on why you want to take advantage of the client-side architecture and you want the agent to somehow get into that architecture. But that's that's one example. So, that's why typically you will see shopping carts as examples of how to where to implement WebMCP. So, uh the shopping cart lives in the client, not in the server. So, then you uh you can just work with that client side and and you make an order and that that making an order is actually the one that is going to the server anyway. So, the question of why is the agent not going directly to the server and making the order is actually a fair question.

19:07 And I think right now it has to do with more uh following the standard UX flow. And instead of just designing a web app that is 100% optimized for for an agent. Maybe in the future we will be creating this system for agents only. So, then it would be a completely different architecture.

19:33 I think so so very interesting. And you were saying that in the initial iteration of it, it is focused on on visible browsers. And you're right that that introduces also that sort of user interaction. So, some of it is client side, some of it is user interaction, and then you know, some of it might be convenience or sort of you know, like a preference that you want to prefer. You You want to provide this via Web MCP versus Open API. I I'm curious about the path or the jump from there to headless browser. I mean, per your starting point description, you know, like headless browsers also do this, you know, as unpleasantry of sort of like trying to grab an image and reverse engineer in a variety of ways a webpage. And so, they probably benefit from even access to content and things like that on it.

20:21 What what is your expectation in terms of the primary use cases for a headless browser? You know, I guess I don't I don't know what what's your guess on when does that happen? Most of these agents are using Playwright or Puppeteer. So, these tools that were originally created for automation. So, browser automation. So, actually they are creating a headless browser executing your your web up there. So, ChatGPT initially was just using a tool that was converting the HTML into markdown. So, it wasn't even executing JavaScript. So, the initial ChatGPT plugin was really simple.

21:02 And to be honest, if you're browsing if you're asking ChatGPT or Claude or Gemini to to read a website and and summarize this website and you give the URL, on most situations, they're still doing that. They're just downloading a markdown version of your HTML. So, if you're if your website is is 100% client-side rendering, it's not going to work. So, it will say I can't read this website, right? But if you're using an agent, that's like a next step, the agent is typically assuming the user's role. And in that case, it's using some tools like right now Playwright, Puppeteer are the most common one to try to act as a user. So, in terms of expectation, I think that we need I'm not sure if the Google Chrome team will think too much about that. So, they are targeting how to increase the Chrome agent mode. So, they are working on that API. So, we need more companies, maybe

22:05 OpenAI, maybe now OpenAI that is uh is the sponsor of Open Claude. We need them on board of the W3C to discuss the API on for those agents. I'm not seeing them doing that right now. They were not part of the web community, so probably something new for them. But Anthropic that created the MCP protocol may be also interested in getting into Web MCP.

22:32 I'm not seeing that yet that's happening yet. Yeah, quite yet, but I guess that will happen at some point so in the next few weeks or month month and that will trigger some updates in this API. I think that makes sense and I think Anthropic is actually an interesting example because they are actually not that strong around the image analysis as compared to some of their competitors.

22:57 So, it's actually kind of interesting for them to maybe bias in favor of do it do it correctly. I guess as you So, you mentioned MCP as well. I guess what what what's the trigger behind the Web MCP name? What is similar and what is different between that and the the non-Web MCP? Well, I think that what is similar is a concept. If you look at the technical details, it's it's going to be two different protocols. So, yeah, MCP works with with JSON in and out.

23:28 It's kind of So, the not Web MCP. The standard MCP for AI agents, yeah. They work over different They are different versions and different protocols that they work originally. They were working on HTTP, but using a using a technique that was really old that they brought about technique in an ancient book 30 years ago. And now they were using that technique because they were not supporting sockets, web sockets. So, it was kind of a long polling technique over HTTP.

24:00 But and also they have a socket version for like a binary socket for local server and clients. So, when you have a local thing, you can just talk with sockets, but it's a completely different idea. And you expose but the similar concept is that you expose your services to an LLM. And by the way, exposing the services is just typing a name of the service and then a description in English.

24:28 And the LLM will understand that description. And and the schema that you want. And

2 Highlight · Implementing Web MCP with simple JavaScript

24:34 that's a formal schema. So, I really need four arguments. The first one is an integer, the second is a string, blah blah blah. Well, the same idea is here, but you cannot you cannot just export your MCP into Web MCP. You need to write it from a scratch because it's a it's a completely different architecture. This is JavaScript. Web MCP is JavaScript based. It's actually pretty simple from a coding point of view. You just say navigator.modelcontext.registertool and you pass three arguments. The name, the description in English, like a long description, and a JavaScript function.

25:11 And and that's all. Yeah. So, the tool that the agent will actually query on all those tools. It will read descriptions and will say, "Oh, I need to change a flight reservation." Okay, it will look into all your services to see if any description matches that behavior, that tool. And it will say, "Oh, this one." And it will execute that function passing the arguments that you requested as the input schema.

25:40 Yeah. That's roughly how that works. And your your API will return some data. It can be a boolean, it can be an integer, it can be a message, an object. And that object goes in JSON back to the agent that will process that with the LLM. Yeah. It actually I think simplicity is It's great to hear that it's simple and I think it's actually probably one of the things that helped MCP also get adoption because it creates something that is very streamlined and slightly capitalizes on the fact that you can communicate all sorts of complexity and things like that in that natural language line and rely on the fact that you have a consumer there. And the same goes for the response, right?

26:21 You can You can reply the whole complexity of what is it that you are returning can be simplified because of that. In

26:29 that sense, it is a lower lift than any even REST APIs which were already in turn a simplification. But as compared to that, it is still simple. So, it just lowers the barrier and just allows allows more of these types of interfaces to be created. Yeah, sure. Also, there is even a simpler version because you can also use the character version and in that case it's just HTML. So, you don't even need to write JavaScript. If you have forms in your HTML, you add some attributes in the form and you're specifying to the agent, "Hey, if you want to trigger this action, this is the form

Declarative HTML forms and agent detection events

27:07 and here you have the fields that you need to fill for for triggering this action." So, Web MCP also works in HTML without JavaScript. And something interesting is that it's also adding some ideas that we don't have right now. For example, as a web app developer, there is no way to know if your website is currently being managed or used by an agent. So, there is no way. So, if you want to detect for whatever reason, even for for analytics, you want to detect that it's actually an agent using your website, there is no way to actually do that. So, on X, Twitter, they they changed something now to stop agents for publishing content on on Twitter.

27:51 And they say that they are actually verifying if you're actually if there is an actual finger touching the screen, which on the web, I mean, you cannot actually do that. I mean, you can check with the touch events if there is a touch Yeah, but basically with any any tool like Puppeteer or Playwright, you are you can emulate the touch and the website will never know it's it's an agent. Well, now with Web MCP, we have new events that you can listen in your website over the window object. This is the JavaScript technical stuff, but there is tool activated and tool cancel. That means that the tool or the agent is it's it's named tool within the API is actually controlling the website. So, then you can change the UI, you can you can There is also some CSS pseudo classes that will let you change the use the the user interface a little bit when the agent is in charge.

28:52 So, then the if the user is actually seeing the screen, you can tell the user that the agent is in charge despite the browser UI that is also making that that that notice somehow. Right. Yeah, I think So, it's interesting to think about the drivers for adoption I guess of people like why would they bother writing this?

29:18 And and I think that was one like maybe I'll go on a tangent a bit and then come back to that question, which is there is a broader trend in the world of AI of unifying things that we used to split apart. And so, the separation of the API from your webpage is is something that in many aspects is good architecture, right? Like it is the right way to do it so that the two are not coupled and you build those. And in in in a sense, here you're kind of building an API into your webpage or you're intentionally violating it, but in in agent land, when you start creating multiple uh entities that the agent has to orchestrate within them, uh it gets confused. And so, for instance, there's been a big rise of mono repos, right?

29:57 And and pulling everything into the repo because then, you know, you can just have the agent load the repo, and it's all within the reach. And so, it's interesting to think about like one of the advantages of a web MCP is that it puts the API, or it kind of puts the actions within the web page, and it's a more contained unit, and therefore easier for the for the agent to to uh interact with, right? But coming back a bit to my uh my uh my question there was like, what would you say are the are the top motivations, and I think you've alluded to a few, for someone to to embrace this? Like, if I'm Who who would be the first cohort to come in when this is no longer experimental, uh which this is AI land, so maybe that happens faster than we uh than we think. Um uh who would be the the What are the strongest motivations to to put in the effort? I think that the strongest motivation will be uh will appear on e-commerce websites.

30:58 Wherever e-commerce website you have, I mean, you want to sell products or services, you actually don't care if it's a human or an agent. So, you want the money from the consumer. And so, that's I mean, that's a motivation to actually offer as many tools as possible as fast as possible so they can actually purchase your products or services quickly. So, I think that's that's uh that's the first one. When you look at other kind of content, I don't know, you have a blog, you have a newspaper.

31:27 Actually, I'm seeing the opposite sometimes. And so, I'm seeing authors actually rejecting agents. So, I don't want the agent to come into my website. Yeah, more about IP than content.

31:39 Yeah, because of IP or because they're they they user will not get into my website, they will just get the content, and I mean, the authority is get gets out, so there is no credit of that or not enough credit or whatever. I mean, you can still apply web MCP there, but I don't think it would make a lot of sense.

32:01 Yeah. So, in a blog or in a newspaper, but um if you have uh anything that has to do with uh service support, so if you have a problem with I don't know, your your phone bill, uh then your phone company, your agent, if you're asking your agent to go and try to solve the problem, I mean, all that working with support tickets and all that stuff can be also another uh pretty useful use case. E- Either way, it sounds like it and by definition, you know, by design, it is oriented at interactive pages. And so, it will pages that are that are predominantly content, you know, beyond their own agenda and sort of like or dislike of agents, uh it's also probably a little bit less useful for them. But if you if you look at if you look at the API, when when they have when you use the declarative API, it works on forms.

32:51 So, if you have if your website have forms, then it's it can be a target of web MCP. If you don't have any form in your website, it seems like you don't have a use case. Yeah, there's no no real need. Uh what about the web testing? So, we know the the other the other uh entity that suffers from the ever-changing visuals of uh of of websites is the kind of pursuit of end-to-end testing uh that doesn't require you to rewrite your tests uh every uh every uh whatever pull request. Um the Is there a use case for this for like web MCP to be it it can't do the visual testing, but to do like a navigation, you know, to the extent that, you know, you've blended that it's the same form that is now marked as tool, right? Or the same JavaScript function that is called by the web page and the tool?

33:41 Right now, when you're doing end-to-end testing, uh we're talking about also tools using uh tools like Playwright and Puppeteer, and those are the same tools as the agents are using these days to browse the website. So, but um I think that web MCP is more for unit testings because you're actually testing a function and see if the function is working properly. And it looks more than that like uh the end-to-end human user testing because that's to do how how everything looks like. So, I think that you would still need to do that um that thing That that doesn't mean that you can not start using You can start using also LLMs and maybe local image models to improve the speed and accuracy of that testing.

34:32 Because it's also possible that right now with some uh very small models that can actually do web testing pretty good, even locally without consuming cloud tokens. Super interesting. So, let's actually shift into this sort of local models uh path, and just to sort of wrap up a bit this sort of uh web MCP. It sounds like um you know, you should like I guess kind of everybody trying to kind of keep up on this sort of content. So, if you're if you have an elaborate uh web applications on it, you should stay on top of it. It's in experimental mode, which again, I assume it will graduate out of uh appropriate to the sort of the pace of change uh that is in the AI as probably a reasonable bet. Uh and it is uh currently focused on interactive websites and visible browsers. And I guess kind of the people that need to be most attuned to it are probably people that are either looking to block bots and things like that like that post on X, or uh people that want to enable browser-side functionality like

35:33 purchasing in e-commerce. Yeah, sounds like a good summary. If you want to try now, you need Chrome uh 146 and enable the flag, and you need to install an extension that is called model context tool inspector that will let you debug your web MCP code. And it's pretty simple, so you don't need to spend like 2 months of training or It's actually pretty simple to implement. Hey everyone. Hope you're enjoying the episode so far. Our team is working really hard behind the scenes to bring you the best guests so we can have the most informative conversations about agentic development.

36:08 Whether that's talking about the latest tools, the most efficient workflows, or defining best practices. But for whatever reason, many of you have yet to subscribe to the channel. If you're enjoying the podcast and want us to continue to bring you the very best content, please do us a favor and hit that subscribe button. It really does make a difference and lets us continue to improve the quality of our guests and build an even better product for you. All right, back to the episode. Yeah, probably the harder question is is to sort of think about this mindset and say, what are the actions that you want and what are the user flows as opposed to the technical complexity of it. Yeah.

36:42 I'm correct. And so, let's let's move a bit to that sort of model. So, the other topic that I've sort of seen you you know, kind of discuss and and you know, been been kind of tracking at the cutting edge of uh is this whole world world of of models within the browser, you know, actually you know, like we talk about uh you know, giving an interface now to a to an external browser that is agentic, but this is literally within the web page, I think, browsers. So, first of all, tell

Client-side web AI reduces cloud costs and latency

37:07 us like I know there are two modes there, right? Of sort of ship your own model uh versus something that's in the browser. Like, what is this world of of browser built uh models? As of today, most uh if not all um AI-based API or apps powered by AI are using the cloud. I mean, it seems like the normal situation, you're consuming tokens from Gemini, from um OpenAI, from Claude, from whatever provider cloud provider that is actually inferring your models or the models you want in the

3 Highlight · Cutting cloud token costs with client-side AI

37:43 cloud, and you're paying for that. But that's that's cloud AI. That's the standard today. But there is another way that is growing every year that right now it's called web AI. I'm not sure if all the developers actually are using that name, or client-side AI. Mhm.

38:02 And when you go to that with the web AI, the idea is that you would execute AI models, even LLMs, locally on users' devices. So, we are not talking about executing locally on your web server. It We're talking about the client, so Mhm. the users' uh device. And for that, maybe you're thinking, well, but we don't have the power of ChatGPT. Yeah, of course, we don't have the power of the latest Opus 4.6, but the thing is that on a lot of cases, you don't need that power.

38:35 Mostly when you are when you are integrating AI in a web app, maybe you just want to, for example, categorize a post. You want to check if the user is using any hate in the text, or if it's uh adding insults or whatever, you want to filter that, and you don't need the latest Opus 4.6 to do that. So, you can use a very small model that it works perfectly for that, and it's going to be cheaper for you because you don't need to pay for the cloud. And you were talking about like that uh uh

39:12 uh just sort of clarify, is this about like a web model? So, it's different.

39:15 Like, a part of this don't pay for the cloud, run locally is already available as well by running actually sometimes even like decent-sized models, not Opus, uh on your machine through Ollama or like one of these systems. Or LM Studio, yeah. Yeah. But this one is something that that's in between, or like well, it's like maybe a uh another another variant. To be honest, it's the same. The difference is that with Ollama or LM Studio, you're downloading your models to your computer. And in this case, the models will be downloaded and executed in the browser for the user without the user even knowing that.

39:50 So, the user doesn't need to install anything. The user doesn't need to understand about models. The The model is already on the user's computer, and that's the first mode to execute this. Uh that's a built-in model that right now that will will be basically be available on a lot of browsers. Today, only Chrome is supporting that feature. That's built-in AI.

40:18 So, in that case, if you have Chrome right now on desktop, this is on desktop right now. Android is coming. Right now, it's on on desktop only. Chrome can download uh a small version of Gemini, Gemini Nano. And it's available client-side to infer from JavaScript from any website. Cool. So, then if Gemini Nano is uh that is even smaller than Gemini Flash the one that is on the cloud.

40:47 But for a lot of For example, you can translate content. Um it's pretty good at translation. I think that the test is like there are 25 languages where that passes the test, so it's pretty good. And you can translate content completely client-side just by using a JavaScript API that talks to a local model that is already in the user's Chrome. You said that Chrome can install. Like, is it If I'm writing a webpage, can I assume that it's there? Because by default, when you install Google Chrome, the model is not there. The first website that requests the usage of this API will actually say will download the model for the first time, will install the models on users' devices, and it will be available for the rest of the website, for that website, another website. How big is it? Like, what What is the size of that?

41:40 8 GB. Something like that. So, that's why it's not coming with Chrome. Yeah, yeah. It's not a small investment in terms of a

41:46 But once it's downloaded, it's between 4 and and 8 because actually there are three models, not one. So, based on what are you requesting, it will download one or the other. But it's between 4 and 8 GB, so yeah. It's It's It's more than Chrome itself. So, that's why it's not built-in with Chrome yet, but maybe in the future it's going to be built-in with the OS. So, maybe we're heading towards that future, right? So, then the browser will just ask the OS, "Hey, do you have a model?" And the OS will say, "Yes, we have these models." And then the website will just execute it locally. So, I I think that we are heading towards that future.

42:23 And So, that's one way to execute models locally, to use built-in APIs that today is Chrome only, which is good, but okay, not good enough to say I will use this and nothing else. And the other way is to use libraries. [snorts] And today, there there are there are some low-level APIs on every browser, including Safari, including Firefox. And that's WebAssembly to use the CPU, WebGPU to use the GPU.

42:57 And now on some operating system, you have WebNN neural network that actually if you have a a computer that has an special AI chip inside, it can actually use it from JavaScript. So, and on top of these three APIs that are low-level, there are a lot of open-source libraries. Uh there are like eight different libraries now from Chrome, from different providers that you can use that can run models using CPU, GPU, or uh AI chips, the TPUs or there are different names for that, MPUs. And so, that means that your JavaScript, your web app can download an open-source model.

43:40 You can download Llama. You can download uh Gemma from Google. And you say, "Well, it it it it can be Now you have some versions that are half a GB, so 500 MB." You can download that in JavaScript and then execute it client-side on an iPhone. Uh it works in in Safari, in a website. Yeah.

44:03 So, again, it depends on what do you need to do. Maybe it's it's not for for using uh for for a therapist, okay? Or for for uh teaching users history because those models are not pretty good for that. But but for summarizing, categorizing, to actually for chat for support chatbots, you can create your mini rug. So, rug is a like a an architecture that will let you connect your data with an LLM. So, you can make a support chatbot that talks to your own um information with those LLMs.

44:43 And in that case, you the cost of the inferring cost happens client-side. Right.

44:49 So, it's it's being executed client-side. And you always need to fall back into a cloud API. So, and now Google is offering a Firebase API that will execute client-side and as a fallback, it will execute a model server-side.

45:06 uh yeah. If it couldn't run

45:08 reason, yeah, the model can't be uh inferred or executed uh client-side. Yeah. I think that's uh it's super interesting uh as you as you think about sort of future capabilities and use cases for it. It sounds like right now, I mean, the numbers sound a little bit scary until you compare them to like a YouTube video, right? Or stuff like that, right? Like they It's not really that massive a deal to download, you know, half a half a GB, right? Or uh or something like that. It's not something you would want to every every page loads to uh to require. No, but remember that also you can use APIs to store that offline. So, then if it's a recurring user, it's just a one-time download, yeah. But but it's a an interesting opportunity to provide things that are um you know, I guess kind of reduce, you know, one one optimization or like value proposition that you mentioned is is one of costs. And so, you might not need to worry about everybody hammering your support bot or your translation bot. The two is probably some element of latency.

46:05 I guess that depends. Sometimes the cloud servers will be faster. Uh but there might be cases Well, it depends on the case, actually. But for some reason, for for image issues, for example, the latency it's um it's equal, so it's the same. For text, it depends on how much text you're sending. Uh if it's a very small, like you are uh lay- having a label to something, sometimes the local model yeah, it works really fast. Yeah. And then I guess on top of all that, there's also like the privacy or the security aspect of it, which is if you want to allow in certain SaaS applications, you know, to allow some functionality to happen only on the client-side, right? There's a bunch of these, you know, end-to-end encryption functionality and things like that, so the server actually never sees the data at the moment that, you know, might get translated to, and therefore you cannot use LLMs on it. But if you're running them locally, those might be reasonable, right? Within WhatsApp, right? Like within your app. Yeah, also also we need to maybe stop thinking on LLMs as the only solution. So, there are models that

47:05 are uh specifically targeting one use case. So, for example, Apple released an open-source model to detect um to to do some kind of image detection. So, I don't know, you can you can put something on the camera, it will tell you, "Oh, this that's ice breakers means cinnamon." Yeah, I like those ones. And and and that it's maybe 200 MB. It's a model of 200 MB and works client-side.

47:31 So, now you can have an OCR system that can detect objects um completely local. So, it it opens uh lot of opportunities for a lot of apps and web apps that can run even without internet, just offline, and can be very specific for for some um use cases. And And it actually works. And you don't need LLMs. You need a model that was specifically created for that purpose. Yeah. It's interesting.

47:59 It's probably You needed LLM in the ecosystem probably to create those models cuz they probably are distilled versions of a larger

48:07 Yeah, sometimes they are. and optimized. Uh but the the L, you know, like I guess in this case, it's neither L nor L, right? It's neither large nor uh nor language, you know, necessarily. Uh it is kind of an image model, which I guess is a language model of sorts. But yeah, but based on what you said, right now you can download There is a a new version, for example, of uh Qwen, the the Chi- one of the Chinese models, that the 0.5 billion parameters.

48:33 That one is really small, and you can fine-tune it. So, I say So, then you can have a very small LLM that is pretty bad at uh with facts, okay? But you can fine-tune that model in your computer. You don't need a very large computer to fine-tune that with your information. And then you have a pretty decent LLM small that knows about your stuff and can be executed locally on every device, even mobile phones. I think that is important in in terms of like the trajectory of of the industry cuz I mean, I would say that today, you're probably on the cutting edge if you're doing that, and you need to be quite thoughtful. Like, there's probably, you know, of all the people, you know, listening, you know, and kind of engaging with uh with these technologies, probably a very small fraction have use cases that they need right now that truly match the the kind of cost as in efforts uh versus reward uh on it. But I I do believe that open

49:33 models, and I I don't think I'm unusual in that sense, that open models are getting much, much better and they will get better you know, at AI speed, right? They tend to lag six-ish months, you know, behind behind the world. So, tuning in, you know, might further close the gap if you have enough of a specific use case. But even without it, I think as a as a competency to build, if you're building a new business, if you're building a new kind of interaction approach, you know, for your application, these are probably core capabilities you should keep in mind thinking about what needs to go to the server and what goes on the client. I guess not dissimilar to the thought of like the the kind of the responsive websites, but then even more like the the the rich the progressive web apps that run on the on the client side where a lot of server-side functionality has has moved to the client.

50:25 Right? Yeah, that's right. Super interesting. Who are there examples of like, you know, who who who have you sort of seen out there that's like most interesting in terms of using AI right now? That's a like maybe like a

50:38 Client-side AI? Yeah. So, I've I've seen a lot on the on the customer support bots thing because they found that they can I mean, their their tokens bill can get pretty high. Mostly mostly when they're they're hacking their prompts and they're so at some point they're losing control on the money they're spending.

51:05 And so, I've seen that they're interested in seeing if they can replace that with client-side. So, if you want to uh hack your prompt, go ahead, it's your computer. So, it will be you're hacking your own your own model. So, I don't care if you do that. So, I think it's it's about cost at some point, right? Because um everything is fine when you're doing I mean, your first project, you release a project, your MVP. I mean, yeah, maybe it's a 50 bucks, you're okay. But when you're scaling that Yeah. Maybe you receive a bill of $100,000 of tokens that you need to pay. And when you look at that bill, you say, "Well, let's see if we can cut this with without without like um changing the quality of of the service by moving some parts to the client. Yeah, it's so interesting. It's like it's funny cuz it feels like it's like an escalation level, right? You're talking to the front-line support, which is your local

52:06 model, and then, you know, it's like, "No, no, no, let me talk to your boss." And then you go to the cloud, yeah, exactly. And then you go to the to the cloud side, you know, to to get it. It's also interesting how support has become a bit of a role model in terms of identifying trajectory of application use for for LLMs cuz probably, you know, you can debate sort of software development, but you know, you see the pattern over there as well. But in support, you definitely sort of saw that, you know, be the first role to uh I guess you can sort of say be displaced. And in software development, I I have this sort of intro slide for Tesla on it and they talk about agentic development kind of traits. And one of the ones I have on there is that it's cheap then expensive. Um because like it's initially cheap because a single person can do so much, you know, so it's so much cheaper. And then you get the bill, it's like, "Do I need to be spending that much? Can I can I do this a bit more cost-effective?" But most people are not there yet. Like most people are are still in the discovery phase. And so, it's interesting to think about support being kind of the forerunner over there. Now, this is this is getting

53:06 better and better every year. So, it's still I mean, if you if if we look at some data that we don't have, I think that probably companies using client-side AI are less than 1%. So, but it's getting better and better in terms of quality and cost and performance. So, if other browsers, I don't know, maybe Safari with now the new open intelligence contract with Google, maybe they will also do something similar in Safari, so they might get a local model local side and on the the local side. So, if that happens, well, maybe we will see more built-in AI APIs in in web apps. Yeah, I think I think that makes perfect sense and it's always hard to imagine how that doesn't work. I guess one thing we didn't talk about in the context of web AI is the sandboxing of the of the browser. And I guess if you if you operate in this fashion compared to an Ollama or whatever it is, you're

54:07 still working within the sandbox. So, if you have two tabs, Mhm. I guess minus the sort of sharing capabilities that, you know, browsers support, in theory, each of them will have its own copy, its own sort of data set. So, one malicious website wouldn't be able to sort of siphon off, you know, there's probably going to be some set of vulnerabilities, but today, you know, putting those aside, wouldn't be able to siphon off another tab's information.

54:32 Yeah, correct. That's correct. Yeah, also we need to remember that the LLM is a still a today a black box that you send input and it gets you something back. So, if in this case, two websites will send different inputs and they're not tied together in any way. It's just two two prompts to the same LLM. So, the security thing is going to be the same with or without the LLM. That's what I'm saying. Yeah, yeah, but but it's interesting to think about I think so so all this conversation, you know, really revolves around the sort of the browser as the AI sandbox, right? And and how you know, you can have the browser since the browser is the entry point to the world and to your systems and to your data in so many fronts on it. How do you get it to interact with agents? And increasingly, we have, you know, the rise of the agentic browsers and so very much the AI sandbox and that's in the web MCP land and, you know, create better interaction points, many of which revolve around security, authentication, purchase. It's

55:34 about trust delegation, right? Sort of actions that you might bump up. And so, I think that's very interesting. And then you can augment all of those with some local element LLM action that is that is part of your browser, which is independent of the other path, but they also point to the browser will have LLM capabilities itself. So, maybe in the first layer, it's more about like a brokering or an interface layer, but increasingly, it might actually embody the engine to to perform activities. So, from a from a website builder perspective, first you need to think

The web will outlast native apps in the agentic era

56:08 about how do agents interact, and then you need to think about how do you embed agentic functionality literally into your local web, right? Yeah. Also, I think this will become more important because even when some people thinks again for the probably nth time that the web is dying. So, if you think about that, by coding and all these new apps that are appearing because AI is coding them, our web apps.

56:35 So, actually, the web is growing, not dying. So, and I think that that means that we will have more opportunities than even before then to actually start thinking and using these new architectures. Yeah, super interesting. So, I guess, you know, maybe to to close off, if you can kind of take out your crystal ball and, you know, we've sort of seen the journey on how responsive web applications, I'm not even sure if I'm using sort of the word. That was that was the name that I uh you know, moved a little bit away from the web in my The rich internet application, that's Rich internet application. RIAs are RIAs now, not the RIA. The rich internet applications have become, you know, the I don't know if they're the norm everywhere, but like in in many in many facets of applications, they have become the way to build many apps and a lot of functionality moved to the client. I guess what what what is your if you if you dare to cast your eyes 3 years out even, right? In the world of of AI, do do you envision there's going to be

57:35 applications that really innovate and that this will become the norm to have substantial in-browser, in-website LLM functionality? So, I think that yeah, so in in the future, we will see less native apps and more at least on on mobile devices because maybe on the desktop, there is a nowadays a new approach that a lot of people are doing native apps, which is a security problem, but anyway, on the mobile world, I think more more more web apps actually in the future.

58:08 Without maybe without using the term, it's just a link, it's just a QR code, it's just an interface that users are using or agents are using. And maybe we are not going to call them web apps anymore, just AI apps or apps by coded apps or however you want to call that. It's going to be web. And on that, I think that the the price and the quality of local AI will improve a lot in the next few years, which will trigger a lot of new use cases of local AI. Not all of them will be local, not all of them will be web apps, but a lot I'm seeing a lot of them moving into that future that will be faster and probably more performant.

58:57 Yeah, super interesting. And it's funny how every every time the web gets stronger, it raises this debate of why do you even need native? You know, can I can I just have it be the web? And I guess there's a claim here that if functionality becomes more agentic, if more if the interfaces actually get simpler because they get very um kind of chats driven, right? Or sort of, you know, natural Like API based. You're you're you're you're you're picking up that gauntlet again, right? You're sort of picking the glove, whatever it is, and saying, you know, you think this debate will come up again and and the web will win. Yeah, I think so.

59:37 Cool. Max, thanks a lot for coming in in those super interesting conversation. I think exciting new frontier, you know, for for the web. It sounds like it's still like today today probably still for the pioneers and maybe for the e-commerce shops or or those who really care about controlling agentic behaviors. But again, this is kind of AI pace. So, it's probably going to become relevant to many many more very quickly. So, thanks for sharing the great information coming here on the podcast.

1:00:06 My pleasure. Thank you. And thanks everyone for tuning in and I hope you join us for the next one.