Letting AI Interface with your App with MCP — Kent C Dodds

3,604 words · about 14 min to read

The vision of Jarvis and why we lack it

0:00 Hello, my name is Kent C. Dods and I teach people how to build excellent user experiences wherever they are. I've taught people how to build excellent user experiences on the web. And now, guess what? We're all moving over to AI because that's where our users are going. And so now I teach people how to build excellent user experiences with epicai.pro as my course platform. Go take a look at it. Today I want to talk with you about how user interaction is changing, how model context protocol and things like it are helping make that change possible and what your role in this is as a product developer or as somebody who's trying to reach users where they want to be and where they want to be is inside of AI assistance.

0:45 So let's talk about that. My talk is letting AI inter interface with your app with MCPS or model context protocol services. Uh and normally when I start my talk I actually invite people to stand up because or at least get their body moving because our bodies need blood to flow for our brain to operate properly. Um this is kind of an awkward thing to do in a remote setting. So we're not going to do that. But if you've been just sitting around for a while, I invite you to move your body.

1:14 Stand up if you can and get blood flow because your brain does need that. So, make sure you're taking care of your body. All right, so let's just get right into it. I'm going to show you a video of Tony Stark interacting with his AI assistant, Jarvis. So, if you're not familiar with Tony Stark, he's Iron Man. And um this is from one of the Iron Man movies. Um, as we're watching this, I want you to consider what are the things that Tony is having his AI assistant Jarvis do that we are not able to do with the technology that we have right now and then we'll talk about it once the video is done. Here we go.

1:52 I've compiled a Mandarin database for you, sir, drawn from SHIELD, FBI, and CIA intercepts. Initiating virtual crime scene reconstruction. Okay, what do we got here? Close. The heat from the blast was in excess of 3,000° C. Any subjects within 12.5 yd were vaporized instantly. No bomb parts found in a 3m radius of the Chinese theater. No, sir.

2:21 Any military victims? Not according to public records, sir. Bring up the thermogenic signatures again. Factor in 3,000°. The Oracle Cloud has completed analysis. accessing satellites and plotting the last 12 months of thermogenic occurrences. Now, take away everywhere that there's been a Mandarin attack.

2:47 That's two military guys. Ever been to Tennessee, Jarvis? Creating a flight plan for Tennessee. All right. Well, that's pretty cool. Well, I would I would like to have a Jarvis like that that can help me. Can you imagine Tony doing all of that research and everything without an AI assistant? Uh, it would take longer. Let's just put it that way. So, what did Jarvis do? Compiled database, generated UI on demand, access public records, brought up thermogenic signatures. I I think I know what that is. Uh, and Tony said, "Take away everywhere there's been a Mandroid tag." So, joins across uh different data sets. Uh, showed related news articles. uh created a flight plan for Tennessee and answered the doorbell or at least showed Tony um who was at the door with like all their you know don't go to Tony's house if if you want your privacy but uh yeah so this is really exciting uh exciting stuff that to think that like we could maybe have

3:47 some of this and so the question is can we like what what did Jarvis do that we can't do okay so there were actually a couple of things it's not just nothing um We probably uh couldn't uh create a database based off of field FBI and and CIA and uh shield database stuff but not for technical reasons and the holographic stuff I think we're we're still working out. U but generating UI we absolutely can do that and people do that all the time. Uh so we have the technology to do this stuff. So my question is why don't we each have our own Jarvis already like what what's stopping us? And the the interesting thing is that we've actually tried this.

4:30 We've been trying this forever and it's just not it's not quite there. And so why is it not quite there? What's what's holding us back from being able to have Jarvis? Like isn't that just the most amazing experience? And and Jarvis isn't just like a like something that you talk with. You know, if you're in the office with a bunch of other other people, you probably don't want to have a talking interface. It's also like he uses uh typing and he also uses gestures and and everything too and he uh generates dynamic UIs and then he can interact with those as well. So like it's a really really awesome user experience.

5:08 So why don't we have a Jarvis even though we've been working on it? What what is holding these different assistants back? And the answer is

1 Highlight · Why custom integrations fail to scale

5:15 integrations. It's just really really hard to build all the integrations for all the possible things. Here are a bunch of logos for things that you might recognize. some things that you don't. Some are technical, some are like normal people things. And I want to have a robot that can interface with all of them. And like here's the the real kicker. I want to have a robot that interfaces with everything. And the uh challenge of building an integration for every one of those things just means that Google is not going to build an integration for your local city's government website for reserving pavilions at the park. And I want that.

5:52 I want my robot to be able to do all of those things. And if it can't do everything, then why spend the time wiring everything up together so that it can do some things? And there are some activities that we take or some activities that we do, some things that we do that don't really um like don't make it worth it to spend all the time wiring things up. Like we just don't do it enough. But it would be really cool if there was some standard mechanism for my AI assistant to interface with something that I've never used before and I won't ever use again, but I don't want to have to like click around the website and figure it out. I want my LLM to figure that out. And that is what model context protocol enables for us is a standard mechanism whereby our AI assistants can communicate with these uh various uh tools and services.

6:46 So let's talk a little bit about the

The evolution of LLM tool integration

6:49 history and architecture of MCP model context protocol and and just like tool calling and that sort of thing in general. So phase one about 3 years ago chat GPT comes out and uh this is a really really exciting time because it answers our questions. It can like we can say like tell it something and ask it something and it can answer the questions and maybe the answer isn't always right because it's not intelligent. is just generating tokens.

7:17 But like lots of the times it is and it's gotten way better. And so this is really exciting. And what made it so pivotal is the fact not not the fact that we had an LLM. We've had LLMs for a long time. What made ChatGpt so awesome is the host application layer around the LLM. Chat GPT itself. not the LLM, but the application around it that brought a good user experience to interfacing with an LLM. And then once that happened, a lot of investment went into LLMs and they got better, better, and better and better, much faster because we started to see a lot of value. The problem was that you would have to provide the context yourself. You copy paste your code and say, "Hey, what's wrong with this? Please refactor it, whatever." and then you'd have to copy and paste its result back into your codebase and then you find out it wasn't working for uh some other reason. Um but you there was a lot of context like manual context bring it in in the form of text and then

8:17 we added images but uh having to manually bring in that stuff and then manually take it out and then of course you've got your context window and all that stuff. So it it couldn't do anything. it could answer questions but like couldn't do anything and managing context ourselves was kind of a pain. So then we get into phase two where the host application is like hey I'm like regular code I I interface with this LLM I could tell the LLM hey I need you to let me know if you need more context and I can go get it for you. So now we have search engines and hey if they want to schedule a time to meet with somebody I have a a calendar integration and uh oh if you want to summarize your Slack messages or your notifications or something I have a Slack integration whatever. So the host application is now enabling it to do stuff. I can tell it hey do this thing and now it will do that thing. And uh the the problem here is that you're kind of limited with the the things that it can do and also the sources where it can get its um its

9:20 context. You're limited by the uh developers time uh at OpenAI or Anthropic. How much time do they have to dedicate to building out these integrations? And this is really the same problem that we have with our existing assistants and the existing efforts that we've put into building integrations with all these different services. You're not going to convince OpenAI or anthropic developers to build an integration for your city government website or anything like that. That's just not going to work. And granted, OpenAI has their uh own GPT plug-in system, but it's proprietary. And so it's uh difficult even though chat GPT has a lot of uh buyin from the users that you have um it's difficult to justify building something special for chat GPT and then something special for anthropic and then something something special for Google and whatever. Uh it'd be much better if we had some like standard mechanism. So yes, it can do stuff and and for a lot of people who couldn't get buyin from the um the

10:21 company to build a built-in integration, they would just build a wrapper, their own host application with their own tools. And the thing is, users don't want to do that. Users want one Jarvis, just one, just one Jarvis that is able to augment itself with any capability in the world, any possible capability. They don't want to have to load up your uh LLM wrapper with the context necessary and then grab that context and put it into the next tool and whatever. They want one thing that can interface with all of them. And so, yeah, it can do stuff, but it can't do enough. And that is a big pain. And that's what pushes us into phase three, MCP. Now with MCP, it can do anything because MCP is a standard protocol that all of the different assistants support or will very soon support. And uh now you can build to the MCP spec and you're usable by any of these AI assistants because it's a standard. So we can do anything.

11:22 The clients aren't quite ready at the time of this recording just yet, but soon they will be and it's going to be

How MCP architecture connects hosts to services

2 Highlight · Deconstructing MCP architecture and components

11:29 phenomenal. We are just one really good user experience application away from Jarvis for everybody and that's very exciting. So to expand out on this a little bit uh as far as the architecture is concerned the host application is communicating with the LLM and the host application tells the LLM what services are available to it and those services can be dynamically added. Uh they uh can be um added and removed. So like this context can be managed by the host application and the LLM just knows what's available now and the user's query and then it can select the most appropriate tool. The host application creates a client for every uh service that it wants to integrate interface with and this client is a standard uh client. So it doesn't need to have any special uh integration with anything.

12:20 There's a standard interface for this and then the service provider creates the MCP servers and uh those servers interface with tools, resources, prompts, sampling like lots of really uh cool features on that end of the spectrum. And this is what makes it so great is that the service provider is in control of this stuff. This is the stuff that's going to be unique to every uh service. uh everything else like the the server and the client that interface that communication is standard and that's why this works. That's why it's so cool. This is how uh the way I say it is this is how we give Jarvis hands to be able to actually do stuff. It's very exciting. So I want to show you a little

13:01 demo of um a couple of MCP servers that I put together uh to kind of explore what's possible here. Like I said, the clients aren't quite ready. So, the experience is not the uh what the Jarvis experience was, but I I don't see any reason why we can't get there. So, let's get into that. So, first, um I made this uh in London, so it's going to find out my current location here in a second, but uh here's the prompt. This is in in Cloud Desktop with which has been configured with my MCP servers, three of them. Please write a journal entry for me. So, one of the MCP servers is a a journaling uh server. Uh here's my email address about my trip with my daughter.

13:44 I would like you to derive my location and weather conditions from my device location and make up a creative story with relevant text. Thanks. So, we execute this and as part of the context, um it uh has an MCPP server called locationator which will determine the current location. And so, with this, I'm going to allow it for this chat. Now, you notice Tony didn't have to approve different tool calls and stuff for Jarvis because Tony has a lot of trust built up with Jarvis. We don't quite have that trust built up yet or that uh capability and so that's why we're not quite there yet. Um and we have to approve every tool call and stuff, but we'll get there eventually once uh things get more reliable. So, here it gets my current location. That was my location at the London Hotel, so don't worry, this isn't my home. Uh and then I check the weather. This is from another tool uh another server uh get weather for getting the current weather at um uh given uh coordinates. And now it's going to run another tool from EpicMe. This is

14:45 the authenticate tool because I haven't logged in yet. And so I'm going to provide my email address to the EpicMe server. And at the time of uh this demo um I didn't want to actually open up my email in uh as part of this recording. And so um I was just running it locally and I logged it to the uh console. So let me just grab that token and I provide that O token. And so now my MCP client has been authent or it's going to be authenticated with this validation token. So the MCP server has authentication built in. It's based on OOTH 2.1. And so it's uh as secure as anything else that you're using OOTH with. And um my uh Jarvis client or or cloud desktop has been authenticated so I can perform authenticated tasks which uh one of which is to create a journal entry. Now of course the LLM is going to write the journal entry for me. I didn't want to take the time to do that but you could write your own journal entry uh of

15:46 course and uh have it just insert that yourself. And with the right kind of interface you could uh speak your journal entry and it could format it nicely for you and everything. That's what makes these LLMs so cool and especially when we're talking about multi multimodal stuff. So, um we're going to create a journal entry um with that information. U my MCP server configured the inputs and things. And now it's going to check, okay, what tags do you have available? Okay, great. So, you've got a travel tag. Uh let's or let's create a travel tag because you don't have one for that and that makes sense for this journal entry. We'll add that uh tag to the entry. And uh and then like it continues uh in this manner. And here's another really cool and interesting thing about MCP in the fact that it were working with an LLM.

16:37 Um is I I can actually ask it, hey, could you show me that journal entry? And remember that journal entry, it had a title and it had contents. Um, but here it's going to uh retrieve that journal entry and then it decides based off of what it retrieves that there's a a good way to format that that's better than just the JSON format that is provided. So the server can communicate in a way that makes a lot of sense for the server. Um, and then the client can take that as context to display whatever it wants to the user. Right now we don't have um very many uh clients that support any mechanism for displaying like a dynamic UI or like a card or or something like that. But you can imagine that we could totally do that. There's nothing stopping us from doing that. And I think it's actually also interesting that it rendered this as kind of markdown with a title and everything too. Uh which I think is pretty cool.

17:31 And additionally uh I could actually ask it to translate this into any language and it would do a pretty good job of that too. So maybe the MCP server is uh sending responses in English, but you're talking to your LLM in Japanese, and the

3 Highlight · Building apps without traditional web frontends

17:46 LLM can convert that English response into Japanese, which I just think is very very uh cool and interesting. So here now, I'm telling it to delete the post and log me out because it's fake. And um again, all of the authentication stuff that you would expect uh all works here. The EpicMe MCP server is

Moving away from browsers toward natural language

18:05 completely only accessible via uh MCP and uh clients like this. You can't actually access it as a web application. And I think that is the transition that we're going to be experiencing very soon. Users don't want to use a browser. They don't want to Google and have to figure out the right way to phrase their question. When I was teaching my parents how to Google, uh they just typed out a full-on question and their results were not very good.

18:32 So, we had to train them. No, no, no. You have to write it in this special format and then you'll find like with these keywords and stuff and then you'll find what you're looking for. Well, now we're going back to what's more natural. Just speak your question and the AI will be able to figure out what you're trying to accomplish and not just what you're trying to search for, but what you're actually trying to do. And it'll actually be able to do that thing for you. That is what MCP enables and that's why I'm so excited about it. So just to wrap us up, I've got a bunch of

19:02 resources for you. There's the specification for model context protocol. It's very good. You should definitely take a look at that. I am very active on EpicAI. And so you can go check out epicai.pro and learn all about not just MCP, but AI in general. I've got a whole bunch of uh posts that you can take a look at uh all about um the the future of user interaction and how AI is changing the game for us. It's very exciting and I hope that you join me there. I also have workshops and cohorts and things that you can go through uh join me on and the the at whatever time you're watching this there will be uh something to to go and learn there. So go check out epicai.pro. With that, I just want to say you're great. Thank you so much for watching my talk.