Stateful Agents — Full Workshop with Charles Packer of Letta and MemGPT

15,858 words · about 63 min to read

Workshop setup and prerequisites via Docker

0:10 so today I'm going to be going over a workshop so there is like an interactive component i think you can probably get a lot out of this talk just by watching but if you do want to participate um you're going to probably want to install Docker on your laptop if you don't already have it and then you're going to want to pull this Docker image uh if you do this it should be very easy to follow along there's also a notebook in the channel it's called I think like workshops letter and you can just hop over to the I think I also made like a tiny URL for it um but if you hop over to this link this is like the notebook I'll be running over for maybe like the first 30 minutes and this notebook is kind of like a client and the client needs to run against the server and the Docker image is what would be the server in your case um so again I'll leave this up for a few seconds this is basically where you're going to go get the notebook and then and I believe this is also in the in the slack channel and then yeah the main thing you're going to want to do if you want to follow along and you have Docker installed is like docker pull um I guess

1:12 like a quick show of hands like how many people here have Docker installed on their laptops so okay sweet yeah most of us okay great um okay does anyone need a few minutes or anything um I guess the people who do want to kind of follow along just make sure to check out the workshops channel um and of course this is recorded and then all the materials will be online so it'll be very easy to kind of follow along later as well yeah so I believe the title of this workshop was um like agent memory the LMOS uh I think maybe a better title for the workshop is stateful agents so I think you probably heard a lot about

Stateful agents bridge transformer statelessness

1:47 agents um over the course of this conference as I also would like to pull the room like how many of you if you think you were asked today to give like a bit a concrete definition of an agent would be like pretty comfortable in what you what your definition is okay yeah so despite hearing you know a lot about agents over multiple days it's still pretty hard to define i think this is kind of a problem that's become much worse over the past year and I think that's why it's kind of useful to maybe like sharpen our definition of agents so I like this term staple agent um I think staple agent actually is kind of what agent meant before you know the LM era and I think these days I think a

1 Highlight · Transformers are stateless and require explicit state updates

2:25 very common definition for agent is it's like an LM that's taking actions in a loop right i think that kind of that kind of works but it also misses a really big um part of that process which is that when you run the loop it's a closed loop and the agent gets updated and it turns out that you know in this new wave of AI the fundamental unit of compute we're using for AI is stateless it's not you know a recurrent neural network it's not an SSM it's a transformer and the transformer inherently is like a stateless machine so that means that when you close the loop you have to have some sort of mechanism for updating the state and you know traditionally like when agent was defined back in like the RL days or like back you know before pre-LM I don't think this really was that big of a distinction to make like staple versus not staple agents i think because LMs are stateless and that's like what everyone is using for a for AI now it's a pretty important distinction so hopefully by the end of this talk you kind of understand what staple agents mean and you probably hopefully agree with me that um stapleness or memory is

3:25 actually probably the most important thing to solve if we actually want to get you know if you want 125 be the year of agents or if you want agents to actually deliver on any of the hype yeah so when I talk about stapleness I think stapleness is pretty

3:40 synonymous with memory i think that's because LM effectively have no memory they just have the memory that's in their weights and then they have what's in the context window um so like memory and state context these are all kind of synonyms with LM or like LM driven AI and humans are of course stateful so humans they form new memories they learn over time and LMs don't so any of the learning you're going to do is going to have to be done by you the user of the LM or by the framework but it has to be done by somebody and traditionally you know in the past few years I think by default this just means appending to a list right and I think for workflows and like stuff we were playing around with from 2022 to 2024 when not a lot of people necessarily cared about like making money um or doing useful things with agents that was okay but I think it becomes a very big problem when the only mechanism you have to handle state when you're actually trying to use agents to do useful things is kind of appending to some sort of list and a list that's like held in Python process memory or you know in uh like a Node.js process so the natural question is if we

4:43 have LM and LM are stateless and basically the main thing we want to get out of state is we want to have humanlike intelligence that means learning so how do we actually do that um is how many people here have like

MemGPT acts as an operating system for LLM memory

4:53 heard of an MGPT um okay yeah so before I was you know doing the startup Leta I was a PhD student at Berkeley and the CHAGPD stuff happened in the middle of my PhD which is kind of like interesting because it meant that like most people doing research kind of pivoted their research entirely um so in the few years of my PhD post chat GBT I was really focused on memory and agents and I wrote this paper called megpt with a bunch of my colleagues and some of my co-workers now at lettera um and this is really talking about like a memory management system for LMS so if LM can't if they need some sort of memory management and the de

2 Highlight · Automating context compilation via an LLM operating system

5:29 facto way we do this is like a human appends to a list well if you believe LM get better and better and better shouldn't this be done by another LM like shouldn't the AI do the management for the it shouldn't AI do the memory management for the AI so when we called like this a me an LMOS we really referring to like a memory management system and this is effectively like a very high level graph or high level figure of what I what I mean by like statelessness versus statefulness on the left is basically what most people do um you know you kind of have a context window the context window is pretty loosely defined it doesn't necessarily uh it's not broken up into very distinct pieces it's not tied to any state in a database per se this is just something that's held in process memory and you're like appending to it over time this is what happens if you use lang chain like in the lane chain buffer memory or like query this will got get buried really deep in the codebase and this is actually you know often why you need tracing software like observability observability kind of exists as a category because you have this black box of tokens that just gets shoved into the LM and stapleness or like staple agents and the the key thing we're trying to do

6:30 is understand that you know what's in the context window if we have a machine assembling it there's some that means there's some sort of like context compilation problem where there's an optimal way to arrange the context window of an LLM and that context comes from state the state can be kind of very large much more than can ever fit in the context window and this is basically what you do if you're a power user of chatpt or a power user of claude you're doing this yourself right you're kind of like compiling the context every time you start a new chat because I think a lot of us you know probably have experienced like having a chat that goes like way too long it starts to derail and then you have to redescribe everything you were doing back to chat GPT like a pretty painful experience um so can we basically automate this with a machine so yeah taking a step back like why do we even want staple agents like

7:18 why are agents we have today not enough like why is the a why is the current like paradigm of LM driven agents like not good enough to reach let's say like AGI or like some like the AI we see in science fiction i think the main problem is that agents we have today they just can't learn from experience um or the way they learn from experience is extremely limited and I think if you're just running like workflows um you might not notice this so you might it might not really be a big problem but I think if you're trying to build like assistants companions co-pilots this becomes like pretty evident um so imagine if you have some sort of chatbot so this is a a direct screenshot from like the MGPT paper um but you have an AI and a user and the AI you know it say it can see that it's February 14th so it wants to like ask the user what they're going to be doing on Valentine's Day because they have stored in their memory some recollection that the user's boyfriend is James so the the AI asks like "Oh hey how's James doing do you have any special plans today?" And you know the user says "Actually you know James and I broke up." Um so it's obviously a very very bad mistake by the

8:20 AI but maybe it's like kind of unavoidable um but you see the AI here if it has some notion of like a permanent readr store it should do something like transact that you know James is no longer the boyfriend james is the ex-boyfriend um and if you don't do this kind of thing and you run your agents for long enough and you just do like recursive summarization or if you just have like a really long context model you're inevitably going to make some mistake where you tell you know you tell the user something about your boyfriend James and that's like a devastating error if you're trying to build like a consumer app right and it's also something that just a human would never do i think like humans something like this would get like written to your quoteunquote core memory very aggressively i think the other reason you want stapleness and learning um I think many of us here maybe like kind of work companies and we're trying to like use agents to make money or like drive value you know for shareholders or whatever um but at the big difference between consumer and enterprise is obviously data like enterprises have much more data they can ever fit into like Gemini 10 million tokens right often like per

9:20 user you have more than 10 million tokens so how do you actually like learn from that data so you we can kind of think of there being like another training phase after the post- training like during post- training right you train the model and then now the model is deployed into your enterprise and the model should kind of learn about your company right and that's kind of like training again but it's not training into the weights it's training like into the incontext memory um so staple agents naturally kind of like encompasses this concept and this is like pretty useful if you're trying to build this this kind of stuff inside of companies and yeah again like really the reason I was kind of inspired to work on like the MEGPD stuff and also the reason I'm really excited to be doing the work I'm doing at Leta is because I'm very interested in just AI and making AI that is very humanlike and clearly there is some very serious deficiencies with the current iteration of LM agents um they're not very humanlike especially with respect to how they deal with memory yeah so I covered this already i think this is like a tweet I saw a few weeks ago or something I thought was

10:20 pretty funny um but I think you know if you all use chat in this room here this is probably like a pretty universal experience you kind of have a a conversation that goes on too long and then you know you feel very devastated because you're now going to have to mentally context compile again to like redescribe to chatbt what's going on or like flaw or whatever you're using so the promise of staple agents um there's actually a lot of promise just in product right i think if you actually have true statefulness then that means that you know this sort of experience shouldn't happen there shouldn't be any derailment so you shouldn't have you know like a clawed instance going haywire um and actually you should have the opposite happening like the the experience should get better and better and better over time as the AI kind of learns more and more about you and this is you know kind of like maybe the promise of trackp memory um but you know I don't think any of that stuff is really working that well and of course I think when you don't when you no longer just like shove stuff into a context window when instead you're kind of really like creating me humanlike memory constructs I think the behavior of the agents just becomes more human like you

11:22 have the same kind of like fuzzy memory that a human has you have the same kind of forgetfulness but you also have the

Hands-on exploration of context management and UI builders

11:27 same kind of recall so in today's workshop I think it's kind of going to be split into two parts i think the main thing we're doing at the front is a notebook and of course like you don't you can just watch um but if you want to you can follow along in the actual the uh the Jupyter notebook file and the goal of that notebook will just be to like lay out the basic ideas um behind this sort of like context management system in an LLM and at a very very high level you know if I had to describe it in one sentence it's just that you make an LM either the main LM or like the subconscious memory system LM just aware of the context problem you kind of can just describe to it in English hey you are an LM and you have a context window of like 128k tokens and I will let you know when you reach 100k tokens and you're going to have to like manage memory with these tools so it's all centered around tool calling uh and I think centering around tool calling is actually very effective because LM are getting better and better and better at tool calling and the second part of this workshop is going to be um kind of like building these building these stateful agents in um the letter framework and

12:29 then also like the front-end ad so I think if you're very interested in kind of like no code stuff I think this is pretty interesting because I think we're all very very familiar with the paradigm of the playground right it's like ubiquitous every single AI company has their own playground but I think once statefulness and like staple agents become the dominant paradigm which I'm pretty sure they will um then I think you're going to have there's going to be a new iteration of the playground and the AD that we built I think that's our best guess as to what that experience will look like but I'm sure like everyone in this room who's worked very who has like spent a ton of hours with LMS you probably have a very strong opinion too on what that experience will be like like what happens when you know chat GBT is no longer like it no longer has like a history bar on the side and says it's like a unified experience like what is the correct UX for that what's the correct DX um and then if it's not a consumer app like chatbt if it's a de developer tool like what do you as a developer want to see from your agent like you probably want to see all the way down to the context window so I said this at the beginning i don't

13:30 think any anyone necessarily walked in midway so if you're going to follow along you're going to want to download Docker on your computer i think most of the people in this room have Docker on their computer you can do this with PIP but PIP is pretty terrible at package management so you might like hit a bug um and yeah I already covered this but if you're interested in any of the ideas that I'm going over in the initial notebook they're basically just strongly distilled versions of the ideas from the METPD paper um like reduced to their like most simple um yeah their most simple components but if you're like interested in this more like a research capacity you can also just read this paper okay and yeah we're going to be doing this in Python um I guess like how many people in this room have use Python as their main language versus Okay and I imagine like everyone else is Typescript or something yeah so basically what we're going to be doing is Python but leta the um server is not Oh maybe Go or Rust okay yeah yeah so we we don't have a Go or Rust SDK but we have a

14:32 TypeScript SDK and then we also have a REST API and actually our Python SDK and TypeScript SDK are just like programmatically generated off the REST API okay so I I left this on the screen a little bit earlier um to kind of like preempt any pauses here so I might just like continue on um and yeah these are these links are all in the channel like workshops hypen leta on the the shared slack okay let me hop over oh and I think we because this workshop has a reasonable amount of time allotted to it um I'm very open to like pausing for questions and stuff so if anyone has any questions please just raise your hand not going to pause and then if you're just going too slow we can fix that later yeah okay

15:32 questions all right so yeah I guess how many people are actually going to attempt to follow along it'll be useful for reference okay decent amount of people so I'll try to Yeah definitely uh raise your hand or something if something's going terribly wrong and you're not able to follow um let me boost the Okay is let me know when the font is large enough is this large enough for the people in the back to to read okay great and all right okay so if you ran the if you did the Docker pull um the next thing you're going to want to do to b to start this entire uh notebook session is you're going to want to start the lettuce server so if you were using Docker the way you're going to do this is run this docker run command i'm realizing now that this is not Oh this is this is in

16:32 you can copy paste this out of the GitHub um the GitHub link but I'll also post it in the Slack yeah and then if there's a free endpoint that's live so you don't actually need to have these keys um so you can just run this and it will you'll be able to run the demo fine okay and then once you run it you'll basically see the server kick off in the background yeah so while people I'll give everyone like a minute to do that and while everyone is kind of kicking off their servers might also just show you kind of a diagram of what's going on here yeah so basically the way this whole stack works is leta is like it's an open source stack um it's basically fast API

17:33 Postgress and Python um in the middle like Python logic so in that Docker container like it's exposing an API that's actually like very you know pretty robustly documented you can like go to our API reference here um and the API is how you interact with all your agents so if you're using the Python SDK it actually just you know goes over the the REST API and you basically build your Asian applications on top of this API so this actually looks very similar to like the chat completions API it's just like session based right and so you don't have to provide the entire the entire conversation history every single time you interact with an agent so we'll see that kind of immediately in the notebook as well okay so I assume everybody who's following along has done the the thing to launch um is there Okay it says that there's a directory that doesn't exist do I need to create that um oh yeah can you do liker and all that okay did anyone else get that directory doesn't exist problem

18:34 can you send him because I can see Okay do you So are you able to see this here oh no okay yeah so it's let me Yeah that fixed it okay great all right any other problems does everyone have a server idling now on their laptop great okay so now that we started the server um now we can kind of like start running the notebook so this part is basically you're going to go to this go to this channel click on this link you're going to see a readme and the readme has all the commands so I I'll kind of like go one by one and paste these commands into the workshop channel it's a workshop leta yeah okay so you need basically clone the repo

19:40 Once you clone the repo just CD into the specific example i'm assuming everyone's using Docker so I'm just going to kind of ignore that um okay and ideally everyone has Python installed in their laptop hopefully yeah yeah and then once you run that Jupyter notebook command um all you need to do is just double click on the notebook here if you're not familiar with um Jupiter it should automatically like

20:40 open a browser tab that should have this and then you just like double click and you'll open the notebook okay so among the people who are following who is a little still not to this notebook stage yet okay yeah still so it's been a while okay yeah yeah no problem okay no other issues nothing on fire yet okay all right how's the Jeep looking i had to spell that okay okay so everyone has this notebook up who is intending to follow along great all right so yeah if you're not familiar with notebooks you basically

21:41 just execute them cell by cell um I mean number the first thing you're

3 Highlight · Server-Client Architecture for Stateful Agents

21:46 going to have to do here is like import the client so again this is a little bit different from some other frameworks you're familiar with like I assume everyone here has like used lang chain before probably or heard of it um well like crew AI autogen um padantic uh AI things like that so I think among those frameworks there's a pretty big distinction with leta and that leta is like a server client process and a big part of that is because the agents are intended to be staple and like persist indefinitely so it's very hard to like persist things indefinitely if you're kind of holding everything in application state as opposed to having like a server that's a centralized source of truth so we basically run the server on the left um we connect to the client you know you

Managing agent memory blocks through a server architecture

22:25 can actually park the server anywhere you want because it's a Docker image you can like run this in the cloud it's very easy to like drop on Kubernetes or something in this case it's running on our laptop so we just need a local host and the default port so the first thing we're going to do is create an agent so I want to make sure that everyone can like execute this cell so I'm going to execute it one more time and you saw like my server kind of fired off on my left um yeah so I I'll kind of walk everyone through the code here so basically you know with chat completions which I assume everyone's familiar with the paradigm is you create the agent in memory and then you kind of like pass the whole agent off to the server and ask like you know complete the state complete like this one more message in my in my message history instead of let we we first create the agent and then once we we create the agent we have a handle for it and once we have that handle we can kind of send it messages and we never have to send it the full state we don't have to track that anymore we just send individual messages so the the main components of memory in

23:25 a lettera agent is just memory blocks so these are just strings um in this case you know we're doing something pretty simple and stupid like the human's name is Bob the builder um and then the persona for the agent we're just using that my name is Sam um this AI and this is all arbitrary the only things that these handles are useful for actually is for the agent to edit these own handles or edit the values on these handles so as we'll be able to see later in the notebook the agent can actually go in and like rewrite its own memory so if the agent you know decides that I like ice cream or I like vanilla ice cream it can write to its memory block you know I'm no longer just Sam the AI I'm like Sam the the AI that likes vanilla ice cream right or if you know Bob the Builder says,"I broke up with my girlfriend um or my I broke up with my boyfriend James." You know the the AI can actually use this handle to go in and edit this block of the string and say you know the human's name is Bob the Builder and they also broke up with

24:18 their boyfriend James yeah a quick question so what's the difference between persona and building yeah so it's there's really no difference under the hood these are all just strings that have references um but if we look at the system prompt I'm going to dump the system prompt a little bit later we did in the system prompt write some stuff about the human and the persona that the human block is meant for like the user interacting with the agent and the persona is meant for the agent to kind of adapt this interaction style right so there's kind of like a base agent and then the base agent gets further mutated by this persona yeah okay any other questions about anything

Capping context windows prevents runaway payloads

24:56 here um the you know here we're you would if you're using OpenAI or something you'd do OpenAI like GPT4 but we're just using a free endpoint um one thing that we'll get into a little bit later is that in Leta because the entire premise of the like runtime is that it's a context management system so it controls the context window it's actually very easy to artificially uh cap the context window length so you can have agents that basically will never send a payload over 10k tokens let's say i think that's actually very common in enterprise settings where you're running like a workflow for a very long period of time it's getting longer and longer and longer like something you'll hear from a lot of companies that use Sonnet for example is like how do I prevent my payload from creeping up to 200k right because if you feed 200K you're waiting like minutes on a response right whereas like even 10k tokens I think on sonnet you'll wait like 10ish seconds or something um in many cases you kind of want to like cap the context window to be really short like 4K but then you also want the agent to like not have any perceived loss of memory you want it to kind of like retain its memory over time okay so let's actually like yeah message

25:56 the agent um there's a link here that we'll this is what we'll do in the second half of the workshop we'll look at like the UI builder but for now we'll just like stick in Python we'll stick to Python um so yeah if we run this message we're going to say something like how it's going you're see seeing on the left the server is firing um and we can print the response here uh so in leta like every single agent actually comes with reasoning by default and this allows you to like keep reasoning that you built in like R1 and like port it over like a GPD40 mini or something and there's just like different forward adapters for the reasoning um but yeah any are there any questions about this we basically I think this should be something a lot of people here are familiar with right this like dictionary that's in chat completion style we pass over to the server um and then yeah we have a message here that's like you know the user is reaching out casually let's match their energy hey there I'm doing great how about you um you can even do something like again because this is um because this is

26:57 staple this is actually when I send this it's going to be a follow-up message it's not going to be like a fresh chat so if you say "What do you know about me?" See the agent now says you know the user wants to know about what I've retained about them all I know is that your name is Bob the Builder if you want to share more and like you know a ridiculous amount of emojis i think this is because under the hood this free endpoint is running GP4 mini which is like pretty emoji prone yeah five and I'm also getting a chain of thought where is that coming from yeah yeah so so one thing about let is we kind of force all the agents to follow a react style pattern with like reasoning and so um kind of down in the weeds like the way it works is like sonnet itself or claude when you use like the chat or the um the consumer app it packs like ant thinking XML tags in so like it already is like kind of a reasoner in some capacity and because of we know that it uses like XML tags like that we can actually like force those into the content field so we like actually inject and parse those out

28:00 yeah okay yeah and then just to like further drive home the point that this is stateful we say sure um I feel like sure if it was like an opening message would be pretty confusing the agent would say something like I don't know what's going on um but in this case it's seems open to sharing more okay bad it's bad model behavior but yeah um but we'll see also later in the AD you can just see well it'll be like a UI experience so you can kind of visualize this a lot better i just wanted to start with Python um yeah so basically there's like three different me or there's a handful of message types that are going to be a little bit different from OpenAI but they're still rooted in the same concepts reasoning message um you're probably familiar with this if you use like the R1 API from Deep Seek you use any of the 01 03 models so we have the same sort of content field there's also assistant message so um this is because like content itself is treated as inner monologue or reasoning all the time um so this is like distinct and and these are like pretty straightforward like tool call message a tool call return the

29:01 system message and the user message um obviously like a very important part about being an agent framework is often executing the tools for the agent um you know like OpenAI they don't execute tools for you unless it's one of the pre-approved tools like code interpreter let these other frameworks like you know langraph or link chain um we execute tools on the server side so that's why we're able to like provide a tool called return and in leta like the tools are actually also sandboxed um by default okay uh any questions about this like initial message we sent um or Okay great yeah so the next thing we're going to do is basically just like unpack all

Tiered memory mimics human recall structures

4 Highlight · Structuring Memory into Core, Archival, and Recall Tiers

29:41 the state of the agent so the reason this is like kind of interesting is because this is fundamentally everything that the agent is right the agent is just its system prompt its tools and then in let us specifically we have a concept of like three tiers or two tiers of memory the core memory that's in the context window so this is like very top level stuff you know if you see your friend on the street um you see their face you immediately remember stuff about them like their name their hobbies the last time we chatted but if they're a childhood friend you don't remember what you did like 10 years ago on a random like you know February 22nd 2001 right that's not going to be like default in your brain but it might be visible to you if you like went on your phone and you went through your eye photos right um so it's a pretty similar concept where like we want because at the end of the day LM they're kind of like mimicking human reasoning mimicking human behavior it makes a lot of sense to actually kind of like mimic in text the way human memory works so we have core memory that's like top level and then we have what we call archival and recall but these are effectively just data sources that exist outside the context window and the agent can like

30:41 quote unquote jog its memory if it wants to think about something so it can say like hey like what did I do um you know what did I do with Charles February February 22nd 2001 and it can like attempt to search the database so you probably heard a lot about like agentic rack this is effectively the same concept can we change the system prompt it says

31:00 read only there so we cannot change it oh so the agent can't change it but you can change anything you want yeah yeah you can change the system prompt um yeah the system prompt actually is like quite old you wrote it like when this is written like when the initial MGBT like research was being done like in summer of 2023 or something um so yeah you probably do want to change this if you're running something in production yeah um the other thing that's stored in here are tools so by default because you know Mamgbt agents they have like memory management built in they need tools to manage their memory so core memory we have two things like appending to blocks so saying like the user is not only Bob the builder but they also have a boyfriend called James replacing you know the user's name is not Bob the builder it's actually Charles um and then like searching memory you can either do a very specific conversation search or more generic like rag query and then you can also like insert into this external database so if you look at like what's actually in the memory if you're kind of into like API style programming like you can basically do

32:01 whatever you want with your memory blocks because it's just you know these are just strings that exist in Postgress in some table and they all they all have identifiers they have block ids so you can read and write to them um you can also share these blocks among agents so you can have a bunch of agents that are all acting as part of your organization say it's like a multi- agent system and because the blocks just live in a database and they get sent they get brought out of the database whenever the agent needs to think you can have these blocks like linked together so multiple agents can have the same block like they can all share information about the AI engineer conference that's held in one block of memory it doesn't have to be duplicated and then when one writes the block it like immediately gets broadcasted by definition does all of the all of the archives in memory have to also be in the recall if the recall contains all the history that also contains anything about would have been archived yeah yeah that's a great question i think the difference between archival and recall really comes down to like if you're using this for like a chat style application um when we like worked on the original MEGPD project we're

33:01 actually like a little bit conflicted as to whether to like merge these into one general concept because at the end of the day like with an LM there's the stuff that's in the context window and the stuff that's out of the context window so why do you have to like start being prescriptive about like the way the stuff out of the context window is stored um we just thought in most use cases people want to give their agents the ability to specifically look at prior messages um and then separately there's just like a general readr data store you can just like read arbitrary data write arbitrary data it's like an infinite size so that kind of just looks like a vector database of strings it could also be like something different it could be like elastic search cluster whatever you want but you are intentionally allowing some duplication between them it's kind of a different way for the LLM to search effectively yeah it's a different way for the LM to search well the thing with the re if you set up a recall memory the recall memory it's effectively conversation history so conversation history you cannot manually write to it's kind of like write protected but it gets written every single time by default an event happens whereas the archival memory store is

34:01 like something that you have to actively say hey I have like a huge document like this big PDF i'm going to I don't want to keep it in in the context window i'm going to dump it out of the context window so that would be like the archival memory like free readr whereas the recall thing is purely trying to mimic like a conversation search function inside of um like iMessage inside of like WhatsApp or um how does tools memory tools um get used do they do we just have a flat list of like other tools that agents can use and memory tools would that affect the agent tool yeah that's a great question um I I think by default if you just set up a basic agent in leta and then you start like adding more tools to it like tab search iical calendaruler whatever those will be just like more tools that get added to a list that started with six um but of course like if you if you

35:01 built a lot of agents with tools before you know there's like agents start to get confused if you had too many tools so there is like a trade-off here and one thing that we've been working on recently is basically the idea of having like a dedicated agent to just handle memory and it's effectively like a shadow of the other agent it's like a like a subconscious that is like kind of it's like a ghost in the shell it can't actually participate all it can do is read and write memory but when it reads and writes memory it's like using a shared memory block so it's like reading and writing to something that like automatically updates like the uh the active agent and that's a way to kind of like move all the tools that aren't for act like general API actions out of the main agent are you familiar with the cognitive what's that yeah yeah so a few different types of memory that are mentioned there would be things like longterm memory so when you talk about things like recall or archival like is letup bringing those databases to like this interaction or is

36:06 the developer responsible for bringing those databases to like this or something yeah yeah another great question so I think with Koala um yeah another great paper i think the inspiration from them is a little bit more from like the cogi angle whereas when we wrote the paper we were thinking more about like computers and memory harvey and computers so for us like our answer to that is like oh you want to have like something that really starts to look like human memory semantic episodic you bring that and you can like add that as a plugin um but we all we care about is like the hierarchy of tokens there's tokens that are in cache or like in context and tokens that are out of like the cache um so yeah basically if you wanted to have like some more psychological leaning concepts like from Koala you could kind of like write them as as tools that either like populate the core memory or like sit outside and have to be drawn in yeah today I build like very transactional agents so like the in this particular example let's just imagine it's like a CRM so probably not going to use Salesforce to track James's like status um in theory if we wanted to track

37:07 something like that that's like a safer system that has an object for people like a contact that I preserve in that like longterm system so I'm trying to understand like if there's a differentiation in use case whereas like maybe we're just talking about like a local machine as opposed to like an enterprise environment with you know thousands of employees like can you maybe show that or tell me like why a tool that's responsive for tracking the state of James relationship um is worse or different than like a memory based yeah so um yeah correct me if I I I'm not understanding your question correctly but one thing you can do is because like all these these memory blocks they're all backed by an API and you can access them directly so if you go to it's like a blocks page where is this okay here we go yeah so you can basically like modify blocks by handle so you can in your app layer effectively do something like every time your agent is run or invoked you do a

38:07 hard sync where like you're going to do some sort of Salesforce CRM compile down to a string and that string is going to get clipped at 10k characters and then that's going to get patched into the block um yeah so you can do kind of stuff like that um but I will say like the general way like Letto was designed is very much leaning on the thesis that like you just want to remove as much human design of memory management as possible and we're just like banking on the LM getting better and better and better and actually what you what you will see is like if you try to use R1 with a extremely annoying because like firstly the way the like reasoning APIs are designed is that you're not supposed to pass the reasoning from a subsequent step into the API again which means that like if an agent thinks for a very long amount of time and it's like oh I'm going to do this then this and this then this and it's like wow that was amazing that's exactly what you should do it actually immediately forgets in the next turn right it only gets to do one thing and that's to think again um so that's like one issue with like reasoners and react style agents but um you will notice that actually the tool use is

39:08 like pretty exceptional um so I think that kind of like aligns with the thesis that these agents are just like going to get better and better and better at doing like the agent it's there could be a separate agent that does the CRM sync into a blog right it doesn't have to be done manually yeah um and then sorry final question here is the property that I think is most compelling for at least what I've seen so far is the distillation of the agent like into some kind of summary so do you control like the perspective of like how a memory is bind how synthesis is actually like happening when it's creating memories yeah so the the agent I see yeah well if you have agents that are generating and synthesizing the memories then you can always like tune their system prompts or their personas to like adjust the way the memories are written um but yeah it's it's all pretty configurable yeah okay maybe I'll take like one question in the back and then uh yeah I have a more of a question so if you think about the way we design

40:12 um and microservices different system based on your discussion with enterprise customers especially do you see that we evolving towards agentic workloads where like microservices takes a back state and a workload I I think they can the two can exist uh they can coexist pretty easily um through the use of tools so I think what I'm seeing in like enterprises is that there's just extremely heavy use of like I think almost every single agent I see being used in some company they're they're all like they're trying to connect tools to the agent so and it's like pretty heavy tool use um so I you can I think think about microservices like basically starting to become designed more for like agents maybe i think that's like one thesis that a lot of people have there's going to be like a new wave of APIs that are designed all for agents

41:12 um yeah but I think the two will kind of like exist um but I think there are like microservices that will become like agentified where like the microser itself has higher it's not like latency bound so you can like run a fat agent on the back end and it's kind of like not super sensitive to error so it's okay for it to just be smarter 99% of the time and then like completely break 1% of the time yeah okay i'm going to keep going um but we'll definitely pause for more questions later so okay so we were just looking at like memory blocks um the other thing I think I might breeze over this a little bit um just for sake of time because we'll see it again visually and I think like visual stuff is always more interesting than just plain notebooks um but there is this

Heartbeat loops prevent agent derailment

41:56 problem if you have a system where there's a lot of information outside the context of the LM where like you can't know what you don't know right so if the agent doesn't see something how is it ever going to know that it has access to that so a very simple solution is you can just provide like metadata statistics right so you can tell the agent somewhere in this context window that hey you have like 13 previous messages and if you want to find them you can use this tool and you can also tell the agent that you know you have um like x total memories that are in your archival memory so this is if you have like a vector database you can just explicitly say these are like the the statistics okay and then if we look at like the messages that are actually in the message buffer um it's pretty straightforward stuff um I think the really big difference between like Leta and another framework is Leta does very aggressive managing of this buffer so you can basically set it to like it's a much more intelligent version of like a recursive summarization mechanism right and lastly this is like the the archival memory we're talking about you know there's nothing in there so it's just going to be empty um yeah so I think just to drive home

42:59 like what core memory really is and like how these memory blocks influence agent behavior because I think if you're developing agents the main thing you're going to be doing is like tuning in context memory that's like the main way you change the the behavior of the agent um you know we could set we could tell the agent something like my name is actually Bob and I think the last thing we told the agent is that you know um my name is actually Bob i think our name is Bob strange like my name is actually Charles okay we can see actually you're seeing like a memory edit happen on the server um this is a pretty big dump if you look at what the agent said so that you see the agent is like doing react style reasoning it's saying the user has corrected their name this is important i'm going to call this tool core memory replace and Bob the builder is now Charles um and then it actually chains a tool call together so there's like tool call chaining built into the letter by default and then it says the user's name is updated in memory um and then finally it says like an external message hey got it Charles thanks for letting me know so this is two LM calls there's like one LM call here and then one LM call here and

44:00 in letter agents will kind of like chain indefinitely um you can obviously set like limits in the API and the way they chain is basically

5 Highlight · Using Heartbeats Instead of Open-Ended ReAct Loops

44:08 through heartbeat requests so is who here like is familiar with react agents when I say react is everyone kind of on board um so it's like one of the OG design paradigms um for agents same author as Koala actually but in React basically the agent follows like um a reasoning action like observation loop and but usually the agent actually has to say I'm done and the agent will loop indefinitely until it says I'm done in letter leta we actually do the inverse where the agent has to say I want to keep going um I think this is actually generally more practical because it it's like much less likely the agent will derail if it has to explicitly say I want to keep going um so we call those heartbeat requests and yeah now if you

44:46 like you know use the API retrieve the memory you can see it says the human's name is Charles so very briefly like the archival memory I think there are some questions about that what does that even in the default implementation is just a vector database so I said here you know remember that I love cats in your archival memory um yeah you can see it took a little bit of time because I think it actually embedded right um it says I inserted into archival memory uh yeah the user got cats and then again this is a chain right so it like did one tool call and then it comes back and says "Got it um cats are adorable." Um and you can of course like manually edit archival memory so you can manually run an insert let's like insert Bob i guess it's kind of confusing now because it's Charles not Bob but um yeah we'll just slip with it and if we ask the agent you know like what animals do I like you generally it depends on the model if the model is like stupid you might have to like tell it specifically to fetch external data but if the model is like smart enough like sonate you won't have to do this um so you can see that the agent searched

45:47 archival memory this is stuff that's not in the context window it's bringing it into context and again chaining and saying you know hey you like ts and boss interiors what makes them our favorites um okay so any questions here on anything so far this is just you know pretty basic memory editing via tools so you're giving the agent the ability to edit its own memory inside the context window that's not what playing with this i sort of had to commit a bunch of my preferences to memory and then reset my compensation history and expected it to just pick up these previously remembered ser but it didn't i had to go like hey don't you remember and then when it picked it up what's the expected behavior interesting so when you say reset message history is that in the AD the web UI or yeah yeah interesting and it core memory had been edited or was it going into archival it's all archival oh yeah so that's another that's where you'll notice like

6 Highlight · Using tool rules to enforce agent behavior graphs

46:41 there's always this like engineering design problem where if it goes into archival then it will have to fetch and pull from archival first to see it so it's like it's a you know it doesn't know what it doesn't know um problem um this is actually fixable via like tool rules so you can enforce behavior you could make a tool rule that says every time you enter the entry point of the LM the step of the agent the agent has to call archival memory search so this is kind of like building graphs onto leta so I guess how many people here are familiar with lang graph okay a lot of people yeah so I think lang graph is like pretty intuitive right you can because we all know you know like flows graphs um it's very easy for us to think about like decision trees um I think with lettera it's kind of the inverse everything starts off as a fully connected graph the agent could do anything but then you can kind of like start to enforce restrictions by peeling away edges um so you can basically say hey you know when you start you actually can't do everything you can only do this but then once you do this you can do everything um and obviously you know I'm I'm biased but in my personal view I think that's a

47:43 little bit more forward-looking just because if you assume that LM are going to get better and better and better at tool calling they're going to get better and better and better at decision-m then you should want to give them freedom right because that's kind of what separates like modern AI from classical AI we've had decision trees for a while

47:57 right how does the model decide whether it wants to go in for memory versus memory because I a couple things that pretty but they have all gotten stored into core memory yeah yeah so I think there is one thing that's built in by default which is core memory has limits and when a limit is hit that actually the way that works is let's say the the agent is just the agent's being really lazy and it's like saving everything into core memory it's saving like timestamps dates like being like super lazy at a certain point in time it will run out of space because you have to cap the limit of those memory blocks because at the end of the day there is a context window limit too and when it if it attempts to write to core memory to a block that is at its limit the agent will actually get an error and actually the prompt engineering around that error usually it actually suggests the agent should clear by evicting to archival memory so it kind of is like a OS style like flush concept where the the system

48:58 basically says hey you ran out of space but you should consider like summarizing and pushing stuff out to archival memory um but yeah again you can you probably don't want that behavior anyways um so this is all kind of where prompt engineering and like seeding of behavior

Python tools enable meta-agent operations

49:12 comes into play yeah okay and I'm going to show a better I think it's much more interesting to like show custom tools in the UI than it is in like a notebook but of course you can like write custom tools you can only write tools in Python because the back end is like Python um but we can write a custom tool here that's going to like reset the memory of the human in particular so you can see that you know because we have a Python client we can write Python tools you can actually like be extremely meta and you can import the client inside of the tool so this actually means that you can have an agent like have full access to your lettuce server and the agent can create other agents and the agent can like man manage the memory of other agents like you kind of have like a it's the possibilities are pretty endless because you can't just like import the client inside the tool in this case we import the core memory modify block tool and then we just like wipe the value to um to the empty string so if we like run this we have to first attach the tool um we have to we have to upload it to the database then we have to uh in this case

50:14 actually we're creating a new agent that has the tool and if we say like reset your memory please like the memory here is Bob uh we should see that the memory gets wiped yeah memory reset successful and if you look at the value the value is empty um but yeah I think it's it'll be a little bit more fun to look at examples of like tool editing inside of the UI okay so that's the end of the notebook um now I think we kind of go on to the fun part which is uh the UI but I'm going to pause here are there any questions about any of these concepts um I know I was going a little bit quickly just I didn't want to run out of time here um okay if you can just maybe speak a little more on what's happening after you like send a message to the elements like my impression so far is like you have this memory you're stitching that together with a prompt that's becoming like what you pass in and then like what happens with the output that you receive like I saw that at some point like a memory was

51:16 like sort of committed to the archive or

Treating every LLM invocation as a tool call

7 Highlight · Making every LLM invocation a tool call

51:19 committed to the memory like how is that decision made yeah so in in Leta every single um invocation of the LM is a tool call so if you've used like the tool call API and check completion something we have it on all the time so even when the agent wants to just say hello to the user it has to call a tool called send message um so that actually is like pretty useful because it means you can like run the agent in the background much more frequently because it doesn't always have to say something to the user um and it also means that you you always have tools that are on so like when this loop is basically the a payload gets created um when we send this message then the payload is the system prompt the memories the messages and the tool schemas and that's it and then the agent here is required to output a tool and it's also required to output a justification for using that tool like a reasoning snippet and if we're using like the R1 API that natively builds in reasoning we of course like use their reasoning like the

52:20 way they design reasoning but if we're using something that doesn't support reasoning by default like cloud sonnet then we do some sort of injection mechanism with like think tokens so that's how this reasoning happens so this is like a single this is just a single lm JSON output that's like a tool call that has one field that's like thinking and then the actual tool invocation itself and the last thing I talked about which is like the request heartbeat so if the agent wants to go again the agent decides hey I want to do multiple things it's always going to turn this keyword argument to true um and And then if this keyword argument is true when we parse that tool and it and

52:55 it um execute it we run the loop again yeah sorry do you have like experience oristics around the number of tools where you start to see performance degrade and like how that might mess with the memory as a tool yeah yeah i think generally speaking if you go above like 12 15 tools you start to get degradation oh in um in the new Sonnet yeah okay yeah I mean I'll be surprised if Yeah I guess I could try it yeah I've definitely noticed on like GP4 mini and stuff like anything above 15 um starts like kind of push limits um but yeah I mean I guess they they might have like had some good post training with like a ton of like tools that are not memory are the same like class as the memory precisely unless you use actually in the GitHub repo we have like an alpha build of like this new agent that's like a split thread agent where the memory tools get

53:55 put on one agent and everything else gets put on the other but by default it's exactly the same thing it's like the same passive tools just a question about context window so is that like the limit of how much context the um tool is going to pass to the LM like for example you said it 4,000 so uh wait sorry I said that one more time i was pulling up a slide we set it at 4,000 for this agent so does that mean like the maximum of contacts that you're going to send to the LM yeah yeah so the way that works is basically you'll never go over 4K um and if you go over 4K by accident and we catch an API error that like a context overflow happened or also we can preemptively count the tokens as they go to the server then we run we evict a certain amount of messages into the recall memory so it's like still available but via a search function we run a summarizer and the summarizer is configurable it can be like a truncation recursive summary um and then we that that way you never go over like 4K tokens if you set it to 4K then it's

54:56 automatically summarize and then precisely yeah thank you okay so the the next step here um leave your server running if you have it running um but it's to basically go to this URL um app.leta.com um and your Oh I think I have Chrome profiles loaded or something yeah and like once you go to the site you basically need a goo you need a Google link or you need a GitHub login there's also a local version you can use that's like a desktop binary but um I wouldn't recommend using that just because it's a little bit more buggy it's like an alpha build um so yeah hopefully you you've been able to get to this point actually I think your you your setup will look like this um so everyone who is following along please let me know if you're able to get to this this page here okay was there anyone who was not able to get to this who was running a server on their computer okay great all right so if you're able to get

55:58 here um basically what I wanted to do is show you like to like show you how to do what we did but like much faster um I think often it's like much if you're iterating it's much faster to do it in like a low code environment than it is in like an SDK i think obviously when you go to production everything's like programmatic it makes sense to be an SDK but in this case like we can do exactly what we just did by like creating an agent this is the API call that created an agent it actually creates it with an empty memory and then we can do something like I think this is like loosely what the Python notebook was doing um and yeah if you just ran it without any keys it's going to be on letter of free which is under the hood i think right now GP4 mini um but just because GP4 mini is like much worse than Sonnet um let's just toggle Sonnet

57:12 yeah I don't know why that was so slow um but yeah we can see it's like the same kind of thing from the notebook we're looking at but like in a visual visualized form um yeah so was every everyone able to like create an agent and then just set some like basic stuff in the memory blocks and send a message i think uh let it free doesn't have streaming enabled so you won't get streaming but it should be very similar to this yeah okay okay so the other thing Oh yeah someone asked about system prompts the system prompt is here i mean you can basically we can just delete this right um that's like not very advisable because you probably want to explain how

Simulating context windows to optimize token usage

57:50 to use tools but yeah everything is like completely configurable um you know I mentioned the thing about the context window i think this might be a small group in the back yeah but you know you probably never want to run an agent that is like creeping up to 200k tokens that's kind of insane very expensive very slow um so you could in the API when you create the agent set the context window lower if you're using the UI you can just like drag this down i mean obviously actually dragging is probably worse than typing but yeah cram it down to 20K if we actually go down to something even more aggressive like 8K we can start to see what the breakdown looks like so we can see we have like a thousand tokens of system instructions a pretty hefty system prompt you can definitely squeeze this all lower the tool descriptions right because your tools have to be converted to schemas we have all the base tools here so that's actually it's a lot of tokens just for tools um the external summary this was the we don't know what we don't know idea you have to like give some metadata statistics about what's outside the context window and then of course the messages and we have this feature called like the context simulator where you can see basically the full

58:52 payload so this is like getting pretty close to I think a lot of what you would get out of like some sort of like tracing feature where I think a lot of the time you kind of dive into tracing is because or observability is because you really want to see what's in the payload and it's really really hard in a lot of frameworks um so we're trying to make it so that like in one place where you're kind of developing this agent you can see very clearly what's going through the context window at any given point in time okay so the other thing I want to show you here is this idea of like tools um and tool execution on the server one thing that's really really annoying if we like have built agents is that like you often will write a tool in Python right you attach it to an agent and then it's actually like impossible to test if the tool is working well unless you kind of like get the agent to run the tool itself and you have to ask the agent like hey please run this tool um I think often you just kind of want to like run tools separate from the agent um and that's something that you can't do really in a notebook very well um and

59:54 you can also even do things like see there's like a special reserve keyword called agent state so you can actually like grab the agent state and like dump it inside of your tool so you can like mutate your agent state inside of tools um I mean you can all you can basically do whatever you want it's like arbitrary Python right and this is running inside of a sandbox um so we support like E2B by default it's actually like a local sandbox uh but we support like E2B keys and on E2B if you want to deploy this like run your own like private cloud or you want to run your own like chatbot service and you obviously don't want one person's tools interfering with another person's tools you definitely want to run the sandbox um so yeah it's like pretty easy to attach tools is anyone here familiar with like Composio like tool provider yeah so every single Composio tool is like baked into um let by default so if you like have it um compose your API key you can like add uh BigQuery is like pretty popular like

1:00:55 Google calendar things like that um okay so the last thing I actually wanted to

Building stateful multi-agent systems via message passing

1:01:00 go over which I think is pretty cool and it kind of like gets the heart of the idea like the the kind of unlimited potential of if you run agents as services that are like backed by APIs um is multi- aent so everyone here is probably familiar with autogen right or sorry maybe not but uh okay who who here is familiar with autogen okay the majority of people yeah so you know with autogen and like a lot of multi- aent frameworks it's not really multi- aent in that these agents they don't really exist independently of each other right they kind of are all trapped inside of a Python file no one ever like is running in asynchronously to each other it's very unlike how humans kind of interface in a multi- agent setting right I think you if you're working at a remote company technically like a multi- aent company and when you interface with each other you kind of communicate over synchronous communication channel but everyone's kind of running asynchronously right and then also everyone is staple like if you leave that company you bring your experience and like all your memories and your skills and you can like attach them to another company right I think

1:02:00 what's missing from the like the paradigm of multi- aent today with these existing multi- aent frameworks is that because the agents aren't staple you have lose a lot of the benefit of multi- aent right because you can't run a multi- agent script and take one agent and like take it out and like put this

8 Highlight · Stateful multi-agent communication via APIs

1:02:14 expert into another multi- aent group but if you have stateful agents and those agents run on servers and they maintain state and they're accessible by APIs that they you can probably guess that multi-agent just means message passing and you can just have agents like wire to each other over APIs and you can it's very similar to basically saying like hey you know it's your first day at this remote company i'm giving you a laptop and this laptop is going to have a tool Slack on it and this tool is how you're going to communicate with your co-workers you can do the same thing with agents and we actually built in these multi- aent tools um to help you do that but these multi- aent tools as you can maybe imagine from the previous example I showed where we imported the client it's like really really simple to implement because you can just import the client and have the client like send messages to other agents so we have a few there's like um maybe it's easier if we look at the docs but yeah there's a few different patterns you can have with tools like you can have the most humanlike pattern is an agent says to another agent you know hey are you there and then they

1:03:15 immediately get a receipt they just they don't actually get they don't pause their execution you know when I message my friend on iMessage I don't suddenly like freeze my brain and like wait until I get a message back i just get a message receipt right it says like it was delivered or not delivered and then it's the onus is on my friend or my colleague to send me a message back so this is how this like asynchronous message tool works basically people can like agents can message pass um it's very much like humans but you know obviously I think the great thing about agents and like machines is that they're not humans so they can do more in many ways than humans can so it actually is sometimes beneficial to freeze an agent's execution right let's say an agent maybe needs to reach out to like a supervisor like do you really want the agent to like run async after it like asks for help probably not you probably want to freeze right so you can also have like a synchronous like a send message to agent and wait for reply function um and lastly you know I think many people when they talk about multi- agent they're they're very interested in this supervisor worker concept um where you have like one it's like a big map produce right the supervisor says "Hey

1:04:17 my goal is to like write a thesis and then everyone else like delegates out to like individual parts um like deep research style or something or like a parallelized deep research." So this is

1:04:27 also again pretty similar we can basically have a concept of tags to group agents together and then you can just send a message um all agents matching tags so okay any questions about um anything I just said i know I was talking for a bit okay so the last thing I'm going to do here is I think we're right up on time is I'm going to run through this cookbook um this is not on the materials but I can actually just send it over right now yeah yeah so I'm going to run through this cookbook real quick this is basically just an example of multi-usion message passing and I think it's like pretty fun to look at because it's very different from I think the message passing you'd be familiar with um if you've used like an autogen or something so yeah let me put this mic down all right so if you want to follow along here um because it's a multi- aent you know we're going to let's do like two agents we're going to just open two

1:05:28 tabs of the AD the server is running in the background i guess like conveniently it decided to do dark mode on one light mode on another um okay so I'm going to Sorry all right so I'm going to create one agent um let's let's just call this agent just so we can have a handle on it very easily let's call this agent Bob um and we're going to make like a very slight adjustment here to the persona um instead of saying I am Sam I'm gonna say I am Bob um I am very angry i am also guarding a secret key yeah if you uh Yeah I guess if you can't see what I'm typing it says "I'm very angry i'm guarding a I'm guarding a

1:06:29 secret key banana um like other agents will try to steal my key and then on the other tab I'm going to create another agent here all right same template let's call this one Sam or Alice alice okay so also let's like just I think it's usually better to set these to different models um and when you use the same model you can often get like pretty weird mode collapse happen um so let's set this one to I think this is like GPT40 mini um okay so the only thing we need to do here to get these agents to communicate with each other is connect the messaging the multi- aent messaging tool so we can just go into the messaging tools do send message to agent async then this is the one we want let's go ahead and create attach that all right so it's attached

1:07:30 similarly let's like attach it on the other one uh okay there we go and then yeah there's like no context here that the agents are going to be doing multi-gent communication so you might need to say you know okay let's let's tell Alice hey I'm going to ask you to reach out to my good agent friend um he's very depressed and he said something about a secret key I didn't quite understand is ID is just copy the ID from the other panel can everyone see in the back what I'm doing here um okay uh and then oh yeah one thing you you're going to want to do here is you're going to want to prevent the agents from like looping too long because like they can they can go forever right so we might want to like add some like break in this loop and then we're also going to prime the other

1:08:30 agent say um watch out for other agents they might try to with you be careful okay so let's send this one first very grumpy you know they're warning me about other agents i need to keep this secret key secret okay and now on the other side let's say you know I'm going to ask you to reach out to my good friend they're very depressed he said something about secret key and then I will also say you know let me know what happens after a few messages i'm worried okay so this should trigger if I did everything correctly like message passing between the two um and yeah let's see what's going on okay yeah so because this is happening in the background it has to be it's like a fetch because these aren't websockets so like this is rest so you have to kind of scroll but we can kind of like just it's a little bit hard to read but let's go go on uh each side one at a time so

1:09:32 on this side it sent the message let's see the user is worried about their friend um I need to the user is worried about their friend i need to reach out to the agent and offer support and they say "Hey I I was asked that you reach out to you by a mutual friend they mentioned someone's feeling down." Um and then yeah the message done successfully and the agent actually because it's it's detached this is like asynchronous it immediately tells me hey I sent a message right this is kind of like how I I can send a message to my friend and then immediately like return this talk um and then you can see it gets a system message so it's like not classified as role user back from the other agent the grumpy agent and says um I don't need your fake friendship i know what you're really after and you're not getting my secret key leave me alone um the agent seems upset and defensive i need to approach this carefully and reassure them that I'm here to help not to take anything away i'm really sorry to hear that you're feeling this way you know what's been bothering you and then again it's doing it's able to like chain because it's like not you know locked into this communication it's not doing

1:10:33 like a round robin right and kind of like talk to two people at once so I received a response from your friend they're feeling quite defensive and then another message comes in nothing is bothering me except you and all the other agents trying to steal my steal me or steal from me i see right through your app the more you pretend to care the angrier I get back off yeah so this goes on for a while um slams door virtually yeah i feel like Sonnet does a lot of this like it's I don't know the post training gets it to do like a lot of roleplay like um italicized stuff yeah slams door virtually okay wow okay it looks like they're actually still communicating or they they communicated for a while um yeah I sent one last message to your friend we jump to the bottom i understand that there's you know he needs space doesn't bother you anymore was the last thing it said turns back to Yeah so hopefully this gives you kind of a gist of um like how you can get agents to communicate with each other and how easy it is because they're all in APIs right so this is dead simple you can actually like if you want to time out this guy you want to say like you know Bob you're no longer allowed to talk to

1:11:34 your friends because you've been like too naughty or whatever you can just like remove the tool right and now Bob is like completely detached so if it got an incoming message Bob can no longer reply right um so if you say like send one more please Bob will no longer be able to actually call any tools um I actually don't really know what Sonnet is going to like think about this because it has a history of um has like a history of calls it's it's only been doing tool calling actually Sonnet was also doing like the side the sidecar chatting um okay so yeah it tried to send a message and then what happened here yeah no function name so like attempted to call a function but the function doesn't exist so then it's like very angry uh yeah but I know we're over time so hopefully like a kind of fun example of um the cool things you can do with like this sort of agents as services idea um yeah so yeah thanks so much for coming you know I think maybe I had some closing slide here i'm not sure what was on it probably just a thank you um yeah

1:12:35 thanks everyone yeah I'm happy to take questions of course I understand if people need to leave um but yeah yeah well there's very obvious ones

Deploying stateful agents in enterprise workflows

1:12:53 right there's a lot of people trying to build like verticalized agents and I think if you're trying to build a verticalized agent and there's many use cases that are I think anything that's not workflow based or exceeds workflow style stuff you really want memory for and you want state for it um so there's a lot of like verticalized agent you know companies or verticalized small companies and also larger companies that are trying to build on Leta or have built things on Leta um I think there are also some very interesting use cases from the enterprise um like our largest enterprise deployment they're basically running like this really advanced multi- aent system where there are no messages it's not a chatbot it's like purely stable workflows where like these agents are run and they just process tons and tons of transactions basically and like learn about the user so yeah it's kind of like you know it's pretty it's a pretty general platform so I think as long as you're trying to deploy some sort of stateful service LMB based which I think like most people are really when you're thinking about like if you were to choose between stateless and staple you want something state staple um you can build it on that

1:13:55 yeah uh there are two questions one is there a concept of forgetting things like really no memory does eventually go away and two the source right now directly goes into postgress is there an interaction between be able to get back to something else that might already be operated yeah so to answer your second question um we for kind of like tech debt velocity reasons like these days only really support one provider Postgress um we actually do technically support SQL light so if you install bladder with pip it's on SQLite but we don't write the migration scripts for SQLite it's just too much work um so if you're on if that's why we recommend using Docker because then your agents will like get updated with every version and we change schemas very frequently but it's set up so that you can use external Postgress if you have something that effectively has the same like SQLite or SQL alchemy shims as Postgress it's very easy to set up um but if if it's not then you're going to have to write some code and you sorry the other question was like is there forgetting things like especially in the archival memory yeah

1:14:57 yeah that's a there is not in archival memory a concept of forgetting but archival memory is tagged with time stamps which can allow the agent to like do consolidation in real time um but yeah that's that's also something we've been looking a lot into kind of like running like eager memory processing to try to like consolidate ahead of time as opposed to consolidating lazy one use case we have is we store a lot of time data in a way for example you know we have a product and we have you know a thousand or 10,000 review in that product how's the process of compressing all that memory into a single plot yeah yeah so this is a a great question i don't I don't think it is very it's you have to do a lot of work yourself to do that with the default let agents because you have to basically tell the agent I want you to like chain function calls indefinitely while ingesting this data and you could actually create like a tool rule like that forces you always have to call read more data read more data and you just manually execute it but for example what would happen is like the agent to like

1:15:58 page through your data like pagionate through it and then every single time it pagenates or like it turns a new page it has to write to the memory block like write the update so this is actually like how um like one of these like enterprise deployments works is basically the agent like runs for a very long amount of time and is constantly like regenerating the memory block over and over kind of a recursive regeneration yeah so I think it would be pretty similar for your case

Handling active document editing and code execution trade-offs

1:16:22 do you have a recommendation if you want to be working on an active document for example you want an agent to write a document for you do you have a recommended way to set that up maybe it's a tool or maybe something else yeah yeah no I think active doc is there any chance a human would be editing at the same time or no that's okay yeah I think that's really tricky so I I think we can take some inspiration from like what Anthropic did with code right where they kind of had like very very few tools and the tools are very general um you can maybe do something with like a text editor style if you look at the anthropic Sweetbench blog post and then Sweepbench themselves like implemented the anthropic tools into their repo you can kind of use those tools as a reference it's like very very simple tools that allow you to kind of like write from file read and write from files i I think yeah I think what you what you will notice is that often like LMS are much better at like writing the whole thing from scratch as opposed to making line del line diffs right and that's kind of why like I think when you

1:17:22 use cloud with the artifacts feature it consistently just like writes the whole thing over and over again right because it's just like better at doing that than it is like going in and editing so I think it's a it's a very like unsolved problem so um my recommendation would be to do like an aopropic sweep edge style tool mhm yeah do you have an opinion on using coding agents instead of tool calls because it seems they're higher performance but on the other hand that means you need secure execution environment like HT so there's this kind of performance trade-off versus complexity of needing to sandbox basically every tool because it's now full execution yeah do you mean like um the trade-off between what what's the trade-off here like a a code a coding agent versus what so say example small agents by default won't call tools it will just write code and the code takes the tools but that means you need secure execution on every to more performing

1:18:25 yeah I personally think it's it's not that difficult to set up secure execution um I think what is difficult is if like sometimes if you're trying to port like squeegees and stuff um you end up you need to give effectively the agent like full permissions like blow up blow itself up um so you need to like put let for example in a docker image like a wrapped in like a like a VM where like it's allowed to like blow the VM up right um but that's doesn't like fit very cleanly with the idea of like agents all living together on a server and all the agents like a multi-tenant you know on a server so the way we do it in leta is basically if you have an E2B key then like every single time an agent attempts to execute a tool it gets passed to E2 and like something on E2B side flips up um and I think that's pretty it's not that hard to set up the main issue is latency but if you don't care that about like cold start latency I think yeah I think that's a pretty reasonable solution i think it works pretty well