The Skills-First Agent Workflow: Reusable Agent Intelligence

9,489 words · about 38 min to read · YouTube’s automatic captions

Agent skills as reusable software units

0:00 There's a lot of hype around it right now. Skills, first of all, this isn't actually a new thing at all. Cursor rules, I can't even remember when that came out. Claude skills, again, something that's been around.

0:09 So skills are basically a standard unit of context to provide to agents that teaches them to do something. So to acquire a skill, sad reality that people are copying skills all over. They're designed to be reusable and yet we kind of copy and duplicate them and copy everywhere. So that's not awesome. When we think about adding skills as a first class citizen into our organizational process, what do we need to consider?

0:33 You have to think about how do you own them longterm. You're better served by thinking of skills not as a markdown file but as a unit of software. You have to think about how do we maintain those skills? How do we keep them up to date? And so kind of the whole life cycle management of it

0:45 really exciting that Tesla last week released support for skills. There is coverage and functionality for evaluating skills, distributing skills through a package manager style mechanism and also support for skills in a more complete life cycle way.

0:59 And there's all of those combined to just create this flywheel and now there's huge excitement over something that we've believed all along here at Tesla which is context should be reusable and there's a difference between intelligence and knowledge. Before we jump into this episode, I wanted to let you know that this podcast is for developers building with AI at the core. So whether that's exploring the latest tools, the workflows, or the best practices, this podcast's for you.

1:31 A really quick ask, 90% of people who are listening to this haven't yet subscribed. So if this content has helped you build smarter, hit that subscribe button and maybe a like. All right, back to the episode. Hello and welcome to another episode of the AI Native Dev Podcast. My name is Simon Maple. I'm your co-host for the for the episode. And joining me

1:52 is Guy Pjani. I'm uh also co-hosting here, CEO and founder of Tesl. Wonderful. And today is going to be a nice special episode where we're going to talk about something which is there's a lot of hype around it right now. Skills uh agent skills just general uh skills that we're going to be talking about. What is a skill? How should they be used? When should they be used? Uh what is a skill compared to other types of context that an agent might may or may not use? Um, and also what are the certain things that we want to that we need to do to make sure a skill is right, a skill is good, a skill is accurate. Uh, and also some sneak peeks into some of the, uh, some of the great things that Tesla announced last week on our support for agent skills and skills generally. So, there's a lot to pack into this episode, guy. Let's get started.

2:34 Yeah, lots and lots to talk about and

Structure and progressive disclosure in skills

2:35 skills are they're all the rage.

2:36 Absolutely. So, why don't we just start why don't we just define what what is a skill guy? Yeah, I think that's sort of an interesting question coming from you. It's like I sort of thought that you're sort of a skillful person.

2:45 No, I'm I'm a highly skilled person, but I struggle to define it. Lots of people find it hard to put their finger on it. What is maybe you learn something? Yeah. No. So, uh skills are

1 Highlight · What are agent skills and progressive disclosure

2:54 basically a standard unit of context to provide to agents uh that teaches them to do something. So to acquire a skill um they have initially been introduced by Claude as part of sort of you know Claude skills capability. uh although they were very very very similar to things like cursor rules or or other kind of a built-in guidance uh that that existed in other agents before um but still claude skills kind of paved the path to definitely this as a content uh and effectively it's just a standard file structure that defines a skill.md file that has some definitions of you know what this does and some metadata but when should the agent use it uh alongside with supporting files designed in a certain way so that the agent can kind of acquire that skill, use it and and do so in a in an LLM wise way. For instance, a core element is this progressive disclosure of having uh sort

3:50 of bits of knowledge that the agent can traverse through so it's loading the right information at the right time and doesn't overload the context window.

3:57 Yeah, a couple of interesting things that you said there. First of all, this isn't actually a new thing at all. You mentioned cursor rules. I can't even remember when that came out. That was that was a while while ago. uh uh Claude Skills again something that's been around for for a little while since

4:10 September I think September. So it's interesting that it's kind of like you know heating up right now. Uh another another thing that kind of really interested me there is when you talk about the skills MD it's just plain text again. We're just talking about you know you're passing some text to um to an agent that an LLM reads that text and then does things based on that. Um, so there's a lot really in terms of us being skillful as well about how we construct that, how we write that and and I think we'll talk a little bit about evals. So why have skills all of a sudden become an overnight success and everyone's talking about it and everyone's releasing various skill support. What what's happened?

The open standard driving skill adoption

4:46 Well, the the key thing that happened is similar maybe to MCP is that it's become a standard. So Anthropic very late last year uh have kind of introduced an open standard. Standard is a bit of a funny term. they've kind of you know coined one uh they called agent skills and they published the the definition of it uh and a little bit similar to what we've seen with MCP the rest of the ind industry lined up actually fairly fairly quickly and so now within a short span of time maybe within a month you've seen comments from cursor and codeex and uh uh I think Gemini and others about supporting uh skills themselves uh and what that does is it suddenly gives the users the creators anybody that's, you know, actually kind of looking to use agents gives them an ability to create one thing, the agent skill, uh, and roll that out. Uh, and so you've seen at the beginning of the year, you've seen cursor make statements that they, uh, will embrace skills and actually phase out rules. Again, they're very similar in in many of their traits. So, they

5:47 will phase that out. Uh, and you've seen kind of this explosion of uh, uh, of ecosystem. The other very smart thing that uh that uh Anthropic has done is that they created a create skill skill that uh just made it very easy for people to start accumulating it. Uh and maybe last important thing is that uh skills actually were initially introduced in claude less as a development thing and more as an ability to to create some instruction for non-developers when you're when you're using Claude to whatever create an Excel sheet, right? or you know perform some type of a of a musical review, right?

6:23 And so now uh uh like that that uh that approach is true for the sort of the skills in agent skills as a whole. And so it also introduced um kind of a lot of non-developers to kind of this opportunity of creating kind of reusable artifacts uh that they can share across the team. That has been very very exciting or like reusable workflow. a little bit of a taste of software development style capabilities of reuse to the non-dev world and all of those combined to just create this flywheel and now there's huge excitements over something that we've believed all along here at Tesla which is context should be reusable it should be you know something that uh you don't you don't want to reinvent the wheel you know and there's a difference between intelligence and knowledge

7:05 yeah oh absolutely yeah good nice nice little uh nice little takeaway there and I think so let's go into a couple of things there first of all I've actually been doing a little bit of work with some of the folks at codeguard. Uh and and it's a it's a uh an amount of context which is talking about how you can actually build how you can review code uh in in the most secure way following very specific uh security uh rules. And I think one of the interesting things or one of the one of the things that they saw as a big drawback was the fact that if they want to produce this they know that the people who are consuming it some are going to be using Windsor, some are going to be using cursor, some are going to be using claude. And so they actually need to structure it in multiple ways so that every different agent can consume it. And I think this this creating this standard that all agents can kind of like build from allows you to effectively be able to deliver one thing that fingers crossed hopefully as long as everyone starts supporting it that

Rules, skills, and documentation in context engineering

7:55 people can then just kind of pull into. I think that's a really important piece. Um let's talk a little bit about context and context engineering and where skills sit in that because really skills are you know they're a part of overall context engineering. What would you say are the differences between uh skills specifically and general context that an agent or or an LLM can use?

8:16 Definitely there are many uh ways to basically introduce more words into the context window of the agent. You know we need to remember that agents eventually still remain you know just interfaces to the LLMs. So every time the there's a request to the LLM and the question is what what is the data right? what is what are the tokens that get put into that? Uh and and so all of these different means of context engineering, they're just about giving the right sort of path to choosing the right words to include in the message.

8:47 Uh so in that sense, they're all they're all a zero- sum game, right? Of what do you put in? Uh and you can easily put, you know, too many things into the context window uh that you know, if you have so much context, you know, so much like set the groundwork that the actual instruction gets lost uh or too few things so things don't get known. So all of that is is the sort of the constant exercise. Uh when you're talking about external context that you provide, I'd

2 Highlight · Three primary buckets of external context

9:10 say there are sort of three primary buckets. There are rules which you kind of shove it into the context window whether you like it or not, right? By hell or high water. Uh you would put it into that cloud MD or into a must rule uh uh a mustuse rule in cursor although cursor sometimes ignores them. Uh and those are are mandated. You know they are very important things but they they take up context window space. You can't put full-on documents. That's why you're expensive.

9:36 You got to be quite careful the amount of context you add in that because that you can kind of bloat the context window if you just try and put as much context into a mandatory uh you know way that an agent has to then read all of that. It has limited context.

9:49 Exactly. So, so it's the most forceful way to guide an agent, but you have to be limited in content. Often times they are an initial instruction and some link, some pointer to another location. Uh then you've got skills. Skills include like cursor rules uh basically a tiny rule like a little bit of data that goes into the context window for the agent to choose to invoke a skill implicitly. You can choose to call a skill actively like a command uh in which case you don't need those things there. But you can also uh uh uh you expect the agent to invoke the skill at the relevant time. So you have to put a little bit of that breadcrumb in the window. There's some development right now, but maybe separating that out with in the agents. Does it go and consult some directory for now? That's the reality. And then you've got docs. Docs are uh just available information for the agent to find uh but they're not naturally findable for the agent. And so you need to either leave a breadcrumb somewhere like with a rule uh or you just need to name them in a way that allows for uh you know gps and sort of other agentic search to find them. uh

10:51 but they purely loaded on demand and they don't have any price to pay. So you can have as many docs as you want. If you have a thousand skills loaded, you might really your agent today. Uh but if you have a thousand docs, they're just available. It's more about

11:04 which docs that is load at the right time and can it actually find them. So these are the three types we have today. I'm sure we will have more types over time. Again, all coming down to kind of reusable context so that the agent doesn't need to infer everything from intelligence every time.

11:18 Yeah. And I think the interesting thing there is something that we call activation. I'm sure many others kind of refer to as well which is and this kind of like leans on to back in the MCP I say back in the MCP days like it was 30 years ago but when we were creating MCP servers whereby you need to be very clear in the name of the tools the the descriptions there so that the agent would use them at the right times. It's very very similar when we when we when we talk about those various skills. You need to be very clear in the tools in the skill name and in the in the in the skill description so that it doesn't just try to do things by itself just because it can. It needs to use those skills at the appropriate times and just try and do things its own way. So so I think those are very very important

11:57 and it's a good time to remember as well that while it's a standard format, they are not standard models. Uh and so the same instruction in the skill text right now would be loaded by different agents by different models. Uh although we know in like for a fact we have like repeating data to show that the same words would not trigger you know haiku and set and opus to the same action. You know opus is much more of a kind of smartass right and sort of can can choose to say no I know better and won't do it. you know, oh might need more sort of detailed instructions and so at the moment skills don't really

12:29 solve for that. They are one standard unit of context but it's not sure that the same words will affect uh will be the optimal ones for different agents. So so I'd sort of say you know think of skills as an amazing new capability. It is absolutely worth leaning into. We're doing that in Tessle. We think they're amazing. They are maybe the most sort of standard way right now to reuse context. But like MCP, I think they're also a piece of the puzzle and there will be all sorts of of sort of tools or just sort of helpers that we would want to reuse for making agents successful.

13:02 That's really interesting. So, so essentially the standard allows it provides a kind of standard bolt fitting in the sense that you can integrate it with the agents, but your mileage may vary depending on which agent you use because it will use it'll it'll assess the wording and it'll it'll use them uh differently depending on how of course they are implemented in the background. So that's that's a that's a good insight as well.

13:23 Okay. So skills uh we can use them as indie developers, as hobbyists, as open source developers. Uh we can also use it as in an organization uh to describe how we want to do certain things in in our certain methodologies in our certain processes that are key to our organizational uh requirements. Now when we think about adding skills as a first class citizen into our into our organizational process, our development process, um what do we need to consider

Professionalizing skills with testing and distribution

13:52 when building owning those skills and actually distributing it and expecting other you know professional developers in our organization to to to make the best use of that? What do we need to consider?

14:02 Yeah. So I think it's a good question and I think kind of allure of uh of skills is that they they have this immediate impact on it. you create some static markdown file, you do it with a create skill, very very low lift. Uh, and it helps like right now you immediately can anecdotally see that it works. But I think just like with software, there are differences between kind of a one-time it worked and therefore it's awesome uh and something that is a sort of longived uh assets that you now need to live with. These are competencies that you want to reuse across the team. So I think to to kind of take skills to a professional level, to a team level, to an organization level, uh you're you're better served by thinking of skills not as a markdown file, but as a unit of software. This is a a competency, a reusable competency that you want the agent to have. Um so this this sort of puts a a different lens on on what are the tools that you need to be able to own skills, right?

14:56 And to to be able to operate them and collaborate them on them over time. Um there are probably many things to handle but I'd say the three to focus on is one you need to be able to test skills just like software you know you if you want it to remain working or even just sort of assess whether it works today you have to think about what's correct and then test or in the world of AI evaluate uh against those two is uh really thinking about how do you distribute that software we can talk about this more but at the moment there's a

15:26 uh kind of easy but you know sad reality that people are copying skills all over. They're designed to be reusable and yet we kind of copy and duplicate them, copy everywhere. We've seen that movie and we know where that ends. So that's not awesome. Uh and then the third is you have to think about how do you own them long term. You know they will fall out of date just like any docs or anything like that. They uh the models will change and so you know a new model will come along it will think it's very smart, right? Or you would want to use it with some sort of a cheap model or open model. Um and so uh we have to we have to think about how do we how do we maintain those skills? How do we keep them up to date? How do we uh allow a team to collaborate when the person that wrote them leaves the organization? And so the kind of the whole life cycle management of it. I think those are the core ones to think of. So these are three key pieces to to to to skills eval uh uh package management capabilities and life cycles. Now it's really exciting that Tesla uh last week uh released support for skills and in fact would you believe it guy there there is coverage and functionality for

16:27 evaluating skills uh distributing skills through through a package manager style mechanism and also support for for skills in a in a more life cycle a complete life cycle way. Can you believe that? What what are the chances of all three of those?

16:40 Thoughts become things.

16:40 Thoughts become things. So it's we're really excited about that and you can go to Tesla.io/registry /registry to learn more and see some of those skills in that registry.

16:48 It's a it's kind of a good point also to just to give the shout outs to the amazing Tesla team because I think this has been built superbly and quickly on it and couldn't be more proud of the team for pulling together. Excited by the capability

17:00 uh possibly more excited even by just sort of the the sheer sort of uh power that we've sort of seen in the team uh as we put it together.

17:07 Yeah. Amazing. Let's go into these three. Um let's you know learn break them down a little bit. Let's learn about why we need them, what they are, and then let's talk a little bit about

Evaluating skills with review and task evals

17:17 what Tesla offers as part of this uh release. Start with evals. There are a couple of different types of evals that we can that we can offer here. There's review evals and uh task evals. Uh break them down a little bit for us. Explain what they are and then and then how does Tesla uh fit in this uh in this wonderful world of eval.

17:34 Perfect. So eval are uh generally more

3 Highlight · Review evals and best practices for skills

17:37 most broadly a way to answer the question of how good is my skill, right? Is it because like you wrote some words I mean I think you'd easily understand that if you whatever like put in unrelated text or you rambled on for sort of an hour onto the LM it won't be good. So it's easy to imagine something terrible. Uh and so there's varieties of just be able to have a systematic way of being able to say you know how good is my skill

17:59 and that's really important because like when we talk about the anecdotal approach which you kind of like mentioned earlier in the episode uh that that's what we rely on today which is just isn't good enough. Imagine if we were writing code and anecdotally this feels like it works. Let's ship it. It's just not good enough and we wouldn't do it for any other piece of software that we consider a key part of our workflow. So why would we do that with skills?

18:19 And we all know people that you know take forever to get to the point of it maybe like I am right now. So it's like you know when I was a child you know I wanted this like all of that type of context would not be very useful you know at the end of it is like and therefore when you get commit use you know uppercase letters you know like that's not very uh very useful. Um so so those are as a whole and we might add additional ones. Uh we do have review evals and task evals as you mentioned. Uh let me actually kind of read out a little bit. Uh review evals focus on just making sure that the skill adheres to the best practices that anthropic has released.

18:51 Agent skills are cross agent but anthropic is clearly leading the charge and they created a bunch of best practices uh on their site that are uh a kind of good guidance on how to build it. They're saying things like uh be concise, right? They're saying concise is key. uh set appropriate degrees of freedom. Um you know make sure you kind of uh uh have the right structure for the skill. Uh and so we the the review evaluations on it just start by by assessing it against those metrics. Uh that already has kind of given us some pretty interesting insights. Uh first of all we've seen that if you use the anthropic kind of create skill skill uh it indeed uh it scores quite highly. You know not that surprising you know this is uh and to be frank we're using claud to assess those on it. So, you know, Claude when it creates uses those instructions probably has a bunch of that input. And so, we're seeing, you know, great job anthropic. You know, if you're using a skill to create skills, you're probably going to score well on all of these paths. Um, the other thing that we've seen uh is maybe a bit of a a

19:49 couple of points against the anthropic, which is they might not be using their own create skill to build that. Uh, and we've actually now run review uh evals on uh several hundreds, maybe even past a thousand of different skills. They include a bunch of the anthropic ones and we've seen some in which they don't score terribly well on it. Uh I want to uh pick a little bit on the anthropics canvas design uh uh skill which uh I guess as the name implies kind of helps you design uh the canvas content. Uh and on the content side we separate between our scoring for the description as a whole and for content. Uh it got fairly low scores 27%. Uh and I just love the pros that Claude gives you know its own creators over here for this tile. It says conciseness one in four extremely verbose with extensive repetition of concepts like craftsmanship and masterpiece throughout. Contains philosophical padding and redundant explanations that Claude doesn't need.

20:45 Very straightforward there like explaining what the design philosophy is multiple times repeating the same principles in different sections. Uh so that's a good example of it. And later on it goes to the progressive disclosure one in four and it goes on to say monolithic wall of text with no references to external files for detailed guidance and so on and so forth.

21:04 I'm glad it's I'm glad it's Claude slamming anthropone in the family and I'm sure this is a mission but I think what this goes to show is is not at all a lack of skill kind of in in anthropic and and you know from the creators of canvas design. It's just the need for for tooling, right? It needs to be automated. uh and you know it's possible that this was initially created with you know say you know the create skill maybe it predates create skill uh and then you know over time didn't get maintained. So anyways the review evaluations are about that they're about keeping you uh well structured and say how are you applying the best practices.

21:36 Mhm.

21:38 So that's review evals. Uh the task evals uh go a step further and they actually create scenarios that you would run the agent against and see how well it did. Uh this is actually another one of the best practices that Anthropic recommends. It says create uh the evaluation scenarios ahead of actually creating the skill like make sure you know what good looks like uh and capture those. But few people actually do it. It requires more effort on it. And so we uh create we we have done this for tiles already. We talked about that in some previous episodes uh for our documentation tiles. We create a bunch of context but we create some coding scenarios and we run them with and without that documentation to make sure the documentation is good. uh we do the same for skills. I will say task eval for skills are more of a work in progress. There's uh interesting questions about how do you best extract the evaluation scenarios out of the skill without leaking too much of the guidance itself into the task. Yes.

22:34 Uh and so there's some work there. So I'll share a couple of anecdotes of examples here but uh

22:39 take them with a grain of salt. Uh we've seen maybe I've got like two interesting examples of those. One is we created task evas 10 different coding scenarios for for instance two fairly similar skills. One is called agent browser from versel and we've seen that the skill was dramatically impactful. It took the success rate of the agent as it tries to implement that uh from about 28% to 71% in success. So very very significant boost. The skill is very valuable for agent browser. Uh and we saw another skill called browser use uh which conceptually is similar about the use of browsers within agents. Uh and with that one we saw that the baseline average was 85% and in fact using the skill was minimally negative if they dropped that by about three percentage points. And again a lot of grains of salt maybe we don't have the right tasks for it but it's a good demonstration of maybe this skill is not very helpful. And we've seen a few of those examples. We've seen cases where uh for instance some of the anthropic skills were not bad but we're just not very helpful because we're just

23:40 describing things the model already knows how to do which once again comes back to these right now are evaluations with set maybe haiku needs it maybe opus doesn't need it and and you want to build those out um I'll have one more uh interesting anecdote which is uh someone codified karpathy's uh summary of like the best practices that he recommends for skills on it someone put it together into a into a skill and published it and we ran through it and we actually found that it had again a minimally negative uh effect on uh on agents. It it it we created 10 scenarios. So it did very well in some scenarios and worse in others. You know one example where it did the worst for instance is that it just over complicated or overengineered a simple CSV related task. Um so all of those are I think great indications to to to say like you need to know these answers like you need to know whether your skill works or doesn't and you need to define what are the scenarios that simulate it and then you need to be able to test that over time as your skill evolves and as uh the models evolve or

24:41 you're using other agents etc. There are so many juicy takeaways in this. Like if you are a consumer of a skill, it's really important to know this data so that you can understand the impact that the skills that you're pulling in is going to make to your agent uh experience.

24:55 The second piece is as a producer of that skill, as as the owner or creator of that skill, it's it's no it's no use just creating a skill and assuming again through through anecdotal evidence saying, "Yeah, this feels like it's working better." You need to particularly with the scenarios, it's important to say, "Hey, you know what? This is making my experience. this is this is going to make my chances of doing this thing the way I want it to to be done three times four times more likely to happen. But I think the key thing here is when you mention the scenarios. So if you have an eval where it runs across 10 scenarios, you can see what it's good at, what it's not good at, and that provides you with the insight to say these pieces these these scenarios working really well. I'm going to leave that description. These are the pieces I need to change. So from a producer point of view, from the creator and the owner of that skill, they can actually create the best skill. from the consumer of the skill they can then say right all these skills are available I'm going to pick these ones because I feel like they're going to be you know they are showing the data is showing me that these going to actually make it more impactful of course then everyone's environment is different it's there is going to be somewhat an anecdotal yes

25:55 this is actually working well or it's not working well but I think this gives us like the start we need

26:00 the second piece which really that you mentioned that I think is is so important

26:05 is as models change and and that variation across models or agents There is going to be variety across you know the the environments as I say but even if something works for a model in my agent today as that model grows as the training data changes my context needs to change as well my my skills need to be updated potentially

26:26 if it relies upon things that are in old training data or new training data they will act differently and we need to use maybe even different uh levels of the same skill based on the agent I'm using or the the model I'm using. So these are these are kind of important considerations that we as users and creators of skills need to constantly be thinking about.

26:45 Absolutely. And I will sort of say that uh there's a similar analogy to um uh to tests in uh kind of real world scenarios right which is you know works on my machine you know versus uh versus something else is very much uh a real risk with skills as well. uh as well as you know developers don't like writing tests you know you uh and some some love writing tests but you know many do not uh and so people avoid this sort of hard question of what is correct and you know at at Tesla we will help you generate things we'll we'll help you build those out but over time the the definition of what is correct and what this skill should do is actually more important than the words that you use because the words themselves they will change from environment to environment from model to model uh uh maybe over time, you know, something that wasn't in the training data now is. And so you can shrink it.

27:35 You can actually say less now. Maybe at some point you don't need the skill at all. But all of those uh uh will be dynamic. What actually would be less dynamic is what do you want uh the result to to be that one you need to evolve it when your actual requirements change, your systems change and you need to call an API differently or you know your sort of business goals have changed. Um, but I think not only are these evaluations critical for you to be able to make the most out of agents today and and be able to not regress over time,

28:07 the definition of of what good looks like is actually more important than the context itself.

28:12 Yeah. And I think the other thing that intrigued me with with what you were saying is how essentially we're trying to create a set of questions from an answer that we already have. We have a skill that defines the way we want to do things. What we're trying to extract from that is, oh, okay, you're trying to do this. You've got this end goal.

28:32 What I want to do is create a set of scenarios whereby uh we're not trying to feed or guide an agent into hitting that end goal as we would want it, but more what is the general way that people would try and get to such a goal, such an end point, and then will it do it in the way that the skill is trying to? So it's very hard to try and pull the question out of the answer almost and eval a good eval just like a good test

28:56 they're hard to do well and I think it's important that you know we rely upon that good framework and that's what of course the team is the AI engineering team at Tesla and others are working on

29:06 and it's a I think it's an important note uh that you know right now again this is all journey and at the moment we are talking about creating evaluation scenarios for public skills which are not ours and with limited information because we just have the skill um when we generate for instance documentation we have so much more info we have the code itself we have potentially other callers so there's a lot more information to extract correctness um but the true uh uh kind of correct evaluation scenarios come and and that's what we do in Tesla when we sort of engage more with a full-on customer it comes when we actually get to kind of inspect reality up front you can extract a lot of that knowledge out of git history and what have been the actions and over time you can extract a lot of learning from uh agent logs and how we build. Um and so all of those are correct when you're when you are an organization, you're creating skills for yourself. You're trying to you know optimize and tune them on an ongoing basis and so you get that sort of feedback data uh from uh from actual agent users of it. It's a little bit

30:07 harder to do it for kind of third-party skills that we consume and and we do hope just like with documentation, we today are creating evaluations for someone else's skills, but we encourage and would love the owners of those skills to come to us and we will pass those evaluations and the ownership of the eval scenarios to those people so they can, you know, kind of tune them and correct them and then we'll just, you know, run the evals and publish them.

30:31 I love it. And um just clicking around actually on the Tesla.io /registry where you can see the tiles and the skills and there are a bunch of featured tiles but also then you see the top performing skills and tiles and what's great is you can see you know I can click through and I can see the skills and I can see the eval results on various skills uh as well with different scenarios that I can click you know I can I can open up the scenarios I can see what the scenario is doing and I can see the kind of like you know the the the with and without the skill with and without test and you can see what it's good at and what it's not

31:00 and if you dig in a little bit you'll also see some of the pros uh you see some of the sort the uh the text that Claude writes as it uh as it assesses it uh yes it's entertaining and that's reason enough to keep it on the website uh but it actually has some very concrete advice uh to say hey this is the problem you know like that monolithic wall of text comment was you know entertaining but it's also uh very clear guidance to the owner of that skill to say hey you should break this up

31:25 we should totally have like modes of how we want Claude to give us that information so if it's if it's if it's saying okay this is how this is the review eval you can get it to be nice to me. I'm I'm I'm fragile. Be nice to me. You can get it to

31:37 a little bit of a I'm in a tough spot right now.

31:40 Well, I think the the other way is like the the glimmer of hope. So, like there will shortly, you know, we do this uh with customers today. There will be an optimize button next to it. This is like

31:50 hold out hope, you know, we have this. Do you want us to fix this for you? Nice.

31:53 Click this button and we'll create a new version of it. So,

31:55 which is what LLMs and things like that are good at as It doesn't mean and that's it comes back to you know you need to define the evaluation that is correct because if we if we generate you know documentation with an LLM and we generate evaluation with an LLM and the two are misaligned you have to choose who do you believe

32:13 and what we want to believe is the evaluation but we need the owner to bless that uh once they do it's actually you know not that complicated to create a genetic pipelines to optimize the context to achieve the goal

Package management over manual skill copying

32:25 we covered eval quite a lot there I EV valid is actually a really important piece though. So I'm I'm really pleased we kind of went into depth on that. The two other places that the two other pieces that you mentioned, package management uh or package manager capabilities and also the life cycle of skills. Let's talk about those. Starting off with package manager.

32:42 Yeah. So I think um there's more of a technical element to it, but it really comes out of thinking about skills as software, not as documentation. Um so today when you want to install a skill first you your option was to just sort of go off and copy it you know from wherever it is that you want or you created a skill put it somewhere so your team can do it and remember that skills are are designed to be a reusable entity so really the core element to them is spread them to your team and uh so that's inconvenient uh we've seen you know a really handy tool from Versel called skills.sh Sage and so today you see you know within a week it took it up and people do you know npx skills I and you know provide a GitHub repo and what it does is it copies something from a GitHub repo to theirsklls folder uh and now they uh kind of commit it to their repo right and they duplicate it and what happens if the GitHub repo got an update well they're none the wiser they don't get that uh we've even seen worse than that we've seen the fact that you know because uh uh people install skills

33:43 but then maybe other people in the team are using different agents They might even duplicate that skill into both the dotcloud/sklls and the dotcursor that's skills. We're seeing an entertaining uh effect of all these different agents going and reading skills out of dotcloud/sklls. Yeah,

33:57 it's really it's a mess and that is just not kind of a good recipe for a for a

4 Highlight · Package management for agent skills

34:02 happy future. You know, we um

34:04 we will uh we need something better and this is a problem that we've already solved in software. You know, we we know how to handle reuse in software. Uh we call it a package manager, right? Or a dependency system. uh and it requires a few things. It requires uh first of all like some form of manifest file that knows what you've downloaded. Actually even before that you need uh whatever piece of content that you have to have a version to have an identifier to say well this is this is the thing that I've downloaded and maybe sever style indication of of the delta and ordering of those versions. Um once you have a manifest file with those skills that you have, you should be able to run an update, run an install, uh maybe add some other criteria, separate ones from different environments like dev and production. So all of those things are are solved problems. They uh they you know we have a lot of practices that we can bring to four to to make them work well. Uh we also know that uh pretty much every uh programming language has its own dependency system because each of them have some subtle important

35:04 differences in how you want to handle the dependency relationships and installations and environments. Um and so what we need is we need a a package manager for skills and I will sort of say skills and context because as we mentioned skills is is not the only type of context. Uh and that's basically what we're building in Tessle. know we've had the Tesla registry for a long time. It's always been the sort of concept of you know this is this is a context that is version that is consumable in a way that is you know like a professional uh software development team uh uh tool uh and now today we you know we extended that to uh to skills uh which is really really exciting. So it allows you to u uh tessle skill install whatever skill that you want. you know, we're not going to get in your way if you just want to download something off a repo. But, you know, without any further effort, we will remember that you downloaded that, you know, from in in a manifest file and we'll allow you to update it over time.

36:00 Uh, we will install it to whatever agent that you want and by default, we will still commit it to your repo. So, we don't get in the way. Maybe other team members have not installed Tesla yet. But we think the right destination for you is to uh change that mode and stop vendoring those dependencies. Stop like committing them and duplicating them and having just the manifest file. And just like you npm install or whatever your equivalent is uh in your repo, you also Tesla install or Tesla skill install if it's just the skills that you want and

36:31 it will uh uh put them in the right agent for the right uh uh for the right user for what they're using. And so we generally, you know, believe and again it doesn't feel like a complicated or sort of a harsh statement is that, you know, skills uh just sort of need a better distribution system.

36:48 Yeah. And and I love the two differences there. Uh obviously like you know what we talked about with some solutions is is more of a directory. I guess it doesn't necessarily uh host various skills. It more points to other skills. There are benefits and and and drawbacks to that to that approach. Uh I think when you have something more like a traditional package manager where it kind of hosts that content, it can do it can it can look after a bunch of those things in a bunch of things for that content including the versioning and so forth. Um in including the manifest that says these are the things that I want to use. Um, so I think I think that ability to say, okay, if I want to, you know, push something to Tesla, publish a skill to tessle, I can have all of that cool stuff, including things like eval, including things like the versioning and things like that. However, if I just want to use it lightly, I could actually say, I want to point to this GitHub repo. I don't want to necessarily do, you know, publish anything or or own anything within within the test registry. Um, and and so that will still happen. We'll still we'll still add that

37:48 to manifest, but of course, there will be a few little drawbacks. We wouldn't have the eval. We wouldn't have things like the ability to to pick a version because it's just there on GitHub. It's not necessarily a version in a registry. So, we can't we couldn't identify. So, we more points to the to the latest. But these Yeah, I think these are really really important things when we think about skills as a first class citizen in a developer workflow. These are the capabilities that we need to to to do professional development to do you know auditing as well. What what skill did I use to create these types of things?

38:18 Well, you actually have a look at your manifest and you can say, well, this was this was the skills I used. These are the processes.

38:24 And I think I don't want to diss the existing capabilities. I mean, JavaScript thrived before npm. You know, Java thrived before Maven. It's not they're not necessary to be able to provide value, but just at the pace that everything is happening right now,

38:38 uh I I think I think we need to move quickly.

38:40 Yeah.

38:40 With this as well. And over time also uh it's fair to expect uh Tessle to introduce other things that you have in package managers like can we inspect skills to see if they're malicious. Can we uh we already run these evaluations to sort of say you know like give the consumer some indication to say well I'm seeing 17 different skills are trying to do the same. Give me some indicators to tell me you know what which one you know might be better for me what what worse.

39:06 And so all of those traits will will come along learning from the past versus trying to uh ignore it. It's absolutely it's a it's a similar pattern to other things that we've used. So why wouldn't we do things in

39:17 hopefully a healthy ecosystem.

39:18 So a lot of the stuff that we've talked about kind of from the value that a package manager style solution offers actually leans into a little bit of the the requirements for what we need for software in a life cycle. Um tell us about what those you know what you see the needs on a skills style approach with uh life cycle and professional

Full lifecycle management and production observability

39:38 development professional workflows as a constraint.

39:41 Yeah. Uh for sure and I guess again they all sort of stem out of the you know once you think about skills as innovative software kind of the list that you need for them becomes almost obvious. Um uh skills need to evolve. Uh and so first of all again you need to version them or something. You need to commit their source material. Fine we have that. uh but then when you collaborate on them you need to be able to know that a change hasn't regressed. So that's where eval kick in. We talked about them sufficiently you know but you need a system that every time there is a pull request or modification you know it can modify it. Uh skills rarely live in isolation. They are description of a of a practice but often times they describe a system or a library or even the practice itself evolves and so you need uh some ability to keep them fresh. So something where every time you modify not the skill but you modify you know an API next to it or something else uh you need to be able to track and know that you need to update the skill or not and again sort of make sure that it hasn't regressed. Um you know those are are sort of core like as you build it uh you

40:42 need to define it. They need a standard sort of CD system, right? If we talked before was sort of the git style and the CI system, you know, you need some element of deployment uh to say, okay, publish this now to the registry or committed to the relevant uh repos if they're changing so that those things are automated and they're not, you know, they're not uh uh overlooked or create manual toil. And lastly, you have to think about observability. I mean eval and synthetic tests are are all good and well but uh they don't represent production right they don't represent uh what is deployed and so you have to think about how do you observe skills in production which is basically observing agent logs and be able to extract out of those cases where uh where the the skill didn't work and the the user had to either manually invoke it or had to uh uh correct its behavior.

41:33 uh cases where failures repeated and this might be an opportunity to create a new skill. So all of those uh represent the sort of full system right from from editing and collaborating on the uh the code of skills on it to a CI/CD system on it to observability and closing that loop. Uh I think you need all of those for skills and you know our vision at Tessle has always been that software development will go from being codeentric to being specentric right or intent centric uh where what development teams do is they collaborate on on intent on capabilities uh on how to build software and what it is that you build. Uh and I think skills are a great manifestation of that uh and so we need the software development paradigm around this. you know, we're going to expand into debugging and others. So, we're very excited about that. That's, you know, a version of this is what we've been building at TESL all along. And we work with some, you know, forwardthinking customers that are, you know, like building and developing context this way all along, long before

42:34 skills got introduced. Uh, and they've been amazing and we're learning from them. Uh and so you can expect from us, you know, more and more capabilities uh both self-s serve. We definitely want to help open source skill developers be able to maintain and do all of that with minimal toil. Uh and of course for organizations. So if you're you'll see more coming on the self-serve side and we'll sort of announce it as it comes. Uh but if you can't wait on it, send us a note at contacts.io.

43:00 Uh and yeah, we'd love to uh to work with you. and you know truly excited about kind of the point we are in the timeline in in the sort of creation of indeed this sort of intentric software development paradigm.

43:11 Yeah, super exciting and of course if you're interested in seeing some of the evals using some of the skills on our registry wanting to publish skills in our registry head to tesla.io and of course you can see all the data there and of course you can start getting involved grab the CLI and do all the things I just said publish consume and so forth. Yeah.

43:30 And uh I will note, you know, we we we'd love feedback. We appreciate feedback. Uh please send any uh any thoughts that you have. Uh and we did add core capabilities. We didn't proactively run task evaluations on many thousands of skills. We don't feel like we're quite ready to do that yet. But if you see a skill that is missing or one that you want us to evaluate, there's an easy button over there and we will quickly get on it.

43:53 Awesome. This has been a fun conversation on skills and I think it's an important one for people to people to learn right now.

43:58 We might eventually acquire some even. We'll see. We'll see.

44:01 Look for a few in the room.

44:01 Let's not overpromise. Let's Let's overd deliver. Awesome. Thank you very much, guy. Always a pleasure. And thanks everyone for listening. Tune in next time.