Snyk's Manoj Nair: Skills and MCP Are the New npm

9,186 words · about 37 min to read · YouTube’s automatic captions

0:00 Someone said, "Mendus basically was a margin call on your technical debt." And last week I was with a, you know, like a hundred plus C cells in New York and people are now starting because of this, like, you know, real notion that there is an autonomous attacker out there. Lots of attacks have happened. Speed has increased. Security has not kept up. We've seen 10x increase year over year on on the backlog. And for every issue that is being fixed, eight new issues are being created.

0:30 The AI Native Dev is a podcast for developers and engineering leads at the cutting edge of AI and agentic coding. Join your hosts Guy Perjani and me, Simon Maple, every week as we chat with the most exciting voices in AI and tackle the biggest questions facing developers today. This is the AI Native Dev. Yesterday, you were writing code. Today, you're prompting agents.

0:57 Tomorrow, you'll be running a software factory. Agentic coding is moving fast enough to give anyone whiplash. This June, AI Dev Con London sold out. Hundreds of developers, engineering leads, and VPs packed in to hear speakers from Anthropic, OpenAI, Netlify, and GitHub cut through the hype and get to what actually works. This November, we're bringing it back to New York, bigger, bolder, and full of people actually building the agentic future. Three days of real talks, real tools, real answers, whether you're directing the agents, building the loops themselves, leading the teams, or setting the strategy.

1:34 As a podcast listener, you can get 15% off your ticket using promo code pod15. We have so much planned and we can't wait to have you join us in New York.

1:50 Hello everyone. Welcome back to the AI Native Dev. I'm Guy Perjani and we're

Enterprise AI adoption is accelerating amidst security anxieties

1:54 here in the Sneak AI Security Summit or the sneak backed AI security summit

1:59 with Manoj Nair who is heading sort of product innovation marketing. I forget now your sort of title but like you know kind of key product leader in a sneak. Manoj thanks for joining the podcast.

2:08 Thanks for having me Guy.

2:09 Um really fun event here today. You know various various companies of various stages but I figured what we should talk about today mostly also relevant to the kind of AI native dev audience is to talk about the enterprise and kind of the the state of AI in in in the enterprise and agent adoption because I feel you know often times in this podcast and in general maybe like the the hype of the era you know we we get drawn and we dig into just like this incredible future on it and enterprises have certain burdens a lot of urgency to move but certain sort of burdens that prevent them from doing it and I think sneak is in a really really good um kind of a point of view on it where you can with some very modern some yeah. So I guess just to kick it off like how like what is it that you'd say is the current state of you know enterprise adoptions as you're sort of seeing it and then we'll dig a little bit into like why are like the ones that are

3:10 behind what's holding them back and the ones that are ahead how are they getting there?

3:15 Yeah it's a really good question. First of all thanks for having me here and thanks for being a sponsor of AI security summit. So it's become a really good you know collection of people talking about security and AI. A year ago no one was really cared now that seems to be the biggest topic so it's it's been a really good you know turnout and good to be here.

3:36 I find just as a bit of a side note like when I founded sneak many many years ago I debated whether to come back to security because I was saying on how like you go to a security event and you come back and you kind of want to cry in a corner you know like curl up in a corner and cry as uh just like the world is doomed, you know, there's so much problem. and And you know, you're going to like a devops conference and it's all like singing Kumbaya and the web is amazing and we can kind of build these great things on it. Um, and and so I think like in the world of AI, it's just this this sort of odd combination of both. It's like you have these conversations about these incredible things you do in building and in security like all these things that AI unlocks. And there's definitely a lot of doom, you know, there's definitely a lot of like the world is collapsing on us, agentic attackers, you know, what can we do on it? So, it's just sort of an interesting

4:23 I think I I'll start there. You know, I think we had I hope we had the healthy dose of optimism too from all the speakers.

4:29 A lot of both, yeah.

4:31 But I liked how a enterprise customer recently classified it and I'll give you a little bit going back to the first question. AI is definitely an attacker now. AI is also a new threat surface. And then like we had like Satya Nadella talking about it like recently, AI is an insider and an agent. Uh, [snorts] the fourth thing actually is the interesting part in all this. Like AI is also a defender. So, we can unpack that.

4:58 But like if you take those first three things, I think from an enterprise

A four-category framework maps enterprise AI and agent maturity

5:01 perspective, you think about using AI versus using agents and are you just using or are you building? Right? So, we classify it like think about this like a two by two. Like, you know, use AI, building with AI, use agents, building with agents. And so, categories of companies that I'll say are almost everyone 100% is using AI, ChatGPT, you know, Gemini. Let's assume that's now pretty much the standard.

5:28 So, we've built into like a lot of their existing kind of enterprise tools. It's just sort of it's there.

5:32 And then you got, you know, building with AI. So, you have the co-pilot era and it was you know, it was so interesting like maybe sitting 18 months ago and we thought like it was game over on, you know, AI coding is co-pilot. Um, but there's still a lot a healthy percentage of the enterprise customers because they were early to co-pilot. They were Microsoft and GitHub customers. So that natural gravity was there, but that was easier because of the connection I feel for a lot of enterprises. So 50-60% of the enterprise probably is kind of, you know, in that some kind of AI, you know, assistant mode.

6:09 Right.

6:10 Still auto-completing and sort of sidebar chats.

6:13 Sidebar like, yeah. It's a human very much orchestrating. And so then enters the agentic, you know, building with agents. And that's where I think like it's really the cursor, you know, Claude unlock that happened. That's not still a factory, but it's, you know, getting to a point where okay, the agents are doing work. I can shut down my laptop and and and go away. And there, you know, there's a risk aspect in all this. That's where the attack surface, you know, AI is an attack surface really exploded.

6:45 Over the last 6 months with skills and MTP servers and all that.

6:49 And then the last one is like building agents. So you're like actually and I am modifying that last category a little bit. I think maybe part of our conversation is you are shipping production agents whether for internal use or external use. So you're not just using agents. You're But I also think those people are increasingly the ones that are you could probably say that's also where the software factory is coming to life too the most, you know.

7:13 As they're building that last category.

7:16 You know, I would say like there's a lot of people shipping agents now. We are just releasing the state of agentic AI in the enterprise volume three. I never thought we would do one of these reports three times in a year.

7:27 Yeah.

7:28 It's so rampant. Yeah, you have to be able to be relevant. First time we did it was January. Surprise, surprise OpenAI was number one in terms of models being used for building. By the second report, Anthropic really closed the gap. And this third one we are really focusing on on the agents, not just models being used. Uh like what are the harnesses, what are the agents, and and we're starting to see like this is interesting term of reach of the risk, right? So you could use, you know, we see a lot of different models, open models have become much more prevalent than than even 6 months ago.

8:05 Uh but there's a lot, you know, a lot more agents in there, too.

8:08 in the path. So let me uh echo back a little bit and just like drill into a few things before we go further. So the first thing you're saying is there's actually still a fair bit of that sort of co-pilot style sort of self uh kind of development uh in organizations. We actually see this a lot at Tesla as well. Uh and I think it's it's almost like the the in action versus the action or it's a little bit of the uh feel-good appeal of like yeah, yeah, of course we're sort of building with AI. Um and it's interesting cuz I feel like Cursor, like with Cursor's cloud agents, there was sort of a lot of this conversation like in in some of the kind of more frontier developments sort of teams, not frontier models, uh the you know, the Cursor cloud agents are convenient, but they weren't necessarily like seen as sort of something that's novel. But I think what they cracked pretty well is actually this like fit and convenience into indeed these sort of enterprises. It just sort of smoothed their journey, right? Into that path.

8:57 Um

8:58 Yeah, and that's the twos to the threes that I think about. Like a lot of customers transitioning, you know, from being just using AI as a co-pilot to using, you know, doing more agentic development. That uh transition was easier because of

9:14 with that sort of Cursor cloud agents. So as they as they as they move into that, maybe let's drill into that and then we'll sort of go a bit into these others that are building agents and into the attack surface. I guess what has been what what do the companies need to overcome, right? Or like what is it that helps them uh uh make that leap, whether it's from a security or risk or compliance or maybe even cultural reason to be able to go from I'm just co-piloting to I'm now kind of agentic developing agentically.

9:45 Honestly, the number one propellant of that I feel is leadership and culture more than anything else. We have stories of, you know, like some of the largest financial institutions in the world where the CEO and leadership team are actually using agents on a weekend and they come and talk about it. And so it becomes a the pace of change of these enterprises are so slow typically.

10:09 Yeah.

10:09 But when it is a top-down pushing, "No, you are going to be more productive doing agentic development." That makes timelines that like think about like 50,000 uh cloud code agents getting rolled out in an enterprise. That would be a year-long project. It was uh you know, we were notified in in uh May that it's going to be by August. That sounded like a aggressive timeline.

10:35 And then uh end of May I got a call saying, "No, it's being pulled in to July 10th." So top-down pressure really works.

10:42 Yeah.

10:42 Um and then there's pressure to this is obviously security concerns, right? And so when you have this conflict between then what happens is goes to the CTO versus the CISO. So the CTO is a proponent for the change and we are seeing a lot more of the power of shifting shifting the culture from goes from CEO to CTO.

11:02 Yeah.

11:03 CISOs, unlike in the cloud era where they were slow to get to it, they don't they don't want to be left behind. And we can unpack that too separately. But they don't want to be left

11:13 You think that some of them are even driving it because of the agentic attacker which we'll we'll get back to.

11:17 Exactly. So so then it becomes like what

Agentic coding introduces output, behavioral, and supply chain risks

1 Highlight · Skills and MCP servers are the new NPM

11:20 are the biggest, you know, known security risks? And it really breaks down to in coding at least what what do agents do? They produce There's an output.

11:29 Yeah.

11:30 And in the process of producing it the output could be insecure. You know, the code might be bad, they might use bad dependencies.

11:36 Yeah.

11:36 Uh no different than devs, but just faster.

11:39 Yeah.

11:39 Um then there's, you know, the environment. Again, there's a new supply chain.

11:44 Yeah.

11:44 So, you started sneak and, you know, uh it was like the dependency supply chain risk was a big concern. Yeah.

11:52 The new ones, you know, like skills and MCP servers are the new NPM or Rails agent. So, so those are, you know, there's toxic flows, there's toxic skills. Like, you know, three lines of English can be malware. What is the intent of the skill? Uh and then finally the behavioral risk, right? So, like the Packet OS moment was like a big moment for a lot of CISOs, even though it was a

12:14 Yeah. It was a very famous one sort of the

12:16 Yeah.

12:16 Yeah. And uh this is, you know, Packet OS is uh this company that's building a rental car, you know, management uh portal and uh software. And I think the headline story of this was the agent had was overly permissioned and deleted the production database and uh you know, Rails API back end or something. And the copy.

12:38 Yep.

12:38 And it was uh it was one where uh the the famous line I remember there was, you know, when when introspected, the agent said, "Oops, that was bad. I should not have done it." And so, this "Oops, I it was bad. I shouldn't have done it" became like a big headline. Like, okay, what are all the ways that what is really a safety concern became a CISO concern very quickly, right? Safety, security. But it really comes down to like those three things, like the output, the behavior, the supply chain.

13:03 Yeah.

13:04 And I think the best CISOs and organizations involved are like looking for cutting-edge solutions, whether they build or buy, uh or some combination to

13:12 overcome that and and move the ball forward.

13:15 Right. So, I feel um again, kind of sort of repeating a little bit, right? So, we have these enterprises, they're they're sort of moving to agentic. They have to overcome kind of these concerns. It's like I guess kind of paraphrasing a little bit, like would they generate secure code? Yeah. So, like this notion of security output, there's definitely like a whole period of like, you know, these like these agents are producing insecure code. We had all these stats on it. It wasn't even wrong at the time, except, you know, I think models have gotten better, but it's true that also you generate volumes of it, you have to secure the output. And we'll get to agentic attackers in a second on top of that uh that driver of urgency. Uh then you have the uh sort of supply chain element. It's interesting cuz like skills and MCPs are there, but there's also this whole world of like uh of tricking the agents to actually just download malware. Like they're sort of falling for the same type of mistake.

13:58 So, I don't know if that's sort of the first bucket or the second.

14:00 Yeah. It kind of blurs the line.

14:02 I forget what type of squatting they kind of called that uh

14:05 Slop squatting.

14:06 Slop squatting, yeah. I'm sort of creating a you know, just like making up a dependencies and then going off to download them. Uh and and then you have this notion of of behavior. So, more sort of securing the the environment, which starts veering a little bit into like enterprise security land on it. So, we'll we'll uh come back. So, so you have to overcome these three concerns and and I guess also echoing like a lot of that is the top-down pressure to just happen. One one thing that is um uh that I sort of see when I talk to like, you know, the big banks or or others uh is that, you know, when again, for most people listening, they think about running an agent, they assume a bunch of permissions for that agent. So, like it is running agentically, surely it has access to the internet, surely it has, you know, like some ability to do things. Uh and I see like a lot of organizations uh that are like the banks, they're they're they're embracing. They'll tell you everybody's using flood code or whatever it is, um but they are running those agents uh in a very confined kind of uh uh setup, right? Like they can use intelligence, but they don't really have

15:06 a lot as much information access. Um I guess is that what you're seeing as well? And do you have a view on like is that um like I guess how do you feel uh about the sort of the the sort of security functionality trade-off? It's like are these agents almost as effective even though they are

Confined agent execution balances security with limited functionality

15:24 kind of locked down? Is it losing 80% of the value?

15:27 Uh I think it's an extension of the AI as a co-pilot era. So, these are agents, they're not co-pilot, they're not doing auto complete.

15:35 Yeah.

15:35 They are taking, let's say, you know, task and going and being able to achieve it. So, they can do that. I think it'll prevent them from being real factories, right? And so, now you see all kinds of like the the the enter the most common enterprise solution for like an agentic coding is, you know, they'll use something like skills scanning and MCP scanning from us.

15:57 Mhm.

15:58 They'll have a secure sandbox, you know, Docker or something else. They will look at either creating permissions and separate identities for the agents or the more sophisticated ones are exploring a lot of, you know,

16:11 More dynamic.

16:12 more dynamic authorization, you know, ephemeral authorization on the fly. Uh I think that's an unsolved problem, but a lot of a lot of POCs going on there. So, the minimum just everyone is a teammate, they have their own uh identity and they're locked out of the internet. In fact, like, you know, how do I let my agent securely browse the internet but not the open internet has become another thing. And so, I think that's that next phase that they'll

16:36 Securely to the internet.

16:37 starting to gear to the internet and there's companies that are are agentic um, you know, bro search tailored for agents that are now taking off and and saying, "Hey, we could do that part where you don't need to expose anything that you don't want your agent to have, but it has most of it, you know?" So, uh I think what is missing there a little bit is is um security's always like thinking about all the layers and the risks, but it goes back to what are you trying to enable that organization to do and that agent to do.

17:10 Uh and I think it's just time to unpack it. So, let's let's allow agentic, but let's close as many doors as possible using a little bit of yesterday's model versus tomorrow's model. You know, so so that's where I feel like that's the step between the three and the four that is a pretty big unlock required before they can go off and do that. And that's what they don't get if they like run in such a

17:34 Yeah.

17:34 kind of more open

17:35 curtail model. Yeah, they Yeah, and I think it's interesting because like I feel on one hand it is a real limitation. On the other hand, this is a this is an era in which like the best thing you can do is just just get going. Yeah, just sort of getting. So if the way to get going is to run agents without internet access, fine. Do that. There's probably enough value in that anyway for you to continue. And then alongside that you have to think about how do I start enabling

Adopting agents requires shifting to shared organizational risk ownership

2 Highlight · Agent adoption requires shared organizational risk ownership

17:59 So So I guess

18:00 There's a compliance aspect too here. Like a lot of these organizations feel like there's a compliance reason for a human approver at the end.

18:09 Yeah.

18:09 I talked to some companies who have already gone and changed their job descriptions or or or their contractual agreement saying, "Hey, it doesn't matter if you're using AI or not, you own the output and you're responsible and and and all those, you know, controls in place." And I I think that's fascinating too because like that is a barrier and I've started challenging like what is the actual law that says this? Like what is the compliance? What is the regulation?

18:34 Or is it just the Is this just how we interpret things? Um

18:39 Yeah, I think it's a it's a little bit like it's like insurance. It's like self-driving cars are running along for self-driving uh driving car while self-driving, you know, makes a makes a you know, like causes an accident, who's to blame? Uh and I feel, you know, in in the in the context of a self-driving car, typically the same company owns the entire stack, you know, you are kind of running along and you're building it. But in the case of, you know, it's a it's a production or like a product change that has happened, Uh, it sort of a bit of an evolution, right? If you don't have a name to attribute to it, there's an accountability problem. So, you need the organization, I think, to more broadly uh, decide the changes of a of a of a certain type or to a certain application or such, the company is willing to take kind of responsibility.

19:30 So, you know, they're sort of stomaching the risk uh, in uh, in favor of the uh, the change. Uh, and then you need to and then you need to kind of convince the uh, and then you need to convince the auditors that that's the

19:42 That and that this is so fascinating. We had a customer summit yesterday. We did a poll of who's responsible in the organization. Is it the dev? Is it the business? Is it the organization? And it was fascinating. The AI the more AI native organizations were like further along. They said, "It is a shared organizational risk."

20:01 Yeah.

20:02 We decided we will back this risk

20:05 Right.

20:05 for the benefits. And I think that is probably one of the hardest things for that next leap

20:12 Right.

20:12 um, for people to let go off. And and there's a lot of agents being built and shipped, too. So, there's that risk assessment happening on both fronts in software development and in the software that is powering their business.

20:26 Right.

20:26 It's different sets of risks. Here there there was a human involved.

20:30 Yeah.

20:30 What can I do?

20:31 someone to fire if a thing was problematic. You could uh, there were heads to sort of roll, right?

20:36 Right. In this case, it was already like you're shipping apps, the business was responsible, quote unquote, the business. So, you had product security, you had app sec, you had all these functions. So, I think they this is why I feel like we see a lot more agentic apps in production than true agentic development software development. [snorts] And it's, you know, like you you think about it, like it's like, "Well, how's that possible?" It should be a sequential journey. No, it's like people are leaping forward there cuz the organizational

21:06 notion of risk was already established. And [snorts] we have to move from that individual risk to organizational risk and talk about Yeah.

21:13 I'd also say that the the agentic, like as you talk about building agents to be sort of a an evolution of using agents, I think the sort of the the caveat to that will be that building agents is is a specific team. So, in in those 50,000 developers you have, maybe 100, maybe 1,000, you know, are building agents. Uh but the changing the 49,000 uh is is really uh sort of like a big endeavor. And so, it's actually easier to get a team to be forerunner uh than it is to sort of get an entire organization to sort of level it up. I guess we've sort of seen this with DevSecOps as well. You'd always have the sort of the forerunning teams that sort of set the North Star for the organization and that deal with all of this, you know, compliance and getting the auditors. You can't get everybody and then you have the rest that you need to like gradually move them off the ramp, you know, as they as they build out. Um yeah, and this risk management, I mean, I was just here in my talk at uh at uh at AI SS, you know,

22:14 talking about how there's there's this interesting inversion of control or sort of inversion of responsibilities uh that is happening in which uh uh security needs to build more automation and fix more issues on it. And uh uh engineers, if they want to manage that risk, have to build security controls. And, you know, for the for security, you don't I guess you do blame security engineers and such, but you don't expect sort of security people to inspect everything to say if it's an attack, you know, you expect them to build automated controls that inspect to see whether something was an anomaly or not. So, to an extent, we probably need to get to that spot.

22:51 Yeah, I think that uh you know, it's fascinating you're bringing this uh notion up. I think it's a good segue into a little bit of that autonomous attacker.

23:01 Yeah.

Autonomous attackers force trade-offs on technical debt

23:01 I just for the first time in my software engineering career or even like, you know, in security or outside security, started hearing the notion of trade-off between operational risk and security risk. It's always been operational risk. Uh and last week I was with a, you know, like a hundred plus CSOs in New York and people are now starting because of this, like, you know, real notion that there is an autonomous attacker out there.

23:28 Lots of attacks have happened.

23:29 Mhm.

23:30 They can automate, you know, finding these chained exploits and vulnerabilities using lower you know, things that basically have been our our you know, backlog, right? Like someone said, Mentos basically was a margin call on your technical debt. And so now how do you how do you erase that security technical debt?

23:47 Right.

23:48 And the only way to do it is make that trade-off saying, I'm okay accepting this because I will automate fixing this issue. And uh maybe that is that culture change that changes that last you know, that the the stragglers that we see because you have to do that in mass. You can't do that in pockets. You can't just have security engineers doing that.

24:07 Yeah.

24:07 And I actually think that could be an unlock, right? Like the silver lining here is as organizations as more organizations start asking their their security folks and their businesses like, "Hey, we need to make this trade because it's more important that we fix all that." That starts changing the culture and sometimes a crisis is what you need to change the culture of to to forward. So

24:28 Do they have I I So I love that and I've actually really felt this in the conversations here with sort of different CSOs. This sort of appreciation and you know, to be clear, this is like some of these are banks or like very large like Fortune 100 organizations. So sort of very sort of security-minded organizations but also very sort of ops and uptime-minded organizations. Like they don't really tolerate and and you really see, you know, kind of explicit statements of like I think we are near, you know, some are already at. Few are already at, but many are saying that they are near the point where saying I would I would just accept an outage. No, I want to minimize the risk of an outage, which I got to say from the Tesla lenses that's good, you know, that's a part of what we do. Good code review and so we build those out and and so like we'll we'll kind of help you maintain that quality, but but it's it is interesting to say okay, you just go all the way to patch. And this was very much like my conversation with Caleb uh Seema as well is very much around it, but he's again he's in the

25:29 in the cutting edge, so it's I almost expect to sort of hear that from him. Uh and I feel what's really interesting is to hear CSOs of of very large companies saying this is the conversation. And someone yesterday told me that I feel like, you know, for the first time I feel like I can go into the CEO's office and say we need to shut down all development as we just invest in building uh in the in the kind of covering the sort of technical debt and building this fast. And it would be a a a viable conversation.

26:00 Which is pretty strong sort of statements.

26:01 Yeah, I think I think it is to your point also, this is why I feel like this could be that you know, urgent crisis trigger for a big change that needs to happen. And the second part will I think generally I feel like software engineers we get trained on innovation and building, not on safety and security.

26:22 Right.

26:23 And so as a result, what are we good at? We're good at fixing our application if it breaks because we understand the business logic, we understand the app, we understand the operational part. There's platform engineering teams, you know, who have for you know, like devops to SRE to platform engineering. Uh you have a lot more of this, you know, automation and scaffolding there on the operational side. Um I think like if I were to say two things, one is security teams how to like push that, you know, and especially product security to

26:52 get that level of scaffolding for that security, you know, auto PR and and and merges. And we're trying to help there, but the second part is assume that we will be good at fixing the operational issues faster than security. What else could you do to get ready if you feel like you're going to take more operational risk?

27:12 Right.

27:12 And maybe that, you know, is part of that impetus to go to a true software factory in some sense.

27:18 Hey everyone. Hope you're enjoying the episode so far.

27:21 Our team is working really hard behind the scenes to bring you the best guests, so we can have the most informative conversations about agentic development. Whether that's talking about the latest tools, the most efficient workflows, or defining best practices. But, for whatever reason, many of you have yet to subscribe to the channel. If you're enjoying the podcast and want us to continue to bring you the very best content, please do us a favor and

Shifting security context from developers to coding agents

27:44 hit that subscribe button. It really does make a difference and lets us continue to improve the quality of our guests and build an even better product for you. All right, back to the episode.

27:56 So, let's uh let's talk a little bit about the use of uh security in in agentic uh kind of development or the sort of world of software defense factories and sort of building those. Um maybe I'll kind of tee this up a little bit, right? When we when we talk about uh um security in the in the agent era and the sort of the software factory era, um we we end up talking about this sort of the true intellectual insurance niche. So, this is actually like a good overlap area uh for it, which is we talk about this sort of this division of securing the product and securing the factory.

28:27 And so, in the securing the product side, very much about automation of uh the um uh of of securing the products. It's a lot of what we've just discussed on it. Just bringing in the sort of vulnerability patching and sort of other security controls into it. And the securing the factory is is around, you know, a bunch of these sort of agent protection and these problems we mentioned [clears throat] before. Um so maybe within that context uh it like how do you feel I know it's Sneak sort of moved into the sort of agentic app sec sort of products on it from the scan. You want to say a little bit about how do you think about the securing the product side of it from from Sneak in the sort of the new lens, you know, and how has that maybe changed your views of you know

29:09 [clears throat]

29:09 Yeah.

29:10 how did we see it before?

29:11 Uh I would say Yeah, the traditional app sec with Sneak, you know, uh thanks to you

29:19 It's a little bit funny to sort of ask you about Sneak on sort of doing it, but you're actually a lot more in the detail of it. I'm just like a boring board member at this point.

29:26 If you If you go back to why you did it, it was to change the traditional app sec era. The you know, I remember our first conversation, the auditor, the you know, the gatekeeper trying to do essentially became a C by A. And if you think about what they were trying to optimize for was that scarcity of human bandwidth. I can have only that much dev dev bandwidth to fix issues. So I'm going to like And And then you do it late, so you just like, you know, piling up these issues you can fix your issues.

29:55 We feel I feel over the 11 last 11 years I've been able to like shift that into the, you know, dev loops and it is we're here. We're obviously successful, almost 5,000 customers starting with the biggest in every category.

30:09 Yeah.

30:09 There has been progress, but I don't think we moved the toil. We moved it, you know, to point where if you really did a good job, you can

3 Highlight · Feeding security context directly to coding agents

30:18 enforce the the ID in the PR and uh but it I think that breaks at the pace of a true agentic software factory. Uh and so one of the notions there is could you What are you doing? You're providing security context automation and learning to the dev. What if you provided that to the agent? Can you shift it into the agentic loop itself? And guy, you brought this example of of uh packages, right? Like there's there's bad, you know, unhealthy packages that are more susceptible to risk. There's ones with vulnerabilities and now there's, you know, with zero days there's a lot of malicious packages.

30:54 And so if you could block that right at the point where the agent is making a choice to provide the agent like, you know, this this one is a QR code scanning package that, you know, has not been updated for 10 years and this has been updated yesterday. And as long as the one you updated yesterday doesn't have malware,

31:08 Yeah.

31:09 that's probably a much better choice. So if I can provide that context in the loop right before you get the output, the best part, right? And so what else could we do there is what we started thinking. And like and it's a prevention part, right? And prevention I feel can be a lot more effective than even in shift left with the devs. So shifting into the dev context is one thing that has changed with Oh, if there's agents producing code, actually good news. They're better at following instructions.

31:38 Mostly you have to keep an eye whether they're doing it or they're getting overloaded, too much context, too much cognitive, you know.

31:44 But it's more about effective. There's no reluctance cuz there's no there's no agenda, you know, the agent is just

31:48 Right. It's it's it's it's just technology problem you have to solve on how to do it effectively. Speed is another thing. Like you do have to be very fast. You can't just put a model in there that's going to take long time crunching. So it's a great way of taking some of our deterministic tools and technologies and intel and customer, you know, like global context. So that's something that we're seeing

32:08 really good pro like this is about 1,500 plus enterprise customers who adopted this notion of a sneak studio, which is what where we are feeding our context to the agent.

32:19 Right. Yeah. And I love that sort of phrasing of it of we've been providing context and tooling to developers. Now we provide the context and tooling to

Clearing security backlogs with dedicated remediation agents

32:26 the agent as a as a as a new coder, right? Like it's sort of a a new entity that's now producing the code.

32:31 Exactly. So that that's step one. Now the second step is interesting cuz everyone has a backlog. I have a backlog. I don't know about you. You probably like you know, you raised your backlog. So

32:42 Well, at Tesla we're like fortunately the sort of thing you have a remediation that it's also a factory on it. So the backlog did actually get addressed. But it's the the the goodwill of like or the the good fortune of a of a new company that is agented from the get-go.

32:55 And so if you have any history, you have backlog, and you got there one way or the other. So right now with the autonomous attacker, that became the biggest you know, problem. Like you know, prevention is good, but what do I do with my backlog? And and and there's like really stark numbers there because speed has increased, security has not kept up. Um we've seen a 10x increase year over year on on the backlog.

33:21 And for every issue that is being fixed, eight new issues are being created. So that's not acceptable. And and yes, prevention can help if you really do that at that point. But I I I have a huge growing backlog. So this is probably the most promising use of like

4 Highlight · Using purpose-built harnesses for automated remediation

33:39 an agent to do that remediation. Uh we've been slowly building towards that. We released skills for Cursor Cloud CodeX, and there's organizations like Labelbox and and and Relay Networks, you know, case studies where we got to backlog zero just using the skills. Why? The skills were more security context for your coding agent to use to safely

34:01 Yeah.

34:01 get rid of your security debt. Then we released a lot of customers said, well, we have places where we don't have uh you know, an actual coding agent. Um it's uh you know, I just need to go do a bad job. Can we get a remediation agent? So we did that, and it was a little bit of bring your own model.

34:19 Yeah.

34:19 We learned a lot there. We were um we were a lot more efficient than just using, you know, we have a benchmark called um wall and bump. You know, my team's gotten really creative in these benchmark naming. So vulnbump.com, you can go see the results, but it's it's you know really good to see that okay, you get a 90% merge success rate using OS Plus Cloud versus a 60% just with Cloud.

34:43 Yeah.

34:43 And it's 61% more token efficient.

34:46 So this is effectively the value of a harness, right? Because you're you're sort of building something that is uh attuned for it. It's not just context and guidance, it is an actual kind of a build for purpose.

34:55 It's built for purpose. And where it's taking advantage of the global context is like things like breakability, like we see across you know millions of repos and upgrades of dependencies which particular, you know, upgrade path is going to for this application be the safest one to do. So you got a network effect data starting to come to play. Um we now just today released a cloud remediation agent, so you can kind of do a you know a broader campaign across all of your vulnerabilities and and we're incorporating a model and a harness and all that in there.

35:30 Uh the step that is

35:32 before that, you do want to use frontier intelligence for detection, too. Which, you know, you have a deterministic engine, it's finding things. Frontier intelligence is finding other things that only you can find only with context. So our first application is we kind of use that for triage, so you can reduce the number of real issues versus

35:51 the so just sort of I guess give a view of it. This is kind of an understanding if if previously this was I found a vulnerability, here's information about this vulnerability, and then these skills and all that are really about engaging better with an agent where the agent's kind of role is maybe to understand a little bit the application, analyze it and all that. And here with this information and these a little bit of guidance of sequences like verify the fix before you do that. The remediation agent kicks in. This is This sort of last part is a lot more about kind of the broader application context of it, like understanding

36:21 Yeah.

36:22 Fine, you told me about whatever like non lack of input sanitization in this spot on of it, but what does that mean for my actual application? Does it mean there's an exploitable path?

36:30 Yeah, and they they use the threat model, use the pull, like, you know, what business logic is understood. So, you can be a lot more effective at saying, like, these are the real things. Like, this is not prioritization. I feel like this is like truly deleting stuff that doesn't matter.

36:43 Right.

36:44 And then focus on the things that matter, and then, you know, like, essentially, you're able to do a batch mode remediation of things. So, we're finding great results with that internally ourselves. You know, we put it at like a 100-plus repos that we have. Subset of our full thing, and 93% marked success rate. Learned a lot. Like, what about tests? Repos that have good tests have a good job of, you know, fully merging. So, you then like start going to the point where like, okay, I need to like, you know, almost like do a a, you know, PR agent, I think we've talked about this, or a critic of that PR that was opened by the machine, to say, how good was that? You know, is this going to be successful? And I think the final step there is like automatically. Like, if I can build enough confidence with the human in the loop that this is working well, then there's more likely by policy they're going to say, okay, this class of issues you can automatically

37:38 So, so I think thanks for that. And I feel like this sort of last bit is really where it starts touching the factory on it. And and I feel like the reason AppSec existed is that it or like

Merging security competence into the software factory

37:47 exists today is that it it carries these sort of two competences in the world, right? Which is like in the organization, one is the the sort of the knowledge of what is secure and insecure, you know, what like what is correct behavior from like what an attacker might exploit. And the second is like how does an application behave and what would break it. And I think agents are just raw intelligence, and so to some extent you can feed in information from both of those, and the opportunity of blending the two together is there. But if you want to get autonomous merging of it, that's the part in which you need to get the engineering to manage sort of risk, and maybe that's like a software factory line versus if you want to understand which of these risks even you know merits the tokens you know and which you know really should like maybe a conveying of what's the other side in that risk equation you know to the factory that's a lot of this sort of like agentic app sec and sort of fitting that in.

38:43 Yeah I think I think I think that's a good way to think about like the separation and this goes back to what we were talking about the trade-off of operational risk and security. Reverse that is a perfect place where this comes to life where the factory with the engineers who are like driving the intent and orchestrating it if we have this kind of you know agents for doing the app sec know how to incorporate that into the whole factory process so that it's going to be a lot more effective we're bringing that global context we're bringing the automation we're bringing the security understanding but bringing it in a way that can be leveraged by other higher order agents in in that factory. Like almost like the security fabric for that factory.

39:26 Yeah. So I I feel we're kind of running out of time a bit here. So we talked a lot about um just sort of in general like kind of enterprise adoption you know and sort of being hindered a bit by that sort of risk and sort of controlling it in the future. We talked a bit about some of the current pains around embracing the factory malicious skills malicious NCP servers behavioral kind of containment and then how kind of the agentic app sec kind of evolution is able to secure the product side right going from starting from delivering that type of context information to the agent versus the human to dedicated harnesses around remediation and the first one was more in prevention you know the second is more in remediation to much more of like full loops that try to collaborate with hopefully an agentic functionality on the other side that is able to remediate things. I'm

Unlocking enterprise trust through intent and behavioral controls

40:18 curious uh and maybe as a as a closing question as we as you as you look at these enterprises that are today running agents in like a more contained fashion. They don't kind of provide you know internet access or or others and and as the factory like I I think today the products are such an easy target for agents that they're sort of pursuing mostly those. But as agents target factories more and so they target that agent manipulation more how like what do you think if you kind of I don't know even what timeline to sort of offer really in sort of factories, but I guess what do you envision as the future of sort of enterprise agentic automation like what type of controls do they need to have to be able to trust these agents from a security lens not you know I think we understand the breakage risk, but from a operating those agents securely

5 Highlight · Bridging infrastructure sandboxing and application intent

41:13 what type of control do they need to I don't know if the right word is let go right or like trust enough they would run them with broad responsibility.

41:24 Yeah, I I think the biggest you know like you talked about like the three aspects the output the supply chain and the behavior um there's obviously the I think about it like the infrastructure layer and the application layer typical security we have thinking. I think at the infrastructure layer like you know sandboxing and all those controls will become more mainstream. Um you kind of need to like curtail your blast radius, but it doesn't do anything at the application layer.

41:52 Right. Yeah.

41:53 It's just protecting your local network and machine in that sense.

41:56 So that's great. You know, do it. Um I think the two biggest things I feel that are kind of fledgling technology capabilities, but not yet fully solved in a robust enterprise forward way is the identity stack for sure like you know what do you really do with these agents and the identity? And eventually I think that part is interesting because it connects back into the you know, the the origin source and you know, going back to like who created this and

42:25 Yeah, like how the the intent like your the agent is coming and asking me to do a thing.

42:29 Yeah.

42:29 Like what is it trying to do? Like what is it trying to

42:34 What is the real intent?

42:34 do that it's asking me for permission to the system, you know, for, right?

42:37 And then like solving that on the fly and understanding the intent, you know, and then deciding how much permission am I going to give that at the moment? Um So, I think that will get solved, but that's one that I think is not that needs to get solved for more more trust. And then the second one is the behavioral part like let's even take the internet disconnected thing, right?

42:59 [snorts]

42:59 Part of it is like what do what does this agent pull from the internet and

43:03 the internet is a scary place with a lot of but you know, you talked about slot squatting, right? Like people these agents are hallucinating a package name. There's a little bit of interesting predictability in the hallucinations. So, the packages are malwares being, you know, put in those drop points. Those are the kinds of risks. So, could you on the fly really figure out based on the memory and the history what in this particular action, what could the potential behavior be? And not block it because that's old school security. And it's like what do you do to steer the agent back into the like, you know, okay, like yeah, that's not something you should use. Steer it back on the rails and I think those are the two big things I feel like truly unlocks it for the enterprise. Uh everything else is, you know, like it's like the future is here, but it's not evenly dispersed moment.

43:56 Yeah.

43:56 A lot of every things it's awareness is missing. Yeah, there's really good supply chain scanning capabilities from us, a few others, good sandboxing, good way to test the output. Uh I think that's an awareness gap. This is really a not yet fully solved properly gap.

44:13 To get ahead. Yeah, I like that and like I really love like you call it you talked about infrastructure and application layer and I increasingly think about these as like security and safety.

44:22 Yeah.

44:22 Sort of yeah, imperfect split which is containment security as in like I told you to be in the sandbox, stay, you know, like don't don't kind of go over there. And then safety is interesting cuz it comes back to maybe like the future of app sec which is um like if if you think about a skill and it's behaving in a certain way understanding whether that that behavior is correct or not depends on understanding what is the skill and what does it do. And so a lot of that intent analysis, a lot of maybe like the skill definition, those are around sort of the the bound like they need to combine an understanding of what the the business understanding of what is it that it's trying to do and the sort of security understanding and kind of comes back to well, this is kind of what we do in app sec. So.

45:04 I think that may be the last you just touched on it. Maybe the hardest of everything we've probably not discussed but maybe this is it's the next time. Businesses have never been really good. Like I built a data classification taxonomy product like 20 years ago.

45:20 Yeah.

45:20 And no one knew how to put the rules in for that because well, the security guys or the admins don't know the business and the business doesn't understand

45:28 security. Yeah.

45:29 I think agents can actually help bridge that.

45:31 Yeah.

45:32 But that's a critical bridge. Like how do I unlock something that is in the organizational brain of the business?

45:37 Yeah.

45:38 Because that is required to then solve this last part.

45:41 Yeah, yeah, and defining what you what you want and what good looks like sort of the thing that we as humans avoid like a plague. And you know, I guess from the Tesla lens perspective we often thinks about this is like these are skills. Like you know, that is where you qualify it. That's not to say how do you kind of extract that and that's

45:56 How do you extract that is yeah.

45:58 Cool. Well, Manoj, this is a great conversation. Thanks for taking the time over here.

46:02 We're on uh you know, your podcast. It's good.

46:05 And uh thanks everybody for tuning in and I hope you join us for the next one.

46:09 The AI Native Dev is brought to you by Tesselo, the package manager for skills and contacts. Your hosts are Guy Pijani and me, Simon Maple. Our producer is Tom Dawler. The AI Native Dev is not just a podcast, it's a community and we host monthly meetups at the Tesselo offices in Central London. Visit tesselo.io/community to learn more and I hope to see you there.