Why Most Observability Platforms Won't Survive the AI Agent Era

10,716 words · about 43 min to read

0:00 The interaction patterns in software will totally change with agents. A lot of the agents are chat interfaces. They are kind of giving you an answer. My view on it is that it will be more like an interactive mode between the user and the agent. As an example, today I'm using Google presentation and there is this nano banana feature. The result is pretty disappointing because it creates an image out of your presentation. I think the way you want it as a user is that I say, "Hey, help me make my presentation slide nicer and then I can still go in and change the size or the text." Before hotel, every vendor including myself as instana did their own format. And open telemetry actually was the first approach to standardize this format. If you look at a lot of vendors, they say we support open telemetry. What that means is that you can send them open telemetry data. They take that data and convert it into their internal format. What we have built is the data is always open telemetry because all the models by default

1:01 understand the format and therefore can really work with it similar to what they can do with code.

1:07 How do you think about the career path in sort of software engineering today and what's a good thing for someone to do today if they're sort of early in their career? Yeah. So, um, before we jump into this episode, I wanted to let you know that this podcast is for developers building with AI at the core. So whether that's exploring the latest tools, the workflows, or the best practices, this podcast's for you.

1:36 A really quick ask, 90% of people who are listening to this haven't yet subscribed. So, if this content has helped you build smarter, hit that subscribe button and maybe a like. All right, back to the episode.

1:48 Hello everyone. Welcome back to the AI native dev. Uh, today we're going to go plunging deep into the world of of DevOps and observability and you know what does that look like in the world of AI? Uh, and to dig into that we have Mirro Novakovich uh who was the founder of Instana has built an observability platform sold it to to IBM. That's where we met uh Mirro way back then and and today is the founder and CEO of D-Zero uh building that out. So Mirro, thanks for coming on to the show.

2:18 Yeah, thanks for having me.

OpenTelemetry native observability preserves semantic context

2:19 So uh just to dig in, start by giving us a bit of context of like Dash Zero, the company uh you're uh you've you've built and are running today. Tell us a few words about it and and some of its core.

2:31 Yeah, so um we are an AI native observability platform. Um we actually started when we started uh we we were not promoting AI native at the beginning we were open telemetry native and that turned out to be a good foundation for AI by the way which we can discuss but so the idea was there is a new standard in observability called open telemetry which standardizes the format of the telemetry data locks metric traces and user events and also standardizes the tagging system on that telemetry data which is called the semantic convention.

3:07 So

3:08 host name now is host_ame or host.name and um that that is the way we have started. So we built a full platform for logs, traces, metric and user monitoring taking only open telemetry data into it and keeping it as open telemetry and making the most out of the semantic convention by creating context. So if you look at a trace, you see all the logs, you see the metrics, the underlying infrastructure, everything in context. based on the semantic convention and we try to make a very easy onboarding flow PLG type of uh

3:45 uh sales motion and and having it and easy to use.

3:49 Yeah. So, so part of it is is kind of just a nextgen observability platform that sort of gives you all the tools and I know it kind of combines all the sort of a the broadened scope of what uh what observability captures today with you know traces and logs and and and all that jazz. Uh but I think that core of open open telemetry or hotel uh is really interesting. Uh I I guess maybe you can sort of say a few words about hotel itself and its distinction between what is it when is it a format and when is it the specific fields right because I know when uh when open telemetry came came around there was a lot of uh uh maybe like a a first set of companies that sort of built on top of it and were maybe overly optimistic at not just the format but like the type of information that would just magically I'll just connect to the system and they already have hotel I will have the info I need and they learned not all the information is always there. Um but but some did standardize. So maybe can you say a bit of like what have you seen in terms of open telemetry? Where has it

Most vendors convert OpenTelemetry into proprietary formats

4:49 standardized just formats and what data is is sort of typical to find in any any hotel based uh capture. Yeah, it's um so let's start by why why I think it's also meaningful right before hotel it is every vendor including myself as instana did their own format right so we had an agent and so you specified your own format and you sent your own

5:15 this is the old school agent this is the tracing agent not an AI agent

5:18 exactly that's the old school agent that you install on your uh host basically that captures the data right the CPU utilization the logs and and the traces Exactly. It's not an AI agent. The benefit was that you could make uh everything you needed for your platform, right, in into the format and you could do do some automatic. So, for example, if the agent was running on the host, you could out add the host name because you were running on it and just send it over, right? Um and open telemetry actually was the first approach to standardize this um this format, right? So that you have no proprietary data and the and and I think the main drivers were the cloud providers

6:00 because if you're running something like AWS Lambda or any managed service and you want to provide telemetry data of that service how do you do that right you either provide it in 20 formats like data dog Newellic dash zero

6:14 or you have a standardized format that everybody can understand and that's how it started right so I would also say if

1 Highlight · Why most vendors convert OpenTelemetry data

6:20 you look at a lot of vendors they say we we support open telemetry. What that means is that you can send them open telemetry data

6:29 but I would say still 90% of the vendors they take that data and convert it into their internal format right

6:36 because that's how the platform is built. So you can send it but then in the system essentially the the the the tagging system etc is gone right because it's now data do format or

6:48 uh format of any other vendor right

6:50 and I think uh uh what we have built is something where it the the data is always open telemetry it stays open telemetry right

6:58 the naming is still open telemetry in the tool

7:01 and uh yeah coming back to your question I think there's only one tag that is mandatory And that's service name.

7:10 So the only tag the the only tag in and and so essentially everything that emits telemetry data is a service, right? In in in that in that that sense. And so if you have a service, I don't know, payment service and you send a lock, you add this service.name equals payment service. And if you have a metric of that, you do the same. And now you can correlate everything on a service level by saying give me all the metrics, logs, traces of the service name.

7:37 Yeah, that's that's the only thing that's mandatory. And that's also that's also a bit of a problem I would say for vendors because we also see that a lot of the data we get from customers at the beginning does not have all the information that would be needed. For example, if you now ask give me all the logs of that pod or that host. If that lock does not have the pot name and the host name as a tag, we can't do it right because again going back to proprietary agents there we could add that information to the log or

8:11 ourselves but now we are relying on the customer sending us the right context and if we don't get that context we can't really

8:22 recreate it right. So um um there are companies like Oliver Garden are only focusing on quality of telemetry data right they actually look at data and then they will tell you oh by the way here we see it's running Kubernetes but we are missing the Kubernetes cluster name and the pod name and

8:39 and so we can give you hints of doing doing better right

8:44 by the way I think there is also a big chance for AI

8:48 we could actually look at data and then proactively add configuration data or

8:55 populate kind of those fields. But this is a but it's a bit of an inverted version which is the sort of the core of it is say uh whoever it is that is operating the system uh they they have metadata that they store on top of it instead of having that metadata be extracted or inferred or captured in the observability system or in an external system otherwise monitoring uh those services. you uh you you kind of collaborate or to sort of push the customer, help them more relevant. Maybe AI uh does that more and and populate that information at the source, you know, to sort of decorate the services with the right uh

9:33 Exactly. We we for example also built an open source project, a Kubernetes operator and uh by using that operator, we do all that configuration stuff for you. So if you use that operator, we will essentially make sure that all the telemetry data has the right Kubernetes information, host information set and we al auto inject the agents

9:53 the agents for example to Java or Node.js into your runtimes on the fly, right? So that you get all this

10:01 agent such a confusing word right now like again these are agents these are not the AI agents these are the uh observability agents the sort ofation of the yeah really like I don't know how we got ourselves in this trouble but uh uh today in observability you definitely have ambiguity around the word agent

10:18 or insecurity right yeah

10:20 that's true as well so so I mean at this point listeners might be a little bit like isn't this an AI startup an I

Large language models natively understand OpenTelemetry structures

10:26 podcast you know like what uh what's going on here. So, you know, you you mentioned that hotel proved actually kind of quite useful uh when it came to agents. Like tell us a bit more about that. Like why why was it useful?

10:39 It was useful. I I think it started by when when these LLMs came out and we started experimenting with it that we saw that we get really good results with those a with those uh uh uh platforms like uh uh uh claude for example. We use clot internally giving us really good results by saying hey analyze this trace right and we right

11:02 put in a trace which is literally a text format of tags right yeah which is specified by open telemetry and now the thing is because it is actually open source it's openly documented it's an open standard all these models are trained on the data and they literally understand that hostname is a host name and they can start uh basically understanding the context getting what is uh HTTP status code 4004 you then know that it's actually a problem right and now it can really analyze these things so open telemetry turned out to be really useful because all the models by default understand the format understand open telemetry and therefore can really work with it similar to what they can do with code right it's like it's a text structure text format very well specified it has a syntax it has semantics And so it it can actually really do interesting things and analyze

12:00 uh uh telemetry data. U I mean we were really impressed by the output right and this is a this is still uh format oriented right because there isn't I don't know at least of any um kind of big bodies of uh telemetry data even open telemetry data that are just available for the LLMs to to train on. Generally I think in the world of DevOps like traces you know what everything you collect even in D-Zero it's in your system it's not published anywhere unlike maybe like GitHub for a code

12:31 it's not published there are a few repositories where you can find a large amount of logs and and spans uh as examples right

12:40 yeah that people posted as like what were they custom created or is it like donated logs you know from a

12:46 donated it's kind of donated yes it's kind of donated and by the Hey, a funny funny story is when we started there is a open telemetry sample application also open source and the sample application has very well doumented problems in it right different types of errors

13:03 and when we started with LLM we asked the LMS about problems in that sample application because it was running and it always gave us perfect answers

13:11 and we were super excited at the beginning but it turned out the LLM was also trained with the problems documented

13:18 on the actual list So it could do it because

13:21 cheat the test.

13:22 It could cheat the test because it knew the problems up front, right? So So results were amazing at the beginning, but then we figured out, okay, it's actually not that amazing with other things and then we kind of connected the dots that it was already trained on the documentation of the problems, right? So um

13:37 so I guess drilling into that a little bit. So like the the LLMs were naturally better, you know, that was like a nice uh a nice uh kind of benefit of choosing open technologies uh that LLMs uh came pre-made or pre- ready uh to to process open telemetry data is that they're not very good at analyzing traces. They're not very good at like understanding time series data. they don't uh um you know the volume of data a little bit is there I guess has

14:06 on those fronts like hotel node hotel like that doesn't terribly matter right or like I guess what has been your experience in terms of like the the native support of like just drop this trace in uh uh into whatever claude or uh jupt and and get some result

14:23 I think there are uh two separate problems right one is you have one phrase and there's a problem in it right an erroneous span or something. If you do if you do drop that trace into for example claude, I think it will definitely come up with an analysis to tell you, hey, I see there's a problem with this trace and depending on the metadata it gets, for example, it's a database problem. It get gets a database status code in there from an Oracle database, it will look up that error code and will give you context

14:56 on that error. So they are really good at saying hey this is actually I don't know exhaustion of a connection pool in the Oracle database and you should do this and that based on the documentation. This is essentially what you would do as a human, right? You you would see that code and then search for

Exposing analysis tools via MCP servers for large-scale anomaly detection

2 Highlight · Using MCP servers to give LLMs triage tools

15:11 it and it does the job for you. Where it is not really good if you have thousands or millions of traces to figure out anomalies inside of it a because the they they can't really do that

15:24 large amount of data right the volume.

15:27 So what you have to do there is you have to provide the agent if you build now we are talking about the AI agent.

15:32 Yeah. You have to give the AI agent the right tools to do the analysis, right? So we have a functionality for example called triage. What triage does is it compares you give it a million traces and ask is there any anomaly in it for erroneous traces and it would look at all the um tags and would tell you oh the the ones with the error always have this customer ID

15:58 as as a tag. And then it will return that result. And now we provide that tool to the agent through an MCP server, right? And the agent can now the AI agent can now use that tool, that triage tool, right?

16:13 And it will use it, right? Autonomously it will say, okay, there's a problem. Let's figure out if there are any anomalies. So let's use that triage feature. So yes, you have to build

16:24 your API essentially in a way that it

16:27 works for the agent and that the agent can use it.

16:30 Yeah. to be consumed. So I think maybe let's delineate. So we want to talk a little bit about sort of the the the way in which AI meets uh you know observability uh needs and sort of that type of analysis. And I think there are sort of three uh uh pillars here to to talk about. One is you know what we started talking about here which is more the kind of AI powered like how do you how do you use AI to and agents to to to be smarter right to sort of you know offload more of that work provide good functionality. The second is about agent as a consumer. Uh and you know I think you started now talking about the agent will consume. So I want to disambiguate a little bit of uh which agent is that?

17:07 Is it your agent or is it like client agent? And then maybe we go a little bit more philosophical talk about sort of product and talk about scope and uh

17:14 uh and the likes. Um so maybe maybe let's start we started you know like digging a lot more like the others can do this it can't do that. Uh so why don't we talk like you have a bunch of

AI-powered troubleshooting agents automate root cause analysis

17:25 agent powered observability um I don't know how you call them I'll let you say that in a second under the under the mantle of agent zero uh is the capabilities um so tell us about that like what are what is what are useful things to do today with AI when it comes to to this world

17:41 I mean there's when we started agent zero was just one AI agent but then over time we figured out that there are actually a lot of use cases where where AI a agents make sense and now agent zero is just a platform for agents and we have different ones and I think the most prominent in the whole space there's also a category AI s sur agents

18:03 is essentially troubleshooting right I mean you you get a call

18:07 3:00 a.m. in the morning there's an outage, you have a problem and now you want AI to support you figuring out what the problem is, right? And and it turns out they are actually pretty good at it. So that's that's the S sur agent. We call him that agent the seeker. So it's an agent. Uh we give every every name every agent a name. And so that's the seeker. And the seeker is essentially the troubleshooting agent that helps you with any kind of problem figuring out in the data what the root cause of the problem is. Right? So it would tell you

18:41 and this is I guess the the uh the root of the root cause analysis uh uh uh agent. Is it mostly about sort of discovering the data? Is more about analysis I guess. What would you say are are the sort of the the core competencies of that agent?

18:58 I think it must do both, right? First, it must understand the underlying system and the dependencies because it's the the agent literally has to figure out where to look at, right?

19:07 Yeah.

19:07 Uh and and and then it's about digging into the data and figuring out if if if it's a log or a span or a metric that is the root cause of the problem, if there's any correlation. Right? So it's a step of things right normally you would see the agent figuring okay which services are affected of the problem and uh what is the underlying infrastructure and then it would ask our server give me the spans and logs the erroneous locks of that service and give me the CPU time of that underlying Kubernetes pod and so it gathers all the data analyzes it and and tries to narrow it down right so that's that's essentially how these RCA

19:50 root cost analysis agents work and again it they work pretty well right because most of the errors are text based

19:57 uh and and um it's it's easy for an agent to look at the data and figure out what to do next.

20:04 Yeah, I think the domains that are sort of getting the most success are sort of the very textheavy domains and this one definitely qualifies there. I how do you think about like you built your agent? I'm sort of breaking my own sort of ordering there of the of the pillars of the of it. But, you know, talk about building your own agent. You're describing a sequence of actions. I can imagine, you know, clawed code or like any sort of other agent that I have myself going off and using tools to do these things and get the right steering.

20:31 I guess how do you think about building your own agent uh versus integrating into into whoever it is that the customer has an agent? you know, why why have your own and and do you think that's a long-term or short-term reality?

20:47 It's something I'm not 100% sure of uh how how it will work. So, we do both, right? We integrate into agents like cloud code or cursor or whatever through our MCP server. So there the use case is or the idea is the developers inside of the ID essentially it's cursor or something and now they can ask a question figure out the errors of that service in production and suggest we fix code right so it will connect to our MCP server do the analysis that that we just discussed narrow it down can then use uh uh the the functionality of cursor to match it with the right code and see if it could suggest a a fix in the code,

Moving from static chat answers to interactive observability workflows

21:30 right? I think that's that's a natural kind of uh use case for developer. Why why would the developer lock into a second tool? Why wouldn't you stay inside of the tool? Right? So, that's

21:42 one sort of integration. But then there's the other use case that uh you get a Slack message or a pager duty message uh and it says, "Hey, uh we have a higher error rate on your payment service." and and you want to click on it

21:55 and get a full analysis in context of the data right with the dashboards with the spans with the logs you want to analyze it and there the agent is inside of our tool that's that's the agent zero then and it it will guide you through the UI right and I I'm I'm just off a meeting with my product team because I do think that this is something we we are working on right now that the interaction patterns in in software will totally change with agents, right? I think at the moment we are still at the beginning where a lot of the agents are chat interfaces

22:31 and and they are kind of giving you an answer. My view on it is that it will be more like an interactive mode between the user and the agent. Right.

22:41 Right.

22:41 Um as an example, today I'm using Google presentation and there is this nano banana feature which says, "Hey, do you want me to update your slides?" And I think the result is pretty disappointing because the result is that it creates an image out of your presentation and now the slide is an image.

23:02 It is nice because it looks like but now it's not interactable anymore for me as a

23:06 continue editing it yourself. You know you

23:08 exactly and I think the way you want it as a user is that I say hey help me make my presentation slide nicer and then I can still go in and change the size or the text. Right. And that's the same for us at the moment. the agent will tell you this and that is the root cause.

23:24 Yeah.

23:25 But may what you really want is I think that it will show you it will actually do the filtering and everything inside of the tool will pinpoint to it and now as a user you can say yeah but I removed this error and now do the analysis again right because I know that this problem is not really the problem. So it's it's more an interactive mode right I think that's something I really

23:47 we are creating right now. Also at the moment we are still at this nano banana type of integration right

23:53 it's it's nice but I don't think it's the final way we want to see inter more as an interactive mode right which in coding you already have right it creates code you can now change the code

24:06 although even yeah and even though there's kind of the conversation right now for instance on sort of the IDE versus the sort of the terminal base you know what is what is the preferred path on it but it sounds like you're dividing it to you know the the maybe the sort of the lowest level is the tools it's just the ability to kind of access uh different parts of the system that's probably just your APIs and you have the MCPs on top of that sounds like some sort of analytical tools of of figure out what the next thing you know might be or like you know agentic uh tools so there might be like as as always with software there might be like composable layers of tools and those you can call them now from here it's like it's an interface question and Maybe maybe there's a case of you know a developer at cloud code at their terminal. Maybe there's a case of like a a headless uh you know when an incident got called something automatically ran and I think for both of those it probably doesn't really matter if it's uh like for the first one clearly it's wherever the user was and so if they're in cloud code then

25:07 then you should integrate as a tool. the middle layer probably can run either way, right? You can invoke a headless cloud code to sort of have whatever the uh relevant kind of MCP tools to do the analysis. Uh but uh but really it's your sort of agentic process of it. So it could just as well run in your system. But but the other part is if you're really in a hey I'm troubleshooting this has the review exercise of like how do I collaborate with the AI uh over here and that's the part that's uh I guess we're sort of most shaping uh uh at the moment it it reminds me a bit of um I had a conversation here with uh with Mel the CEO and founder of u of graphite and we talked about reviews and like to some extent I think code review as it is done today my opinion a lot of that is is would be a bit nonsensical in the sense that you you want the same reviews the same questions to happen before but review as an action of like reviewing the results of the AI's kind of coding

3 Highlight · The existential risk of becoming a mere database

26:07 is is probably more fundamental than ever and will kind of continue to grow in importance and so figuring out the right UX what is it that you are reviewing how is it that you're collaborating is um is kind of an interesting question to to sort out does that sound aligned Absolutely. I also think for us it's an existential question, right? Because if it turns out that the user will use observability in other tools,

26:35 I think then we are just a database. Right. Right.

26:39 And that will mean that it's a race to the button in terms of pricing. And so the value then is not generated anymore in our tool. It's generated in Kursa or any other SR agent tool or whatever. So for us it's really existential to see okay how can we provide more value

26:56 inside of our tool than you get in a cursor or in an AI AI SRE agent

27:02 because if we can provide more value I think it is it is an existential threat right because at the end people will pay for the value not for the data

27:11 and if the value is somewhere else generated then we will at the end be a database right and then

27:17 it it it's not good so that's really how I think about it so For me it's really

27:21 about okay if I can create this interaction with the user and can provide real value uh that's that's awesome right and also I have to say you you probably know but if you if you look at large organizations we have hundreds of users registering for observability but only a fraction of them are really using it

27:41 right

27:41 because you need to kind of be an expert to do troubleshooting such so I think we have the chance with aentic AI to enable almost 100% of the users to get value out of it because it can help you following uh the right actions inside of the tool and it can guide you. So I think that's that's really powerful and then you get more value, right? Because if a 100% of the users can get in trouble self-service and they can build those and the the the the most clear demonstrations of AI productivity in in sort of the the entire software development life cycle, including observability are cases where you used to have a dependence on another team and you didn't anymore, right? Or in general where you skip a whole step. Some of it is, you know, a a product or a support or someone else logs a bug and a background agent, you know, figures out and you only meet the solution at the pull request. Maybe at some point even that gets autoresolved. So that's an example. Sometimes it's a uh something that needed to go from the backend team to the front-end team. Yeah. And then

28:41 get deployed, but now one of those teams can take it all the way. So, you know, it and so I think the the opportunity that you're describing here makes a lot of sense, which is the more self-sufficient, the more sort of single

Automating toil and enriching observability with external context

28:54 owner can take it all the way through, uh, the more the more productivity you get and and now the question becomes just like what's the what's the scope of the product and what value do you get from from which tool

29:06 and there's a second we we just talked about the root cause analysis agent but there's a set of other agent which I call is it's mostly removing toil, Right? It's removing tasks that you really don't like. Right? And as a simple example is dashboards, building dashboards for a service and then you have an update in your service and you you have that service on 50 dashboards and now you have to update all these 50 dashboards manually

29:32 to have that new metric on it, right? And um I think that's where AI is also super powerful, right? It it can create and suggest dashboards for a problem or service. it can automatically update those dashboards. Same for alerts, right? You have alert rules, something changes and you need to update all these alert rules, right? And I think that's where we also we created agents that help you create update dashboards, create update alerts.

29:59 Yeah.

29:59 Um and and do those things, right? I think that makes a ton of sense.

30:04 Yeah. and and and and another big point is uh adding more context and that means we are adding other data sources to observability to understand the context better of a problem. as an example, we connect to linear and Jira or we connect to your notion. And by doing that, we can actually see, oh, well, maybe there was I have a database issue with the with the schema. Maybe there was a task that updated the schema

30:31 and I can correlate it to it, right? Or maybe there's a documentation in notion with a schema update which I can use and pinpoint user to it or I can use the code the connection to GitHub to pinpoint to uh a change in the code etc. Right? I think adding more context is also very important uh uh now because the agents can can use that context to create better answers.

30:56 Yeah. And so this and this comes back to the root cause analysis that agent might

Centralized context and architectural decision records for agents

31:00 come along and and figure it out but you have to relate like give it access and relate it uh to it. And I guess organizationally uh one of the one of the challenges that I hear talking to the enterprise side of it you know talking to the companies using uh these different tools is is sort of this infinite mesh of connectivity between all these different tools like the number of tools I need to connect to my linear I need to connect and then there's going to be another subset of tools that might want to connect to -0 as a source of information I want to read something about the logs because I'm I'm writing code and I want to get that information doing it and all the sort of cross-pollination of it. U and so it's interesting I don't know what your view is. You know at Tesl we uh we more and more sort of think about context as a as a core competency that's a bit agent agnostic and so there are statements about your system statements about your knowledge your practices how a system should operate but also like what should your code do and what are your coding best practices and you know uh how to use a certain library and all

32:02 of those things right now are being derived or sort of intelligently AI powered assessed and extracted out of systems by so many different agents as we all figure out, you know, what's going on. Uh and uh and and we we kind of store them inside. I guess how do you think about the challenge over there clearly is is like where do we go, right? Are you going to curate context in 20 different spots or do you want to curate them in some central uh central environment? And so I guess my my question back maybe to the existential thing as well is what what do you think is is the core of observability knowledge? you know, if if the coding side revolves around the code or maybe the product functionality like in observability, you're a database, but what's the uh you know, what's the most important long-term insight or sort of knowledge that you think lives in operations and not earlier in the system? That I don't know if that makes sense.

32:58 No, it makes total sense. I think uh I I'm just not sure if I have a 100% of an answer to it right now but uh I think the question makes total sense and I do think there are multiple things right uh one is as we discussed the tools that you provide and that can be a lot of uh knowledge and and and context already right how do you figure out what are anomalies and traces and and how do you figure out these uh tags or things you know

33:28 how to ask the questions Yeah.

33:30 Uh I think also there will be a lot of knowledge about how do you evaluate models and and so the evaluation that you have and how do you essentially make sure that uh the root cause analysis does what you think it should do right and also giving some guidance right there. There are more and more of these to-do lists that you give so that it that the workflow is not every time uh uh different but you have the knowledge if there's a certain type of problem how to do the analysis right and which step you should follow on on a on a wider scope I think that's something and then I think last but not least it is really the user experience the the the way you integrate the agents into a collaborative mode, right? Basically coming from a single player mode also to a multiplayer mode where at the end of the day if you troubleshoot, you normally have these war rooms, right?

34:28 And and multiple uh people are are working on that problem and agents are now t part of the war room, right? You maybe have one or two, three agents working with you in parallel, investigating things, giving you information. That information is then processed by a user. you give more context to the agent and and that way you can uh uh I I think that will be a big part of it, right? Who figures out

34:53 the best user experience, the best way of doing it.

34:56 And to your last question, I like the idea to have that knowledge some somewhere persisted right about a system about certain things how it works. I don't know if we would be the system. I I also like the idea that it should be a centralized uh I mean you had these enterprise architecture management tools before right where you had the whole system documented or or should document your whole landscape and and give hints.

35:25 I think

35:26 I can see why you would have a lot of that context into a centralized tool where all the agents can access and get some understanding of how how things work and how they are configured rules etc. I think that makes sense, right? The

35:40 the enterprise architecture management tool for agents, right? That Yeah.

35:44 that has has it in a way that agents can work with it and understand it and understand certain things. I think makes sense.

35:51 Yeah. Yeah. I think uh it's it's a good analogy is like so many things when you're uh when you're sufficiently old like us, you know, you sort of see the you know see you, you know, a bunch of things, you know, a whole bunch of like practices that were uh uh were good practices. They were just too hard to maintain at pace and at scale. uh and now with agents maybe we we do those and so those sort of central systems of record uh well well attuned for consumption we'll touch on consumption by whom in a sec uh is a is an opportunity you know to to

36:21 when when I started my career I worked for IBM and we had uh we had a different set of documents we had to maintain in every project and one of them I really loved it was a document called architectural decisions and essentially whenever you did a decision I used this and that framework or whatever you had to document it and explain why, right? Because whenever you had a new team member join the team,

36:46 they ask why why did you use this framework doesn't make sense, right? You can pinpoint them to the document because there was actually a reason for it

36:53 and once you read it, you you understand. I think that's also would be good for agents, right? If they come in and say, "Oh, why did you do this and that?" But maybe there's a reason for it, the agent cannot understand. And so you should have some sort of kind of categorized

37:10 Yeah.

37:11 knowledge.

37:12 Yeah.

37:12 For those agents, right?

37:14 Yeah. And curated and in a place that you can also choose to change it over time on it. And and then on top of that, what we see is, you know, you you write it down, but not all agents listen the same. Uh and so you you need to um I guess as I think about AI native, I think AI native is is primarily about delegating work to AI. uh and or tasks on it. And I think when you think about delegation, the sort of the really the two things when you think about the human analogy that you need to do is okay, did you did you equip you know the the entity that you're delegating it to with enough information that it's sort of plausible that they will get it done right? Like what is your intent? What is the knowledge around the system? So what information is available to them? And then two is how do you verify their work? And so to me that's basically context and spec and and eval those are like the two uh so you have to have some means the evals are clearly not going to be comprehensive just like with humans you know you're not going to know how to verify every single task that your employee did because then like sort of pointless uh to uh to have them but you

38:15 do need to sort of spot check I guess and sort of assess it and so with like here's the information did you did you listen did you is this uh did you did you understand right uh you can force them to But understanding it depends on

Redesigning observability products around AI agents first

38:28 the the model. So so let's let's maybe talk a little bit about the human side and to start that you know we you know if I talk about a native this was like a little bit of my definition. Uh when we spoke about what is an aative product a lot of a lot of your focus was on this notion of building agent first and so how do you how do you think about the future of your product in terms of its primary constituency? You know is it agents is it humans? How do you delineate the two? Yeah, the way the way we do now design of our product is really that we the we we think of everything first of in the perspective of an AI agent, right? And also what an AI agent could probably do with that information and how we can make life easier for the user then, right? So uh that's that's probably the most important things for for us at the moment is really always say okay can an agent do that work how would that look like and how would that information feedback to the user that it's understandable and also I

39:29 would call it traceable right so so I I think for us essentially in observability one of the most important thing we got from our users is that they want to understand why the AI agent has come to a certain conclusion solusion and you really want to follow it, right? And if we say, yeah, the the problem is is here the database,

39:50 then often the question is, oh, why did you come up with that conclusion, right? And then you want to follow the steps essentially how the agent was getting there. So, it's always about, okay, what can the agent do and how do we make that

40:03 understandable to the user and where are the interaction points of the user with that agent, right? So, but it's not anymore that we say we build a tool where the user is the pime primary

40:17 basically uh interaction point with with us. We think that the agent is the primary action point now.

40:23 Yeah.

40:23 It should do most of the work and the user should just interact with the agent at points where the user has the knowledge uh uh to to make make the agent better. Right? So the interface I guess another way to sort of say this is uh you think the primary UX for the user for the user will be to to interact with the agent uh and yes and so the

40:46 there's like a layer uh of of functionality that is aimed quite heavily at the agent uh to to achieve this and then for the user you have to

4 Highlight · Charts are for humans, but agents need summaries

40:56 solve for a different problem uh and and I guess some things go away. I know is there an example that jumps to mind of like uh how this I guess kind of led you to build something a little bit differently or uh think let's let's think of a very simple thing a dashboard right for a service so you have a service a payment service and a dashboard it used to be the normal thing is you put on the red metrics where you show how many calls do you have to that service what was the response time how many errors did you have and I think today the primary information we give you on a service is is a textual description of the status of the service. It would tell you hey your your service is operating fine. It it it is in the in the range of the performance of the last 30 days, but we found two errors which came up recently that we haven't seen before which are suspicious. And then you say, "Okay, let's investigate them." And then from there, so you're not looking at charts or anything anymore. You're looking at

41:56 that. And then the next thing is that you interact with the agent. You say, "Okay, investigate those problems for me." And so now it jumps to the let's say to the trace view and analyzes already set all the context right it it it has uh uh filtered down to exactly those problems it will show you the right information the context and and now the next thing is it describes look at here it's only for that customer

42:23 uh in in in that scenario and and here we go right so the the steps are now very

42:29 simple uh you get all the context product design. It's not any more about charts numbers.

42:35 Yeah.

42:36 And those things because the agent will do the work for you to look at those charts essentially.

42:40 Yeah.

42:41 And and give you a summary of the things that are important for you to look at.

42:44 Yeah. And and we and I uh I really like that and I think it it comes back to the first principles of like why did we have the chart in the first place? Well, the the the the task wasn't give me a chart. The task really was I want to understand what the status of my service is. uh and charts are are just the way we get used to uh looking at them.

43:05 And by the way, charts are good for users, not good for agents. Right. That's an interesting thing also. Right. We we created charts because we as humans, we are really good at looking at a chart and seeing a spike.

43:17 Right.

43:17 Right.

43:18 Where where the agent will actually look at the underlying data and do a deep analysis of the data. But we cannot look at

43:25 5,000 data points and then see this. We don't see the spike in 5,000 data points, right? That's not how our brain works. We we look at charts. So, but that's a good point, right? An agent doesn't need charts anymore. The charts are just for the user.

43:38 Yeah.

43:38 And essentially, we created the chart for the user to pinpoint an anomaly.

43:44 So now when the agent already tells you, I found an anomaly. Do you want me to investigate it? The the charts gets useless, right? Because you don't you don't need it anymore. But this is exactly how I think we have to rethink

43:56 as you said first principles. Why why have we started with it and and we normally build a user experience to the weaknesses of the human brain, right?

44:04 Yeah.

44:04 And uh uh and now we can optimize it because the agent can do the heavy work, right? The heavy lifting

44:12 and and and uh we just follow follow it, right?

44:15 I I I love that. But also I love how it it actually sort of touches uh both sides of the weaknesses and strengths, right? are saying uh uh uh one part is there's an opportunity to give a user a

44:26 human user a better answer than a chart. You know, here's the actual conclusion of the chart. But also that if you if you work from outside and you're trying to have the agents just rely on the information that is available to to the to the human, it will actually do a worse job because the agents are not as

Democratizing complex troubleshooting for every developer

44:45 good at uh at understanding the the charts as as humans. Um, so I like I I I love that sort of analogy and example and I guess the the next question will be on uh well who who is this for and how does it maybe change the sort of the profile of the individual because you were already alluding to you know today there's a certain level of expertise for people that really truly operate the DevOps dashboards. So some of that is also getting down to like contextualizing I guess the answer to people with different competence just like as a as a human you'd explain you'd use different words to explain the same scenario to people with different level

5 Highlight · Scaling expert troubleshooting knowledge to all developers

45:23 of proficiency. I mean normally if troubleshooting came down to a few people in an organization who have a very at at one they need a very broad understanding of the overall system because a lot of developers and and people have only a deep understanding of their service but not if you have a micros service environment with a thousand services there are only a few people who really understand how everything works together. Yeah.

45:50 And normally in troubleshooting scenarios, you need that understanding to narrow down

45:55 where to look at, right? What could be the problem because I always say or joking if if there is a big problem, systems normally look like a Christmas tree because everything is blinking, right? Because everything is somehow connected. So everything is red and then it doesn't help anymore because if everything is red, you still don't know where it's coming from, right? And then you need this expert knowledge of people say, "Yeah, but it's probably here because I know there's this database that's connected to everything else."

46:24 And

46:24 yeah,

46:24 and I think that's why you only have these power users being really effective with observability because you need that understanding, right? And I think that's where the agent comes in because agents are good at understanding the wider scope and narrowing it down. And now we can enable everyone to basically having that knowledge about the overall system because the agent can give you that context. Right.

46:50 Right.

46:50 And and now everyone can troubleshoot. Uh I'm not saying that we are already there yet. But I think the ultimate goal would be to enable every developer, every sone who who needs to have that knowledge uh uh troubleshooting and understanding the system quickly. Right. Right. without having that as expert knowledge or new people coming in, right? Which is also a problem in enterprises. Now, now these two people who have the knowledge leave the company.

47:18 Yeah. Yeah.

47:19 And as we all know, it's normally not documented in notion confluence or somewhere. It's in the head of the of the people and now that knowledge is gone, right? And and

47:29 and so this is and and this to me comes back to that sort of context. We use the word context, you know, like the human analogy is really uh the separation between intelligence and knowledge, right? Right. And so so you want to enable intelligence. The agents kind of give you some intelligence. You can look at a lot of logs. You can find the spikes. You can go, you can search, you can find related information. But then there's also knowledge of what is the system, what is

47:52 uh uh you know what have been the sort of the past incidents that we've had. So that know that it's always that sort of finicky database you know the sort of in the corner uh you know how do you build those? Um and I think the I think one of the challenges that what people conflate uh intelligence and knowledge and you know like I I sometimes say it's like assume intelligence not mind reading you know like there's no uh you know they can be super intelligent but if you do

Real-world adoption and evolving incident playbooks

48:16 not equip them with knowledge it's anywhere between the setup to failure to just highly inefficient like every time they're going to have to uh to learn that.

48:25 I'm curious about the uh the people side. So maybe like as a bit of a closing as we kind of run out of time here. Uh uh so so this is all good and well and it's the future. Maybe bringing it down to today. What is what is the journey that you're seeing customers or walking customers through in terms of adopting this approach because a bunch of this there's a fair bit of change uh involved in what you've described and people don't always like change and also there's limitations of the technology uh today so it can't quite do uh everything amazingly yet. I would say first of all it's it's it's really changing the way users work. There was a public statement by the CIO the Telegraph in London. He he made it actually on LinkedIn. He said that des Zero and agent zero are changing the way they do incident the playbook for incident resolution are changing because now

49:16 the first place is to ask the AI and not anymore going through the list of steps you do to to troubleshoot and it's I I think so so the agents are already changing the way we do incident resolution etc because they are super helpful right probably also how

49:34 uh a coding agents are changing the way we code we we probably first asked an Asian to do a suggestion and then we iterate with it or

49:41 how we do a post on LinkedIn first ask Gemini or whatever right so uh I think that's already happening uh for sure um what I would say today is that it's really a learning curve on both sides right I mean we have a chat interface and we for example we also look heavily on what the users are asking our system right because it's the first time that we essentially user can do something with system that we haven't designed for them, right? Uh normally you can only use functionality that you have literally designed in the user experience. But now they can ask questions like oh give me a usage report of the dashboards. We we never thought about this and actually it turns out that the agent can answer that question.

50:24 So we now know oh actually users are interested in usage of dashboards,

50:28 right?

50:29 So we get feedback through that interface also to understand what kind of functionality we should bait into the tool, right? because the the the tool today is it's probably how how CHBT when they started they saw people are asking coding questions and then they you you could see oh maybe we build something like the coding agents right it's it's it's a use case and so we see that too but at the moment I think it's a

50:54 it's it's learning on those sides right it's what works what doesn't work how does it change my

51:00 my procedures my playbooks right I we see customers already changing their playbooks on inst incident resolution. But I think over time it it it will become an integral part of observability. I think even today you you we can't imagine anymore having observability without AI.

51:18 Yeah. Yeah. No, I I I love that. And I guess it's a it's an extreme version of what we're doing in general when you're building a product, you should listen to your users. But because the because you're providing users with a with an interface that is much closer to what they would say to a support person or sort of to your to your rep, right? Or to to an engineer on the team uh than the free form uh version of it. It's uh you know, you don't have to like decipher what they meant from the clicks on the sort of the the page or the dashboard they created. Uh you're getting a more verbatim uh version of like this is the question I had. Can you can you answer it, right? Or can you provide that? Especially if you never had this option before, right? They could never click on it. You would never get that answer without talking to the customer. Right.

52:00 Right. Yeah. Absolutely.

52:02 So I think u excited to see this evolve. And I find the world of DevOps uh in general is is like a bit cautious around AI and observability feels like the pioneering uh part of it because it has a lot of data analysis and a lot of like a lot of toil uh to that you know to kind of take away uh so it's exciting to sort of see uh uh agent zero and and in general kind of the kind of AI native approaches and sort of the changes of the practices and I'm very sort of keen to see that and I'm curious about sort of the closing of the loop and we'll see that as uh I guess as the systems become more reliable. Uh so definitely will be kind of you know close and tracking.

Computer science education builds foundational problem-solving skills

52:39 Before I let you go here though, I want to ask one kind of a personal question I'd like to ask many of the guests here. Uh typically I would ask uh well if you had a son that you know was going to university would you recommend that they go take computer science and when we're talking it's like it turns out you have a son that has just started sort of uh uh studying. So I guess uh how do you think about uh kind of the career path in sort of software engineering today? uh you know and do I guess kind of it sounds like you still think it's a good idea to go into a computer science degree today you know how do you think where do you think this is headed and what's a good thing for someone to do today right if they're sort of early in their career

6 Highlight · Why a computer science degree still matters in the AI era

53:18 I mean I definitely recommended it to my son and the reason for it is that I think that the main thing you get trained doing a computer science or engineering uh uh study is to it trains your brain in understanding problems math math etc. Right? I mean I I I was a coding geek since I was a child right? So I learned coding myself not at university for I studied computer science but what you learn there is you learn to solve hard mathematical problems physics

53:49 etc. And I think that will not go away, right? I think having a brain that helps you analyze problems and and structure things, I think that's still really relevant and um I I just think he should do that to get that basic training and then figure out what he wants to do and he's actually he loves hardware and I think that's a very interesting

54:11 spot in the future, right? robots, drones, all these things are

54:15 uh uh also very interesting combination with AI and and software. So

54:20 yeah, I I I think it's it's still relevant. Yeah.

54:24 Yeah, I think it's uh it's a super interesting and I get different roles on it and I I I absolutely agree around the uh uh deal with hard problems, you know, like learn to overcome adversity to begin with. It's just a hard degree to get through, you know, just like life lessons uh of that. Hopefully you have some good teachers. because you learn how to tackle problems. Um I guess maybe what I would add is like I I think the importance as it as it stands if you go to a computer science degree you come out if you didn't do any programming on the side you're probably not a very good programmer you know by that time you know like you're not a but so you learn a lot you you need to learn alongside the degree uh and I think that is probably more important than ever because I think where I put very little faith is in the university's ability to adapt anywhere near fast enough to teach the students the the knowledge part uh of uh of what they need from an AI perspective uh in kind of the AI era. So I guess the sort of the emphasis there is you know do that but then you know alongside the degree learn to use dash zero or like I know like get some experience with some sort of

55:24 agentic uh development tools to uh absolutely yeah I agree

55:30 uh Mira thanks a lot for coming on to the show and sharing these insights and looking forward to see the uh the future of observability uh in this AI era.

55:38 Thank you. It was nice being on your show here. Uh, and thanks everyone for tuning back in and I hope you join us for the next one.