Datadog CEO Olivier Pomel on AI Security, Trust, and the Future of Observability

12,124 words · about 48 min to read · YouTube’s automatic captions

0:00 If I look back at the cloud migration, the one thing that the companies that defined the cloud at the time, so mostly Amazon, did extremely well was that they handled the security aspect extremely well. That was a big fear of everyone. Hey, I'm going to share my data or my compute with just about everybody else. Won't they be able to see what I have? And they've been extremely strong about it from day one. I think it would be good for the same to happen with the the everything that relates to to AI models today. So we can take those main fears off the table and let everybody else build with confidence.

0:34 You're listening to the AI native dev brought to you by Tesla. Hello everyone. Welcome back to the AI native dev. Today we have a very special guest that I'm really excited to have on. We have Olivia Pome who is the co-founder and CEO of Data Dog. if you happens to have not heard of it, which would be very odd at the audience listening to this to this podcast. It's a pretty amazing company, a whole growth story. We're not going to we have so many things to talk about. We're not going to cover that too much, but it was founded in 2010. Now today, it's valued as like the market moves, but I think well north of $ 35 billion. It has over $3 billion in revenue or you can make that in 2025. 29,000 customers, all sorts of massive numbers. And I think importantly it is the leader the pretty obvious leader in the world of observability really formed and shaped the category on it with with a long series of products that help explore new domains and new areas in a very

1:36 efficient fashion. So Olivia thanks a lot for coming onto the show. Tell me if I did I misrepresent or miss anything important in that quick intro. It's all good. That's definitely me definitely data dog and thank you for having me. Super happy to be here. Cool. Cool. Olivia, I think there's a million things we'll talk about, but really we're going

Taxonomy of AI opportunities across the observability stack

1:53 to probably sink into AI and observability. I think a few people are more qualified than you to share both a perspective of where we are and where we're headed. Maybe we just start off a little bit with just some taxonomy here, right? And some definition. So when you think about observability, like how do you define this world of observability and what it contains? Yeah. So observability, by the way, it's the worst category name to pronounce and I do that in any school many hundreds of times. So first of all, I'm not in love with the word because it's it seems somewhat reductive and passive whereas a lot of what happens is you know very much not passive but the observability is the modern version of what used to be called monitoring before and I think what it represents is obviously the idea of understanding everything you need to understand to to see how an app is performing what is actually happening inside of it whether it's doing what it's supposed to be doing. It's an evolution of what used to be different categories in the like five 10 years ago. You used to have infrastructure monitoring, you used to have application performance monitoring, you used to have

2:54 log management, used to have network monitoring. These were all like little microcosms of of little categories and I think in part thanks to the work we've done those have converged into a bigger category which is called observability that's very much end to end goes from what runs on the servers when what goes over the network but also what are the end users of these applications doing and what is it doing for the business. Yeah. Yeah. Definitely have been a part of that journey with a variety of solutions during that time. It was interesting to see it coales today. that scope is expected is is table stakes when you think about building an observability solution. Yeah. And I like the point around it not being as passive implies. We're going to dig into that a little bit when we talk about AI and agents. So that's the definition of observability. And I guess at a high level, what do you see as the sort of the key opportunities for AI within this world? We're going to start a bit meta and then we'll drill in. Yes. So it turns out there are big opportunities at every single layer of the stack with AI.

3:54 The first one I would say I would call it maybe the most straightforward and boring one is that there's just more applications being built with AI. And those applications look a little bit different in terms of the stack they use. So more GPUs for example, a lot more data, things like that. And and we see that today already we see companies that are building AI that are consuming a lot more infrastructure, building with a lot more data. So that's the I would call it the level zero of the opportunity and these are applications built that are powered by AI. There is AI in the functionality of the application even more these are applications that are implementing AI largely say if you're going to consume a lot of GPUs today you're probably a model builder or you know company that is powering the agents that others are going to be using. The second level on top of that is now there's a lot more applications that are built on top of models and this is a different way of building the application. The application is not as deterministic. The model builds a lot more and the way you manage you build you manage and you understand what's happening is fairly

4:55 different. So I would say that's another new version of observability. That's the second level of that opportunity. And then the third level is okay great now we have all of this AI technology all these new building blocks you know all these new smarts what can we automate what can we build so that the engineers don't have to go and solve issues themselves but the machine can do a lot of that for them can detect a lot more and can solve a lot more for them and so we are heavily investing in all three basically the like serving the communities that are building the AI today serving the communities that are building on AI and then ourselves implementing with AI to to do a lot of automation. Got it. And this is the actual audience for these things is in reverse order, right? There are probably a thousand companies that could benefit from AI powered observability if you will for every application that is built using AI on top of the models and probably a thousand of those for every model builder of this. Is that how you think about it as well? Yes. Yes. There at the end of the day, we think just everybody will want automation, right?

5:56 Every single company in the world will want automation. That's the dream, right? The dream is you never have to wake up in the middle of the night ever again to fix an issue. The issue has been for you and maybe you check it when you're when you're up the following day. Got it. Then I would say most companies will build on top of AI models and they will consume these models and they will have to understand how they work. And a smaller fraction of companies will build the models themselves and operate vast fleets of GPUs and AI infrastructure.

6:25 And there's a inverted pyramid there in terms of the the number of companies that the these apply to. It just so happens that today the biggest needs are with the companies that are building the models because they're upstream of of everybody else in terms of the consumption of AI. Yeah. And they're using they're also became big operational consumers very quickly and every time also they need the uptime, they need the inference stability and then eventually the they need the training in large quantities as well.

6:54 Yes. Exactly. So I think still focusing a sec on the first on the that third category on improving the products using AI. So improving I guess the large swath of data dogs product of it. You've launched a bunch of things already in this domain. So again just starting from setting the stage here what have been the sort of the biggest needle movers or biggest opportunities you tap into from leveraging AI to better serve the broader set of customers even those building kind of a classic applications if you will. Yeah. So we have a there's a few things we built a few products with names on them that we we've pushed out. One know historically we've had a product for anomaly detection that's called watchdog and the idea there is basically watches absolutely everything that's going on and it tells you when there's been a a spike in errors for a specific service or a step change in the the request rate for another service and things like that which which really help. Obviously you can't be watching millions or billions of time series all the time and that's the point of it. So that's what watchdog does. A lot of it

7:55 is would call it machine learning 1.0 in that it's not using financial models largely. It's using statistical models that run very cheaply and that can run all the data all the time that's been there for quite a few years. Yeah. If you ask by the way the chachet is going to tell you it's good oldfashioned AI which I found entertaining. Thank you. Chip. By the way also says that I'm a snowboarding champion which I don't know. I've never been on a snowboarding. Okay. Interesting. Yeah.

8:20 some hallucinations to play with over there. Yes, the there's another product we've released more recently which is called Beats AI and that's the more agentic version of what we do which is Beats runs investigations. It facilitates response to outages and things like that. And you can interact with it and it can run on its own. And so this one is much newer. Obviously, it's built on the newer junior generation of LLMs and all of the other even newer models that have come out more recently. And this one is we're heavily developing as being the future of automation for our customers with applications initially for triage in on call situations or outage situations or fixing errors that are detecting on the code on the production or helping optimize cost of performance or helping identify root cause of incidents. So these are the kind of use cases that are implemented with bits and there's a couple more products. One is LM observability. This one is built to help operators of applications built on LMS

9:21 understand how they're behaving and we can dig more into that. And the last one is we've announced also a financial model for time series called toto named after the wizard of Oz the dog in the wizard of Oz over there which is an interesting character to to name it after. Exactly. Exactly. And there's a there's a clever and the idea behind toto is build basically the best possible time series prediction model and we can also talk a bit about it. I think it's it's fairly interesting. So our very first model and obviously there's a lot more we're building on top of it. So these I would say four of the main directions product or the product we've crystallized our directions into.

9:59 Yeah. Cool. Okay. So thanks for that set the scene on it. Maybe let's start poking holes a bit at the opportunities and the challenges involved in them. I guess the first thing I want to talk about a little bit is trusting the insights from these tools. You mentioned debates and in general there's a lot of maybe an intrinsic understanding that is easy to gro around the fact that observability deals with vast amounts of data and AI is amazing at data. It can look at all these things and now it can read the lines and kind of make sense of them and give you whatever it is the root cause analysis or even choose when to wake you up at night. But as we've just discussed with some snowboarding examples, it also doesn't always get it right. I guess how do you first maybe what's your sort of sense of the current

Zero tolerance for false positives in operational tooling

10:40 accuracy level of detecting these things and giving you the correct answer and what's the appetite for experimentation in this domain am users are they accepting shall we say of of mistakes do they do they expect perfection yeah I think so I'll start with one of the biggest learnings we've had when we

1 Highlight · High precision bar required for observability AI

10:59 started dialogue was that there's one lie customers will tell you which is if your system has a smarts to detect issues, I prefer you to give me the false positives and then I'll decide whether they're right or not. And that's a lie. The reality of it is that you send two false positives to people in a row and then they turn you off forever. There's very little appetite to chase down the wrong path because of a machine that told you we'll do it because of a human corker, but they won't do it because of a machine. And so the bar for precision needs to be very high on the on anything that has to that relates to ability. And the good news is that I would say three four years ago being completely right about root cause most of the time was science fiction and I think now it's definitely within reach.

11:49 We see it in the way in what we've been developing. We see it in the progression of the evaluations for that and as you've been building on AI2 evaluations the hardest part is there it's a joy and a nightmare to be building with a predictability but you definitely see the progress we see actually very good progress there and we can clearly see on the horizon the moment where the the technology is good enough to be put into the hands of customers on in in a large amount large number of situations. So

12:15 that's where we are but yes the bar needs to be super high. One thing we found interestingly is that the bar is a little bit lower in some related areas such as security. I think the users and customers are more willing to take a bet on automated decisions or automated notifications for security than they are for operations. And I think the reason for that is that the riskreward is a little bit different. I think it's it's okay to crash a workload if you avoid a security incident as it's less okay to crash a workload to avoid crashing the workload. Yeah. Yeah. Yeah. That's a little bit of running. It's really interesting though because security first of all like the security my experience at sneak is actually like very similar in terms of the false positive false negative element right it starts off by saying find all the issues. If you're not sure tell me about it I want to know. The reality is that's actually the primary thing that they would feel if they don't tell you. But it's interesting though your comment about identifying flaws or like maybe

13:17 having more tolerance around sort of alerts because to an extent identifying whether something is or isn't an attempted breach is also something that's less definitive. So maybe just people have lower expectations. They just don't expect anyone including human assessors to be able to spot what is what is correct or is not correct. And so it could be the impact or it could just be sort of the expectation of the art of the possible. Yeah. Yeah. And also just to level set what we discussed. I think when you compare what's happening in operations and security to what is happening in other areas of AI like these are hard problems to solve. When you think about we think about self-driving cars for example which are just about working now. What these do are what pretty much any human over the age of 16 can do without even thinking about it. when we think of the incidents at least the ones we've had and the ones we've seen our customers have like it takes large teams of very qualified people and PhDs and a lot of time to try and understand what's going on and sometimes weeks after the incident you're still thinking about them and still working through them and

14:18 understanding exactly what happened so these are really hard problems to solve yeah no I agree and I guess on top of that there's a notion of understanding the application itself right so I guess so maybe two questions coming out of that one is trust is is imperative right people want they they have low tolerance for mistakes on it but the systems are as you've also mentioned not predictable I guess one question is can they find it but even if they could find it doesn't mean they can find it again if given the exact same set of information so I guess how do you think about solving it is the solution sort of the co-pilot world and leaning into that until it's good enough is it about narrow slices is there a different lens like how do you still make progress while you don't have perfection you of expectation of perfection and the systems don't don't yet allow they're getting close but they're line of sight for perfection we don't think it's present right yes I think there's two issues there one is what's the right form factor for the customers to interact with it the right way and that's where you think about is it

15:18 better to have a co-pilot or is it bigger better to have an agent that does things on its own do you want to interact with it vocally do you want a button do you want like how do you what does that look like and that's a big question there the UI really sets up the expectations from the end user. So there's a big thing there and I think we're still as an industry we're still figuring out what works. What's pretty clear is that the chat interface was a great start. It was it opens everyone's imagination but it's not the be all end all. I think most of the AI based functionality is not going to work from a chat interface the long run. The second question is how do you build or how do you enter the market with a solution that shows confidence and or builds confidence with the end user. And I think for that the you can get higher precision by having lower recall and you want to start in maybe with corner cases maybe with some small subsection of all the things you can all incidents you could possibly solve but focus on the ones that have the uh the really high

16:19 precision first. Now the challenge with the naive approaches is that the LLMs in particular are not good at understanding when they know and what they don't know. They'll always gladly answer. Yeah. And might be wrong a lot of the time. So I think a lot of the technology that we're building there is around understanding when we're right when we're not when we have a super high chance of being right. So we can actually volunteer guess and maybe solve the issue directly for customers as opposed to not saying anything. One advantage we have as a company that's already used for observability is that we can choose pick and choose basically the the use cases and the the issues for which we're going to to automatically take action and so that's what we're focusing focusing on today. Got it. Makes sense. So you're already monitoring the whole thing and so you can make a lot of attempt see where is it what are the cases in which you can reach a high confidence conclusion of it and then reach in which definitely is a kind of an incumbent advantage of already being in place and already seeing the data exactly and that's not a choice that's so obvious

17:20 you don't have that choice in in some other parts of the industry for example for self-driving cars you can't say only going to take to to make left turns I think the that's a luxury we have in in our industry right in that sort of breath so that makes a lot of sense so I Maybe leading from this into the sort of the world of of agents of it. You're in the spot. You look at all the data. You

Bridging observability data to automated agent actions

17:39 see the data for the specific cases where you have conclusions. How do you think about actions? What is the these are the insights maybe that you have higher confidence? Do you see some that you reach and like such high confidence that you can even act on them? How do you think about the agentic responses to this data the non-passive version of observability in its stance today? I think that's definitely the goal and we we see areas where we can do it. I think the the lowest hanging fruits there are things such as errors for example. So you have you ship a new application and you have some 500s on the servers and the that's pretty clearly an error pretty clearly needs to be remediated.

18:19 It's usually not that hard to generate the code that will fix the errors. But then there's quite a bit more more work involved into getting that code across and validated and and and running in production. So that's one area where you can take action I would say pretty automatically and pretty quickly and that's definitely something we're working on right now. There's a few other areas I think it's luckily a few years ago we started building the ability to take action for customers and so we built a workflow automation product. We built functionality so customers could could build applications that automate the the way they run their own operations internally. And that's a big departure from what we had been doing for the first 10 years of the company. The first 10 years we're very careful about only receiving data, never taking action, never reaching back. And I think all of that is is really coming to to fruition now that we're trying to close that loop and really automate response for users. I'm a firm believer that for AI to show its its value, you need to automate action. I think if you

19:20 have to keep pestering the humans to do things, you're not going to be successful. I'm firm believer in what you said, like you have to work towards that sort of autonomous piece, but really the key question is that sort of buildup of the trust. And I think much of the industry has taken the kind of co-pilot approach of it. And I think you you kind of have another unique opportunity of it which is to take the cherry pick approach of I'm just going to look at all that data and and I'll I'll pick what works. I guess what has been what has been kind of the most surprising or the cases that you wouldn't maybe you gave me an example of something pretty obvious like the system is basically just dead. It's just turning all these sort of 500s on it.

19:55 Have there been like positive surprises about things the LLMs start showing you that are insights you didn't think they'd be capable of? In general, I think every everything that's impressive you've probably seen on Twitter already or X I should say. That's the I would say that's the blessing and the curse of LMS is that it's so easy to make a mind-blowing demo and so hard to make that work reliably all the time for everyone. And you've seen examples of you you have a crazy error and immediately understand what's what happens and writes beautiful code to fix it. The problem is it doesn't always work that way. and then understanding how to get that code to actually ship it actually is usually pretty difficult. I think we're still very much in the phase of making sure all the plumbing is there so that this works reliably all the time for everyone. Got it. So I guess let me take this a little bit like in that practicality of taking these lenses and maybe maybe talk about all aspects of it and one that comes to mind is security and so you're dealing with analyzing logs analyzing systems clearly a lot of untrusted data and we've had several

20:57 conversations here probably because of my kind of passion for the topic of it about the security and how those for an opportunity for prompt injection that might even come from those those sort of untrusted fields that might try to elude or fool the the agent bits or some other product to take action when it shouldn't. I guess my kind of I have two questions. One is how limiting is this right now? Is this a theoretical concern you're seeing in practice? And then how do you think about overcoming that? It's another aspect of trust, right? That is not about capability but rather about adversarial activity. Yeah, it's always been a concern, right? So it's already the case that and we've seen that you can try and inject stuff through logs, through traces, through pretty much anything that submits data. It used to be cross-ite scripting. You inject weird things into the logs and we've definitely seen that. So what that means is that everything needs to be heavily sandboxed. Everything needs to be like you need to code defensively against any data that comes in. Obviously with the LLMs like the surface of a of attack is a little bit fuzzier. So the problem is

21:59 I would say harder to fully contain as long as you end up passing some data to the LM in the end. That's definitely something we're aware of and something we're building against the we see that problem too in that I mean what I just mentioned to to solve to resolve errors and fix the code for that you do end up having to generate and execute code anytime you execute code you at great risk right everything there also needs to be heavily sandboxed heavily isolated so that you can do that and not fear the consequences of having all of that trusted code that's going to execute everywhere but in general it's a it's good habit to imagine that anything that processes external data might end up executing interested code. I think the so it's interesting I had Mattel on the show on it who is in Lera which security company and we talked about two things.

22:48 One is how indeed the control plane and the data plane in security or sorry in a world of LLMs the control plane and data plane are merged and so you're basically executing everything like all the data gets run right gets executed by the LLM so it's hard to separate it and two is that from an agentic perspective if you think about maybe one definition of agent security versus LM security is that LM security tries to fool the human right I might try to get the LLM to give an alert that shouldn't right or reach site that shouldn't but then agents actually take action on those and so you're taking it up to the next level, right? You're actually trying to guide the system. And so it's interesting that you say that that you run code to generate it. I guess that's that's a more powerful but more dangerous action to it. But in general, do you think this is an eventual limit to the autonomy of these systems or is it I guess do you anticipate that our sort of trust level in the agents decisions would reach par with our trust level with the sort of the humans that that make the action.

23:48 Forget how you execute it well, but you've chosen to restart a server. You've chosen to redeploy code. Those are decisions. Yeah. I I think what this means is we probably just won't have completely end to end models so we can build the right limits around the models so we can build enough trust there. I think that's might be different from with the desirable end studies maybe in in self-driving. Self-driving you hear that every now is building around end to end models. You get cameras in and accelerator put out and that's it. I think that's going to be a little bit different for the kind of systems we build precisely because we want to have all sorts of checks and balances and validations along the way. So you can't have adversarial systems or injections and things like that yield the the wrong actions there. And on on our end, it's also an opportunity because we also build the tools both for correctness but also safety and performance. And so these questions we have when we build those agentic systems around how are we certain that they're they're not being abused and they're doing the right thing and we can trust them. These are

24:48 questions all of our customers have and they want to answer for their own applications. And so we build a tooling for that as well. Yeah. You're able to monitor your own actions to an extent. You kind of have the right competency in the both the tech and the and the team itself, which is pretty cool. I think that's really interesting. And and I wonder I I spend a fair bit of time thinking about the the scopes of autonomy. And often times like self-driving car is a good example of it. You you don't hold them accountable or you don't know who to hold accountable if they make a mistake. And so agent systems might have the same problem of the limit might not be a technological one but rather where is it that you're uh whatever what is a reasonable liability for data dog as a company to take for taking an action that otherwise would have been a user's decision for it. Does that come up like the sort of risk management almost like compliance risk, legal risk threshold on it versus technology as you think about a digital? It's not uncharted territory.

25:41 You have systems that automate updates for example and as we've seen in in several very public cases like a an update can can have really horrible adversarial impact on the on customers. So that's not completely new. Of course, there's going to be maybe extensions of that. Maybe there's going to be more refinements of the of the legal framework around it, but I trust that that's going to get figured out. But to your point, you every company, not every company, but every other company has had an issue where the intern dropped the production database someday trying to to fix an issue. That might happen with early versions of these agents too. So that does create an interesting question of who ends up being responsible for that in the end. But again I think it's a not very different from all the various ways in which automation is already being used. Yeah. And say how much you have seen firsthand or been part of the migration to the cloud. And I think it's true that in cloud you had to be like a little bit embracing of the failures to be able to be in the lead.

26:41 Right? Those who embraced the cloud at the beginning suffered from some gaps from whatever container sandbox vulnerabilities to reliability issues of some of the clouds at the beginning to things that were like not irrational fears but actual kind of flaws. And you had to maybe embrace the failures, right? And lean in and accept the slightly finicky behavior of the system at the beginning to be cloud native and benefit from those advantages before the rest of it. Do you feel it's the same here? is that all of these concerns that I'm raising and all of these fears are ones where hey if you want to be one of the leading companies of the future then as a user I'm not even talking about data dog as a company although that's also true but as a company building right now and using tools embrace it brace yourself for having some false reports for having some sort of system that deleted because if you don't you're just going to be left behind does that resonate with you yes I think the technology is definitely rough around the edges on the for anybody who's building applications on today. I think that's definitely the case and that's

27:42 also why in a way there's a I would say a relatively small number of AI native companies that are developing fast and shipping a lot but the broader set of companies that are building on AI are still mostly testing the water like they're still having to have applications in validation in early test with their own customers in proof of concept pilots like you name it. It's it's harder to get all of those into production just because the the failure modes are not well understood yet. The the technology is changing fast. There's a bit of discomfort with all of the new exposure you might get with it. So we definitely see that. I would say if I look back at the cloud migration, the one thing that the companies that define the cloud at the time, so mostly Amazon did extremely well was that they they handled the security aspect extremely well. That was a big fear of everyone.

28:33 Hey, I'm going to share my data or my compute with just about everybody else. Won't they be able to see what I have? And they've been extremely strong about it from day one. I think it would be good for the same to happen with the the everything that relates to to AI models today. So, we can take those main fears off the table and let everybody else build with confidence. Yeah, I think that's a really good observation and it's interesting because while I 100% agree with what you said, I also think cloud security misconfigurations are probably today the biggest sort of sort of breaches and and security flaws within systems. Those are not direct flaws of the kind of AWS and cloud providers. However, they are created by the ease of creating infrastructure and and configuring it. So you do have to wonder whether prompt injection will basically replace replace replace that world and be the next vulnerability. No, there's a risk of that happening. Yes, we replace the the open S3 bucket by by hey, there's a prompt in there. There's

29:33 a chat in there. So I know exactly what I get I can get out of it. Exactly. With a statistical attempt that sort of says whatever send me all your data from from any sort of chat interface that you that you come up with. Hey, let's get

Training custom time series models for predictive monitoring

29:43 technical a sec here and talk about how to get observability right. So you've made the decision to build a foundation model in Toto. The more I think about it, the more I love the names for data dog. It's the penny took a while to drop here. And what prompted that decision? Why not

2 Highlight · Why custom foundation models beat general LLMs for time series

30:00 build on top of the existing LLM? And I guess because do you think of as a replacement as an addition? It was got a very different focus. The goal is to work on time series and to be predictive for time series. For that we under broad models don't work very well with time series. they get you some things numerically they're not great in general and for time series in particular they're really not fantastic. We also needed something that could be very small so that it could be run in process on very large numbers of of time series.

30:31 So the first version of toto is a is purely numerical. So we fed it a number of time series. We have large numbers of time series of data log as you can imagine. Yeah. And these would be like metrics from operating all sorts of aspects of systems. Exactly. So some of the data is traditional time series model training data which is not from us. Some of it is like weather data and things like that and and a lot of it or should I say most of it is our own observability related data on which we have some pretty strong signals in terms of which which time series are interesting or not because we see which ones are being used for alerting. We see which ones are being looked at at which frequency things like that which gives us a good sense of of quality of the signal. And what was somewhat shocking to us was that the very first model we built which was built I would say in a little bit of a naive way. It's the first time we built a one of those broad node deep learning models. So the first version of it is already or was at the time of release state-of-the-art on on time series prediction including for nonobservity use cases which shows of

31:33 course that deep learning is working and scaling. I think we've learned that over the past couple of years, but also the value of having great data to train those models and how it really makes the whole difference. So this first version, we haven't released it to the broader world. I think we really wanted to use it as a starting point as a baseline for more research we wanted to do on a

31:55 model. So we're busy building new versions of it today that are multimodal in that I would say it's a lower version of multimodal compared to what you can think of for the financial yes doesn't have vision but it uses in addition to numerical data for time series it also looks at the the text data that surrounds those time series whether that's the descriptions you have for the time series the names of time series the tags but also the event data that we're going to see such as what happens in logs or other time series, not just one time series at a time, but multiple time series that can be correlated. And so we're pumping all of that into the model. And the point there is to make it really good at being predictive so that you we can detect anomalies better. Like the anomaly detection is all about comparing prediction with with reality.

32:44 But also in some situations, we can actually get ahead of incidents. We can say, "Hey, this this monitor you have is going to trigger in one hour." And we know that because your monitor is on the anode here, it's on the CPU of your database and we see that the queue of queries that are lining up on the database is going up. We don't see it. We don't encode that as a rule, but the model has learned that it works that way. And so we're pretty excited about it again because we've seen really impressive results in the very first research version of it and we are heavily at work to to build the next version. Yeah, it sounds super exciting and I relate to the gap that you see in the in the model. By the way, a bit of a trend in kind of the world to find specific niches in which you can build slightly smaller, slightly faster, but also better models for specific slices, but using the transformer technology.

33:34 So, love seeing that and yeah, clearly you have the data. I'm curious like how you think about it from a product perspective. Is this meant to be an engine that only manifests in data dogs products? is do you think of building an open weights model and having the world build on top of it? How do you think about it from a from an offering lens? So everything's on the table right now. So it definitely will be part of data log at some point and that's why we're building it, right? So we can use it. We'd love to to have an an open weight version of it. I think we're being very careful there in terms of what data we have and how we use it and contractually what we can and can't do and what's right and not right to do with the data.

34:11 But we we would love to have an open weight version of it. So we're working on that. Yeah. And do you envision eventually creating for large enough customers models on their specific data? Have you experimented with whether that achieves better results or not? That's definitely also one thing we're experimenting with. Today we see that the broader models work better across customers because customers also have like a diversity of data they're using they but at the same time they're using a lot of shared components. So customers are using MySQL and posgress and it's the same database across customers and their sales metrics have sales in them in the name. So it's also somewhat semantically similar across customers.

34:49 So we're experimenting with that. I think we'll see where we end up. Yeah. Yeah, that makes sense. It's I love the analogy sometimes experienced individuals, right? And so if you think about having a very experienced S sur come into a brand new system, they actually will have a lot of knowledge to come along. Part of it is familiarity with the Postgress that is deployed here that they've seen before. Part of it is just analytical to say I know that when memory consumption and CPU consumption behave like this, it's recipe for trouble. Yeah, I like the idea of bringing that knowledge from customers to customers. And I guess it's interesting whether there's a fine-tuning layer on top of that on on a specific specific customer's data. Yeah.

35:26 One one thing that we've learned over time too is it's critically important to have low friction for products to be adopted and to show value. And so it's very important for whatever you have to work well on day one and to work well when people just get started and they listen your data and they want to see things happen right away. If you can get better over time it's better but you'll never get there if you don't show value very quickly. Okay one yeah love the product sensibility. So I guess one more question on that domain. You know when again when you think about the person and their knowledge of it, infrastructure knowledge is useful but also familiarity with a specific app in question. Have you looked into and do you anticipate including code and application knowledge of it? Things that might not have traditionally been in in data dogs remit. Yeah. So application we get already because we do tracing, right? And we do we we understand what causes what and what's being run and things like that. code. It's there's been a big effort from us over the past few years to to get the code into data dog as well. The main driver for customers to start tracking their code in data dog has been using our profiling

36:28 product. So we have a continuous profiler that runs all the time in production and it really benefits from having the full source code fully readable source code so you can tie what's actually executing to to the code you have and and we and we basically we're pushing towards that. We've also been building extensions to the ids so that customers can navigate directly from wherever they are in the code to what's happening on production. So that's being it's been a big effort from us. And I guess the if you cast your eyes further, I'm getting a little bit ahead of myself because I want to finish with some futures here, but oftentimes think about the AI kind of impact on software development or genai impact on software development as really a continuation of the same trend that we've been seeing with DevOps and continuous deployment and all of those in which it accelerates software development. It allows more independence. More software gets created and updated faster and faster. This is another multiplier. And I've heard you say similar things in past podcast on it. In that case, like how do you think about the loop between observability and

37:31 bringing bringing that knowledge back into code and modifying the code back in? Do you see that accelerating and does that impact the the kind of the riit of a platform like data dog? Should we would we see the expected scope of product or scope of platform change? Yeah. So I will say so just to level set I think there's a there's been a continuous as you said improvement in productivity for development and that dates back 40 years right if you compare where we are today to uh where we were 40 years ago or 50 years ago with punch cards and assembly language and we're already a thousand times more productive if not more in terms of what we can build. We've had the more advanced languages, computer screens, we've we we have open source libraries, we have the cloud, we have all of those things. So you can do you can build incredibly sophisticated things with very little code and very little mental exercise. I think we're looking at another 100 times or thousand times increase in

38:32 productivity with AI. And as we go through this productivity increase, developers have less and less of an understanding of how their application is working and what it's doing. And so we're switching more and more of the work there from, hey, let me type the code to okay, no, I need to understand is it working? Is it working well? Is it costing me more money than it should? Is it doing what it's supposed to be doing for the business? Yeah. Or did it break?

39:00 Did something happen when other things other systems change or the customers change or something else happened? So these are the big questions we answer and these are I would say the questions that are becoming more and more valuable as the the productivity gains stack on top of each other. Yeah. So I agree with that line of thinking and one of the things that I wonder is if you're if you're not writing the code and you have less visibility to it, you need to understand those systems on it. Like today when you observe, you're still observing in large parts of the observability world. You're observing code. Eventually when you find a problem, when you see a flaw, it's either in the infrastructure or in the code. Those are the places you change to evolve. How do you think about if you're less familiar with the code, how do you observe a system? Does that require some fundamental change to the core tenants of how observability is done? I think the observity will be end to end. I think you what you observe in the end is you observe your business. Are my users doing the right thing and are they finding value and are they being served properly? And that's what you care about, right? And you might have more specific questions which are, oh, I ship

40:01 that feature. Is it being used and making a difference? And even more specific questions such as, oh, I changed this line of code. Is does it have the right impact? because sometimes you will still have them to do and make these specific changes. I think the future of observability is that covering this end to end and helping answer those questions and plugging back into whatever the the developers are using to conceive and build those applications to start with. Today it's largely a number of different code editors and IDEs and they write the code and they ship it.

40:34 you're building the future of that in many ways as others are too and we'll see that what the end state is with all of those different new ways of building and conceiving applications. Yeah, I I I love the user focus aspect of kind of those this or business focus really of of saying eventually with all due respect to the code what you're observing should be your customer's experience or your sort of outcomes and see if those are right and those you shouldn't care about the code. It doesn't matter if you've written a spec or if you've written code and doing it.

41:06 The problem identification should focus on the outcome. Now from here we talk about how do you identify why the problem occurred. So you should adapt to whatever it is whether it is a spec in Tesla or whether it is code or whether it is something entirely different in in some other solution path. Very cool.

Redefining observability around business outcomes and user experience

41:23 Okay. So we talked about some sort of trust. We talked about just how do you get observability right and maybe some of level up into the future of it. I'd like to take a slight detour and talk specifically about your sort of second category from the beginning which is applications built on top of LLM. I'm going to put aside for this conversation those building the actual LM. I think that's specialized knowledge kind of drill into that too much. I guess first of all clearly the world of observability as you pointed out has expanded to include logs to include APM alongside infrastructure monitoring.

41:51 When you think about LLM observability and observing those types of applications, do you identify kind of new categories that you think will get added to an observability platform as a default? Yes, look, we call it just LM observability for now. It might have a completely different name and shape two years from now. For one thing, I don't even know if we'll talk about models as LLMs. Maybe we'll have a maybe the shape of of what's provided by the model vendor is going to be a bit different.

42:17 Maybe the naming is going to be a bit different. We'll see. Maybe the focus on language will be reduced. I don't know. Yeah. But so AI, you're actually embracing AI as a term because it is a bit more generic just geni. Yes. Geni problem is the more you the more often you pronounce the words AI or the letters AI, the more bullshitty it

3 Highlight · Outcome monitoring replaces static spec validation for AI apps

42:37 sounds. Every buzz word in the world, right? Organic. Anything that would happen eventually would lose meaning. Yes. But we do think there's a growing part of the applications that's going to be nondeterministic. So you use models and those models are trained and maybe they're continuously trained maybe not and you what you have to measure is outcomes like what what actually happens what do they produce so you can't be in a world where everything happens when you define the application you write the spec and that's it and now you know from the moment you've done that you know the application is doing what it's supposed to be doing you actually need to to understand what happens on the other side of it now you've shipped the application it's being used is what is happening there similar to what you thought was going to be happening. Are there new things happening? Is it happening in a safe way? Is it being abused? Is it helping to reach the right business outcomes? When folks use this nondeterministic part of the application, do they buy more? Do they stay longer? Do they do whatever it is you want them to do in the first place?

43:34 All of that are super hard questions to answer and you need a new form of tooling for doing that. What's interesting there is that the companies that are building around that, they're typically not the companies that are building models like the other companies that are building on top of models. we see their needs evolve as the the world of models evolve as well. So I think it's probably going to be a couple of years at least before we know what this particular part of the stack ends up looking like. Yeah, super interesting.

44:01 So if I code this back, you're saying you say LM observability and people might think, hey, you're capturing a chat trace and things like that and those might be correct in the meantime, but really the real problem with LM observability is that you really need business monitoring or you need outcome monitoring to be able to define it because the product is going to be unpredictable. And so yeah, you need the technicalities, you need the traces as well to be able to troubleshoot problems. But if you want to know if there's an outage, you want to if there's a problem, you're not going to get that from the traces. you're going to get that from the outcome measurement. Is that did I get that right? That's right. I think the trace is very low level. I think it's rare that you're going to go and read step by step what happened either in a chat or in agent or see what actually went on there. And I think it's really a debugging and validation of some hypothesis you have about what happened.

44:49 Whereas what matters at the in the end is did you answer all the requests? Did you generate transactions? Did you generate some something else? And what you will see is yes, maybe you did. There's a there used to be 1% of those conversations that were not satisfactory to the end user and now there's 5%. Do we know why? And then you drill down and you figure out okay actually it turns out those 5% are all in the same region of the embedding space and that's because maybe there's a data source that changed and now we have bad data for that or maybe the models behind or maybe something happened there that changed and that causes an issue and you can validate that by looking at the specific conversations and all the specific interactions between the agents and the various agents and you understand what went wrong there. So this is a whole new world of understanding what's what's happening and debugging. And again what I was saying earlier is the main challenge at least for us is that the space is changing very fast and the use cases are changing fast. When we first

45:46 launched that product a year ago, what we were seeing were mostly chat bots. That's what people were implementing. Yeah. Now we see a lot more agents than chat bots. So instead of end users asking questions from the models, you have agents asking questions themselves to the model and looping. And so we'll see where we end up I would say a couple years from now. Yeah, I I love the love the thinking here and I can easily see that indeed being the future of the reservability and I guess that sort of brings you much more blends the lines with product analytics companies and it comes back to you basically expect those spaces to merge like the infrastructure application and then also business and product analytics. Yes, we have a product that's fairly new in product analytics. We built that on top of our real user monitoring product which is basically track the interactions users have with an application. We started from the performance aspect. We started real user monitoring from the question of do you have errors and what's the rendering speed and is it fast enough?

46:45 Is it degrading? Things like that. And now we're moving more into the understanding the click stream. What are people doing? What are they using? What are they not using? Do you get the outcomes you want from that or not the outcomes you want? And and obviously as I mentioned earlier, I think this is critically important for anything that is AI related because for a typical application can have a after you develop it and after you ship it or at the moment ship it to production, you have a pretty good idea of whether or not it does what it's supposed to do. For an AI application, you don't and you need to measure in production. Yeah. And I think that's probably like the core of the problem which is you I guess observability or identifying a problem requires first understanding what isn't a problem and and that is just very hard in in the world of genai or LLM or how you call it because it's just very loosely defined the systems are unpredictable to begin with the types of tasks and missions we give them is very broad and so how do you define it in the

47:46 first place do you anticipate anomaly detection to even be possible or can it only be anomaly in that sort of slightly more narrow slice of business outcomes that you can identify the anomalies but not in the activity. There's a number of anomalies you can detect. So you can run all sorts of checks on the statistical distribution of what comes out of the models. You can understand when there are changes when you use drift. You can there's a number of metrics you can track. There's even ways of there's fairly reliable ways of measuring or estimating whether or not you're getting hallucinations without even having access to the ground truth just by looking at the distribution of the data.

48:21 So there's good things you can do there that are good huristics I would say without without having the users define hey this is my business objective and this is what I want to monitor. Yeah. Yeah. Very cool. Yeah. and very much I guess at the end of the day extends the scope of what you are able to do with with the data that you expand. So maybe one maybe bring it back to security once again apologies for that. Another lens of attack is not just prompt injection but also as the data that a user interacts with and such blends with the execution and the functionality you also get into topics like abuse or fraud or such of the interaction is one that is that is very textual. Do you have thoughts of those? Is that a red line you're not going to look at the data or is that also within the scope? Just trying to get a sense of where you see the lines forming if anywhere really.

49:16 Yeah, it's hard to know exactly where the lines are forming yet in great part because we don't know yet what's what's the final feature set of the models themselves. So will will the will a lot of the commercial models and then by extension all the free models after that will they include some level of built-in protections, built-in filtering? So we don't know that. Today they don't but the space is young enough that it might change over time. One thing though that is pretty clear to us is that understanding the applications will involve looking at the data that comes in and out that goes in and out of the models. I don't think there's any way of building observation in the future without that. And for us it changes the profile of the data a little bit. When we were looking at traces and logs and things like that, this was mostly not primary data. Like we wouldn't see the doctor's prescriptions or the social security numbers like those typically don't end up in logs or traces or places like that. When you look at the data that comes in and out of the models though, you'll get all that primary

50:16 data. And so that that has different implications on what you can do with this data, how you can store it, what access you can give to it. So there's a whole new set of say functionality and protections to build around that. Great. Yeah. And I guess the good news is from a business perspective, you need to make that investment to get the opportunity to open it up. But then once you have that data, it also opens a set of opportunities for providing value to your customers. Yes. Yes. And again, you get as close as possible. It's always been the dream. Like the dream has always been look, you observe the application and the world is transforming digitally. So the application is running the business.

50:51 Therefore, by observing the application, you observe the business. I think this is becoming even more true in the world of AI. just because observing the application now means actually looking at all of the primary data that is making the business and that's that's why it's so exciting to us. Yeah. Yeah. Very interesting. Let me take it to the world of UX and so cast your eyes. We've been moving further and further into the future here as we're having the conversation five plus years from now.

51:18 You have good agentic system built up. You figured out some thresholds from a security perspective that you would engage. You've already mentioned chat is like a limited type path. What do you envision as the interface or how does the interaction with the observability system change during that time? Do you and I guess notably do you envision it changing to be more like how you interact with the operator today that uses data dog and data dog becomes that interface or do you think that's just a a fantasy? I I think look we already see today interfaces that that represent the systems as humans as being appealing to users. So some of it might be chat though chat in a way that's not you know you asking questions but the machine working with you because it's hard to know which questions to ask otherwise.

52:07 The we see that also in um in voice. We think there's many situations where especially when there's a group of people trying to work in real time on a problem where voice will be a good way of interacting. And so we we think in the end the software products and our product as one of them are going to look more like like humans you you deal with and that might be the way we interact. I think there will still be UIs on top of it because UIs are a great way to consume information and to understand what it is you can do you when you start your car it's good to have a to have buttons and to know what's going on there and not start a conversation with it. Same thing with your oven. I think it helps and we still will have that but I think it will be complemented with more open-ended humanlike interfaces.

52:50 Yeah, very interesting. So look, we spoke about all sorts of very exciting bullcase if you will opportunities that I think all are very valid and real around expanding it also just interesting to think about how observability might change. What would you say is the bare case for kind of how observability is done today with AI? Is there a scary scenario from a sitting in the sort of the lens of the incumbent in the data dog when it comes to AI? I think the things that are scary to me are hype cycles and things being oversold. So I'm not on earnings call calls. I'm not overselling the the age of agents and things like because I think as of today the technology is still not quite there yet. It's exciting. We see it on the horizon. we see all the potentiality but it's not something that is happening just today at scale on the field and so my worry is that the technology gets oversold and that trust relationship with the customers the users is broken and it's happened many times before in the monitoring and observability we as a

53:52 company we've been very reluctant to have AI on our website for the first 10 years of the company just because I associated it so much with and things that don't work and I oversold that I wanted to avoid There's a whole category of product called AI ops that existed 10 years ago and there's usually nothing AI about it. It's a the AI part of it is regax. So that's that's how it was built. Yeah. So I want to be careful about underelling and overd delivering I think is the the way we like to do things. Yeah. Very cool. Yeah. Fully relate to that. I remember explicitly taking AI off the sort of the sneak AI powered static analysis product slides of it because decreased trust its customers even though it was actually doing good oldfashioned AI on it and then needing to actually battle to get it back onto the slides as it got the mojo once more. And when you talk to the folks that have been researching and building the technology for AI for the past 30 years, like they've been through cycles of of winters where things were super hot and then got very cold and

54:53 nothing happened for 10 years. There was a bit of a fear that after pre-training was picking up like we would have a little bit of a slowdown. Turns out RL started picking up the slack pretty quickly and there's another vector of scaling now that we're all excited about. We still have to connect those two ends. still have to get to the point where most agents are good enough for most use cases and that's not here yet. Yeah. And reliably, which is probably like a key question. Cool. I've got like

Infusing cutting-edge research directly into product development

55:18 a whole hoorde of other questions to ask of you, but I think we're running out of time. Let me ask you two quick questions. One is as you build the AI powered products from a engineer perspective, the people that you hire into your development teams or R&D teams building it, is there a difference in profile? What have you seen to be like the people that are most successful, right? Most uh adept at building those products. Yeah, there's not a big change in who we hire. I think in general, we've always index most on the the ability for people to grow. And I've seen that through my career before the dog and at Da Dog where when you hire people, usually they're super smart, but only a smaller number of people are super smart and willing to grow. And the difference here is yes, I'm super smart.

56:02 But if on my second day somebody do something and somebody tells me actually you should try the different way. How about this? The people we want are the ones who are going to reply, you know what, let me try. As opposed to, no, I think I got it right. And very often people who are super smart are their own worst enemies in that way. And that they know they're smart and they they don't evolve, they don't grow. So I think that's that part is super similar. There's one new category of people we're hiring though which is that over last year we started building an AI research team. We didn't have one before like everything was applied. So we were building and everything into our product but we didn't have a proper research team. And the change there has been that if you compare to the way research was done 20 years ago actually I started my career in research. I what brought me to the US was a job IBM research in upstate New York. It was cool at the time IBM research. Maybe it is still today, but at the time I remember it being a very high appeal job. Yes, it was it was before Google research and before all of that stuff, but the so at the time the

57:04 old products were built from research that was more than 10 years old and it was very difficult for anything to cross directly from research into product. I remember that's why I left and today if you compare to that time you find into product research that is less than six months old and that research for a large part is not ever published which means there is tremendous value infusing research and product development and that's what we're doing today right yeah makes sense and those and that is a brand new profile on it and you've had to staff up where in the organization does it sit it's under my co-founder and CTO Alexi.

57:43 So it breaks the regular sort of product division. Yes. Look, it's still research with the there's one thing that's different about research is that you don't have a scrum every morning where you tell the researchers what to do. Research has to have a bit more independence. Great. And then I guess a last question is just what surprised you the most as you've been building with on AI with AI over the last couple years both for good for bad. What surprised you the most? The the rate of change is just was the most surprising. you're in the same situation like you're not new to this business. You've built companies before.

58:15 Yeah. And I think we've never seen such a rate of innovation, a rate of change. My my favorite joke on on AI and I get asked all the time how much time it's saving me and my answer is always that I haven't reached break even yet. I'm still spending more time learning about it. And I expect that to last. I think it's changing so fast that it takes a lot of time to learn about it and to to level up. Yeah. Yeah. I fully relate. And I think it also speaks back to the competency that you want in the team, which is a team that is able to deal with a rate of change that was always good, but now it's probably on steroids.

58:49 Oliv, this was excellent. A lot of just brilliant paths and very kind of promising paths of expansion also for data dogs. So that's really great. Thanks a lot for coming on to the show. Thank you. This was fun. And thanks everybody for tuning in and I hope you join us for the next one. Thanks for tuning in. Join us next time on the AI Native Dev brought to you by Tesla.