Data + AI Summit Keynote 2026 | Day 1

33,657 words · about 135 min to read

0:08 The world created an artificial brain, then locked it in a room. It can answer almost any question, but it's isolated, cut off from your operations, your world. It knows how to think. Now let's show it what's real.

0:34 Real is every decision, every transaction, every signal, every discovery, every promise, every life changed. Together, we give the brain context with all of your data. But you're in control. You decide what it knows, what it can access, who can access it, what it can do. Run it on anything, any way you want.

1:06 Because the ultimate brain was just the beginning. The real breakthrough is what you build with it. The systems you transform, the discoveries you unlock, the lives you change. The possibilities are in your hands.

1:27 5 4 3 2 1.

1:34 Let's make it real.

1:42 Welcome to the stage databicks

Massive data and AI summit highlights real-world customer impact

1:44 co-founder and CEO Ali Godsy. All right, super excited to be here. Um, that was awesome video. I don't know if many of you in the audience maybe identified many of those clips that we were showing are actually customers and their use cases that you could see in that. Hopefully we play it again and you can see it again. Okay, so super excited to be here. Uh, we have an awesome event. This is actually the largest data AI summit that we've ever done. Okay.

2:20 So, super cool. So, just a couple days I was looking and we only had 97,000 people signed up, but we now crossed 100,000. So, super excited about that. Um, my favorite thing is actually like we have 174 countries represented, which is an insane number. Okay, this is the number of languages. So, it's truly a international global event. There's people from all over the world that actually flown in for this. So uh super exciting uh and we actually have over 31 actually 3 they have the number exact down. Okay so 31,39 people uh which makes it the largest data and AI conference in the world. Uh so super cool and this conference as you know is really a community event and it started in its root with open source. We started data and AI summit. It was called Sparks Summit back in the day in 2013. And we started with the open source project

3:21 Spark, which now actually has over three billion downloads a year. Uh but we've been adding open source projects over the years. So you know, Delta Lake, Iceberg, MLFlow, Unity catalog that we open sourced. Uh but we've also added really Postgress. We've really embraced the Postgress open source project. So you'll see much more about that uh throughout this conference. Um in fact Postgress just data bricks is really embracing it. We have 17th of the governors of the project actually work for us and over 6% of the committers of the project. Very excited about the Postgress project. We think that's the future of databases. Uh and this weekend my co-founder Mate open sourced the project called Omniant. Okay. So it's a meta harness. Uh so check it out. Um I'll talk more about it and you'll also hear from him uh talking about it. Okay.

4:09 So that's the community. We also have a lot of people from the community that are speaking at this conference. I'm very excited about that. Uh we're going to hear from the co-founder and CTO of OpenAI, Greg Brockman, Satya Nadella, many other leaders and CEOs uh and co-founders of various companies, AI companies like Entropic, Cognition, Glean, Decagon, Agno, Llama Index, Crew AI. So all of those are here. Super exciting program and they're all part of a bigger ecosystem that's gathered here this week. uh ecosystem includes system integrators, people who built on top of data bricks or data sharing and AI partners that we have. Um this is super exciting and I want to give a special thank you to all of our partners that made this event possible. So let's give them a round of applause.

4:58 So please check them out. They as well as the companies that I showed in the previous slide during the expo hall. They have a lot of interesting innovations. So go check those out. Um, and this conference would not be possible without all of our customers. They're the ones that are actually doing the doing the truly amazing things on top of the platform. All those use cases that we talk about, that's really where it's happening. It's all of you that are actually implementing all these use cases and actually driving the data and AI change in the world that's happening.

5:26 Um, so very excited about those. I want to highlight just three of them because I think they're inspirational. Uh, number one is Insulate. So, Insullet has a product called Omnipod and what the Omnipod does is that it helps people with diabetes. Um, it collects actually has a glucose monitor and the glucose monitor just measures the level of glucose in the blood and then of course leveraging lots of data and AI, they can actually personalize exactly what's happening with your um glucose level and release insulin so you can manage your diabetes. So gamechanging for all the folks that have uh diabetes in the world. Uh so that's just one of the main many 20,000 impacts that you know we've talked about here. Another uh favorite example of mine and actually many of the early database employees have one of these devices uh is a tonal device.

6:18 Okay, so this is a really cool device. It's just that big and you put it on the wall but then you can do all your workouts on it and it just creates this uh crazy resistance so you can do the workout on the device. Um, but what's really cool about it is that they're collecting hundred billion seconds every year of workout data and they're personalizing it for you so that you get actually a personalized workout. So, it's like your personal PT uh with diet uh advice and all of that of course possible thanks to leveraging data and AI on the platform. So, super awesome.

6:51 Okay. And then the final one, I'm really excited about this one is Merc. Merc actually trained a transformer-based uh language model. It's called Teddy. Teddy stands for transformer enabled drug discovery. Okay, so it's one of those uh transformer models. It's like an LLM, you know, next token predictor. Um it was trained on data bricks actually with the data bricks research team, but it's not a language model. It doesn't predict the next word. It predicts uh gene regulatory networks. So it can predict which genes in the body are actually causal for diseases. So it can predict okay if you change this cell then this will happen with the next cell. This cell is just a causal cell.

7:34 This cell is you know just a reactive cell. So this helps speed up drug development significantly. So over 100

Frontier AI models have already achieved AGI capabilities

7:40 million cells that it's been trained on. Uh so it's super super cool. So uh truly inspiring AI use case. Uh so AI is top of mind for all of us. We're here at the data and AI summit. Uh so, uh I want to have a little bit of audience participation here today. Okay. Um so, is AGI here yet today? So, that's a question I have for the audience. I want to ask everybody raise your hands if you think AGI is already here that we already have AGI. So, let's see. Raise your hands.

8:12 Okay. I'm going to do like a statistical estimation here. Uh I'm going to say 5% at least in this area. I I can't see all the way down there. Okay. uh are raising their hands saying that AI here. Okay, so I would say like 90 to 95% of you say that AGI is not here. Okay, so I have a question for you then. Okay, which one of you can solve this? So the question is compute the reduced 12dimensional spin board of the classifying space of the league group G2. Okay, so audience participation, you can just shout it out if you know the answer. Okay, actually you can come up here and just share the answer with us if you know the answer.

8:47 Okay, how many know the answer? Can we see raise their hands? Oh, there's like one person. Okay, I should get them on stage. Okay, all the frontier agents and AIs today can solve this. Okay, so this is like a simple question. This is not a hard question. It's a hard question for us. Uh but for the uh frontier agents, this is trivial. It's part of actually um uh humanity's last exam, which is a benchmark that they're all uh really focused on. that benchmark has 25,500 questions. Uh they look kind of like that but in all kinds of categories. You know it could be like image recognition or various kinds and today all of them pretty much can solve half of those questions. So we believe that AGI is already here. Okay. So artificial general in intelligence has already arrived. Um in fact I came to the United States to do research at UC Berkeley at the lab called the AMP lab where the A was for algorithms and AI.

9:45 It was the largest AI lab at this time 2009 and actually this conference came out of software that we built at that lab and the definition of AGI that we had back then we've already satisfied that by leaps and bounds in fact went back and I asked many of the people that were in the lab what do you think do do we have have we passed AGI or not the way we defined it back then in 2009 and everybody says yeah of course of course we do it the way it was back then but you know goal posts are changing and so on so AGI is already here okay it's plenty smart. Uh we don't need it to be more intelligent. Um AI does not have an intelligence problem right now. These are not the kind of problems that we need inside of our organizations. The problem we have is that AGI is not really um per completely um permeating our organizations. So inside of your companies, you don't have each hundreds of agents working for you, doing autonomous work, collaborating with each other, coming up with proposals, negotiating, and then we're just managing agents. Most of us is just

10:45 using chat bots and asking them or maybe we're using coding tools and they're writing code for us and we're using it agentically like that. Um, but we're not actually sort of fully realize that vision that everybody's talking about and that's my why maybe many of you didn't raise your hand when I said is AGI here. Uh so the question is how do we actually enable this at work? So that's what the next couple days will be about. We want to figure out how do we enable all of you uh to sort of lead this revolution. It's basically our jobs here to figure out how can we actually make this AI work. That's what you all are doing. So that's what we're focused on. So in a nutshell, what we want to do is we have all these AIs. They've been released by many of the vendors. There's open source. There's frontier models.

11:28 How do we figure out to take all the data, all the processes in your organization, everything that you have in your head and provide it as context to the AI? If we can do that, we think that the AI can already solve many of the tasks that we wanted to solve. Okay. So, if we just give the context to these very smart models that can solve those super hard problems, they'll be able to do amazing things inside of our companies. You know, most of the task we're doing is not about solving 12dimensional spinboardisms. They're about, you know, getting the information out of Salesforce, summarizing it, putting it in another place, going through it, preparing it, and those kind of tasks. So, you just need to get all

Enterprise AI adoption requires solving context, control, cost, and lock-in

12:07 that context. So, it's as simple as that. But getting this context into the AI is harder than one could imagine. Um, it's actually really, really difficult and getting the perfect enterprise context is elusive. So this is the problem we've been focused on and we're really focused on this problem at data bricks. Um so context is actually itself many other things. It's not just saying context how do we give it to the AI. Context means how do we get all the data that you have in your organization and actually connect it to the AI. Okay data could be what are all the meeting notes.

12:40 So every meeting that happens in the organization should be recorded and we should get the transcripts of that and we need to reorganize our processes so that we can do that. And actually at Terex we have four deployed engineers that come and help reorganize things like that. But it also means connecting to data silos that you have or data that's sitting uh tucked away and you don't have any uh access to it because maybe security has not approved it. So this is step number one. How do we get your data AI ready so that we can actually give it to the uh agents. Okay.

13:10 But let's say we figure this out and we figure out how to get access to all the data. Then we're still not done. the context problem isn't solved because we could just take open claw and just unleash it on all of your data. Okay, when open claw was released over 10% of the skills in it uh were actually malicious. Okay, so step number two, how do we do this securely? How do we do that with control? Okay, so first is context. Now, how do we do this with control? Uh so that's really really important. If we're going to have agents roaming around doing various things, we have to make sure that they uh are satisfying our security policies, that we can audit them and that they don't uh do something that we don't uh want them to do. Okay, let's say we solve this as well. It's still really hard. Um if we just have agents running around for looping live checking through all of the data, then it's going to become extremely costly. So we have to also solve the cost problem. Okay, this is sort of just shooting up at most of our customers. We're seeing the costs just completely run away. Uh Uber CEO said that in one quarter they blew through

14:11 the whole year's annual AI budget. Okay. If this continues um it's completely unsustainable for most most organizations out there. You cannot have your costs completely outrun your revenue. You will go bankrupt. Okay? And your companies will not allow that. So they'll just shut it down or stop it. Okay? So it's just infeasible. So we have to also solve the cost problem. But even if we solve all of these things including the cost, we get to the most important thing in my opinion which is how do we make sure that we can do that uh without lock in. How can we do that in a way where you have choice? Okay, this is super important. This is the hardest one because some of you are representing organizations that are less than 10 years old. So this maybe doesn't apply to you, but most of the organizations in the room, over 90% of you represent companies who have a long history. They've been around for 20, 30, 50, hundreds of years. some of them. And the thing I've seen in the last 20, 13 years is that the stack that you have at your companies is just very complicated

15:11 over the years. You keep buying new software. You keep adding more things to that stack and you just get locked into all those vendors and they come and go, but the software remains and there's always someone who wants that piece of software in your stack. So, it's hard to rip it out. So, how do we do this without getting locked in both to the data, the context, but most importantly also now to the AI? We don't want to get locked in just to one particular AI. We want to have choice when it comes to the AI. So that's what we're focused on today. My whole keynote is just focused on how do we solve these four. Okay. Uh so I hope I can convince you uh today that we're making some steps towards solving these four. Uh I'm not going to say we've completely solved it. Context is solved. But at least we're making uh some big leaps towards solving these four things. So the four things we're focused on is we want you to have choice and not get locked in. uh we want to be able to get the security and the control and the cost control as well as the context and then get that to the AI and that way we can have more impact together. Okay, so how do we do that? So let's start with choice. Okay, it

Open lakehouse infrastructure ingestion via LakeFlow and zero-bus architecture

16:12 started with your data as I mentioned. So we have all the data and we want to get that data in an AI ready form. So step number one is we've actually made huge progress just in the last year to get that data and move it into an open lakehouse. So open infrastructure on top of any of the clouds and we get your data ready there. That's what LakeFlow does. So LakeFlow has actually been developing a lot of connectors over the last year. So two years ago when I was here, we barely had any connectors and today we now have over 100 connectors.

16:46 So LakeFlow can now connect to all these different systems that you have. It's not just Salesforce workday Netswuite. It can pull data from Meta or Google Analytics or from any of your docs or any of the corpus. And if there's any data source that you do not have, please let us know and we can help you get those into the open uh lake. Okay. And it's not just getting the data in there. We can do it in many different modalities. So three things that I would like to highlight. One is we released Zerobus last year. It is now G8 and actually we've added a lot of capabilities to it. So what does Zerobus do? So as the name implies, you don't need a bus anymore for the data infrastructure. So instead of using something like Apache Kafka, you can just hit this API of zero bus and you can give it data as it comes in. It could be bursty, it could be tiny rows.

17:32 You don't need to worry about having millions of files or any buffering or anything like that. Zerobus just takes care of taking all those entries that you hit it with at very high rate and it just makes sure that those appear in your open lake in a way that you can probe it later. Okay, scalable, manageable. Okay, so that's great. Um, and then last year we announced Spark Realtime Mode. Spark Realtime Mode or RTM as we call it is an open- source framework that lets us actually get down to tens of milliseconds. So this is really amazing.

18:04 Uh, so we can do now operations super fast. Historically, Spark was great for everything, but you couldn't really get down to 10 milliseconds. Okay, so the real-time streaming would always take this micro batch of 1 second or so. But with real-time mode, we can now really make that super fast. So many of you for your streaming infrastructure, you might have Apache Flink. That was the only option that uh many of you would pick. But now you can actually do a lot of those workloads directly in Apache Spark with real-time mode, which is open source.

18:34 Okay. And the final one is LakeFlow Designer. Uh I personally actually really like doing all of my data engineering and data movement using LakeFlow Designer. So the way flow designer works is that it kind of looks like Altrix. Um it's visual. You don't need to see any code, but you speak to the AI, which is Genie, and you tell it, hey, I want to do this transformation. I want to join these two tables. And it just visually shows you what that looks like. And then you approve it. But under the hood, it's all just Spark, open source Spark. And you can actually inspect the code, you can version control it, you can make it production ready. Uh, so that's super awesome.

19:11 Okay, so those are three things. They're all GA. Um, so you can now get all this data, use any of these modalities, all these connectors from these different systems and you can get them into, you

Apache Iceberg v3 and Delta format unification eliminate storage silos

19:21 know, well, which format should you pick? Should you pick Delta Lake or should you pick Apache iceberg? Okay, so that's a big question and actually I want to welcome to stage the original creator of Apache Iceberg, Ryan Blue, who was the CEO co-founder of Tableau that we acquired last year. So let's welcome him on stage.

19:48 Hey Ryan, how are you?

19:50 Good. How are you? Good. I want to embarrass you with a video uh from a bunch of years ago. So this video actually had a very interesting title. Uh it said

19:58 you guys

19:59 why you shouldn't care about Apache iceberg. I mean listen to it

20:01 I care deeply about um and I I sincerely mean that um you shouldn't care about Apache iceberg.

20:09 Okay, we shouldn't care about Apache iceberg. Actually, I agree. I think we shouldn't also care about Delta Lake. But why did you say that?

20:16 No one should have to care about formats. You you should be able to use whatever tools you want with all of your data. uh use the right tool for the job which is why we built both formats in the beginning.

20:29 Yeah. In fact, it's super funny. Last a few years ago, you know, I would get to customer meetings and they would get into the nitty-gritties of how iceberg was implemented versus delta and they would ask us like, yeah, but can you do this, you know, deletion vectors this way and that way? And I was like, this is not where the industry should be going. Uh so, how are we doing? Where are we? What's what's going on with the versions and all of that? Well, we just just did the GA release of uh iceberg v3

1 Highlight · Unified data layer eliminates file rewriting between Delta and Iceberg

20:53 support, managed iceberg tables in DBR. Uh that is going really well. We're following that up later this year with the unified metadata layer that is going to be built into Delta 5 and iceberg v4. Um so we are very close to the full unification vision.

21:11 So what's special about V3?

21:13 Uh V3 is the unified data layer. So you no longer have to rewrite any data files to share them across Delta and iceberg tables.

21:21 That's awesome. So basically the data that's laid out whether it's iceberg or delta looks like identical the big data now. Right.

21:26 Exactly. Yes.

21:27 Awesome. And then with V4 just a little

21:30 bit of metadata that's left and hopefully when do we get that?

21:33 Uh we're aiming for later this year. I don't want to make too many promises but I think that we should finish the spec uh you know sometime in maybe Q4.

21:42 Awesome. Super excited about this. Great job by Ryan and the whole community, iceberg community, Delta community. You shouldn't have to care about this. Can we stop talking about this now?

21:51 Yeah, I got work to do if if we're done.

21:54 All right.

21:55 Thank you.

22:02 All right. So, it doesn't really matter actually if you're using database. It doesn't matter if it's in Delta or iceberg. We don't care. Um, it's actually the same now thanks to V3. It's the same data format that's on the disk. Okay. Perfect. So that's that. So now we're done with how you can actually get your data into this open format. It's really seamless. Just one way of doing it. It's all spark based under the hood. Um so let's move on and look at how we actually then uh start doing analytics

Lakehouse analytics and Lakebase Postgres bring transactional branching to agents

22:26 on that data. So the data warehouse that we announced about four or five years ago. Uh we call it lakehouse. In fact our tagline was the best data warehouse is a lakehouse because it's open. So you control the data yourself and it works with machine learning workloads. And we made huge progress on our lakehouse in the last year since last data AI summit. We've added over 110 features that legacy data warehouses have so that we can more easily migrate those over. Um we also released a lot of AI functions.

22:57 This is the main thing that people want with lakehouse. So there all these AI functions you can just call. So you have your data maybe they're in tables that that's the kind of thing you work with in data warehousing. But now you can just in your SQL queries call this SQL AI functions and you can say can you do sentiment analysis tell me for each of these cells what do people think or you can tell hey for each of these cells can you extract for me the company names that you see in it or can you do a prediction of what this number is going to be and they can just do that at really high scale. So this is the most popular thing that people do actually on lakehouse. Uh and we've taken lake bridge which is an automated converter that lets you migrate from existing data warehouses that you might have on prem or elsewhere in the cloud and we can do that now really really well. So very excited about that and actually uh you all are using this. So it's we've over doubled the consumption of our data warehouse just in the last year. So we're very excited about this. Um, but what I'm really excited about is my co-founder Reynold getting on stage and talking about the new engine. Okay, so

23:58 we have a new engine. Uh, he's going to talk about it. It's called Raiden. And what's really cool about Raiden is that it's the world's fastest engine that we've ever seen. We've benchmarked against anything else. We've gone the latency of it down to just tens of milliseconds. So far faster than anything we had done before. Uh, so it's very very exciting for us. So we'll have him uh in a couple of talks in the keynote today uh talking about this. Okay. So that's our data warehouse.

24:24 That's the lakehouse. Um what about transaction processing? So I mentioned in the beginning we really love Postgress. We've really embraced Postgress. We think AIs today really prefer Postgress. If you ask any of the AIs today, what kind of database should I pick? They will say open source Postgress. Okay. Uh but we're excited about what we call lake base. uh which is a way in which you take Postgress and you separate it out so that you have all of its storage just sit on cheap lakes okay so now we're talking lakebased so this is OLTP right it's like my SQL Oracle transaction processing and once you have that data sitting on the lake you can now start doing amazing things on top so what can we do on top well we have a little animation here if you look you can get serverless autoscaling so you can scale up and down Postgress And this is really fast. So you can do it in less than a second. You can scale it even down to zero. That's really awesome because if nobody's using it in the middle of the night, you don't have to pay for it. You just pay for cheap storage that's sitting in the lake.

25:25 Okay, this was not possible before. You know, others had specialized storage, but this is just sitting in open source format on your lake. So why do you care about that? Why should your company care

2 Highlight · Instant database branching for AI agents using Lakebase Postgres

25:35 about that? Because it significantly c cuts the cost of running databases down. So you get significant reduction in TCO uh by doing this kind of autoscaling. So that's a game changer. But that's actually not even the coolest feature of lakebased Postgress. The best thing about lakebased Postgress is that it has this feature that agents really love. And that feature is called branching. So it kind of looks like that. I don't know that's that's our attempt to animate and visualize that. But what that means is that you can take a gigantic database pabytes and you can say hey can you make a clone of this for me? and it will in less than a second say done. What it's doing is not actually copying the whole database. It's just keeping track. The data is anyway sitting on the lake.

26:15 It'll just say okay, we have two copies, but it's actually just one copy. And then when you make a change to one, it'll just keep track of like, okay, this change is in this copy. This other change doesn't exist in the first copy. So what's called copy on write. So it actually does that for you. Why does that matter? Well, it matters because it actually completely changes the way you organize databases inside of an enterprise. Most of enterprises have thousands of thousands of database instances that they're running. And we have DBAs that are running around trying to manage all of these. And the way you do that, the reason you have thousands of these and you pay for all these thousands of copies of the data that are siloed inside of the enterprise is because you need to have uh user acceptance testing database. You need to have production database one, production database number two. You need to have staging database, R&D database. So you have all these copies. Okay. But branching what it enables is to just have one instance and then you just have different branches of it and that's just lower TCO easier to manage and agents love that because agents love to just branch out and just experiment with the

27:16 data try something out and they want to do it quickly. So that one second latency is very important for them. They don't want to wait 10 minutes on a database to come up and then finally if they if something goes wrong they want to be able to go back to an earlier snapshot. if the data gets corrupted or

27:30 the agents did something by mistake, they should be able to go back to an earlier version. So this is really awesome. Um so please try this out. If you're doing any development going forward, please use like based Postgress. Um in the last in the next 12 months, we're going to see more software be written than ever in the history of mankind. Okay. Uh all that software that your organizations are going to write using LLMs and coding tools need a database behind the scenes. Let's make sure it's just open source lakebased Postgress with low cost. That's what these agents love. But today we have another talk that I'm really excited about and I'm not going to talk too much about it here, but Reynold's going to come back and he's going to talk about something we called LTAP. Okay, Lakebase and Lakehouse unification. So it's like a breakthrough that actually the industry we've been working on for 40 years. We think we finally pulled it off. Uh so he's going to come and talk about that as well. Okay. So we talked about Lake Flow to get the data in. We talked about lakehouse which is a data warehouse to do analytics. We talked

Unity Catalog and AI Gateway provide centralized governance and routing

28:25 about lakebase which is transaction processing and uh the talks that we're going to have on that. So hopefully we now have the choice the data layer at the bottom ready for the AI. Okay. So it can be consumed fast by the agents and it's seamless for them and they can experiment with the data. So this is really important. We got to get that data layer right. But as I mentioned we also need to figure out control. Okay. So we can't just unleash the agents on all of the data like that. we need to make sure that we're controlling what they're doing. So that's why we started the Unity catalog project a bunch of years ago. When we started that project, what the industry was doing is that they were having cataloges for structured tables. So data warehouses were looking at structured tables, rows and columns, and they were saying, "Okay, well this column is somebody's salary information.

29:10 So can we hide that?" So that's what this was about. Access control on structured data like that. But we called it Unity because we think you shouldn't just do control on structured data. You should do it on all the data assets that you have in an organization. Any data and AI assets. Okay? So you could have unstructured files, PDFs, you might have AI models, uh, and 50 other asset types that we added over the years. We want to be able to do it with all of these.

29:35 Okay? So that's idea number one for Unity catalog. Number two, you don't want to just do access control, but you can lock everything down and you're done. You want to also be able to do discovery and democratize the data in the organization. You want to also be able to track the lineage of the different data sets. Where does this data come from and visualize that graph? You want to be able to do cost controls on everything that's being done because otherwise the cost just will run out. Um you want to be able to look at the data quality of everything you have in your organization. So these are all the things that are part of Unity catalog and it's extremely important. That's why we never actually charge for Unity catalog. catalog is just free and we actually open sourced it a few years ago and we're doubling down on open sourcing more in the last months. We've added more people that are open sourcing pieces of it. Uh so it's moving very very fast. We're excited about that. And also we announced that you want to be able to share data. So this was really important for us. Um the rest of the industry they were doing this kind of lockedin sharing where they were saying buy my software and then if you want to share it with someone else they should buy also my software and they can share data sets between it. That's why we open

30:35 sourced Delta sharing a bunch of years ago as part of the Delta Lake project and now you could take any data assets and you could share it with anyone in the ecosystem. It's actually have great growth, lots of partners in that. But what we're very excited about this year is that we're actually open sourcing a new project that's a supererset of Delta sharing. We call it open sharing. And what this enables you to do is share not just with Delta, but also with iceberg because they're now the same format, but also not just share data assets, but also AI assets. Okay, so now you can share your agents, your skills and your models and you can do that also on premise. Okay, so we're very excited about that. This is very cool.

31:17 Okay, so we got all these different

Proliferating enterprise AI agents create severe cost and control challenges

31:19 things you can do in Unity catalog. We have open sharing but in the last year what has happened is that uh governance which always has been AI governance has always been part of unity catalog is becoming very complicated in most organizations. Okay, so we're seeing that organizations now have uh lots of agents. You're all building agents. Maybe you're using Llama Index, maybe using Lang Chain. Um the SAS providers are now each providing you an agent that they want you to use. They have agent builders that they want you to use. They have their own MCP servers that the SAS vendors are saying, "Please use our MCP servers. Here are the skills you can use with those." Your organizations, your developers are themselves developing MCP servers. They're themselves developing skills files. there. Um, all these models that are being developed now by the frontier vendors. There's many different versions of them. They all have different capabilities. You know, they exist, they don't exist. Um, there's coding tools out there. Um, vibe coding things. And then now we're starting to see more and more co-work like agents in the organization. So, it's kind of a quagmire. It's a complete mess. Okay. So, what is the problem with

32:22 this? So, problem number one is that actually cost is running away. I think this is the biggest problem right now. uh anyone we talk to, they're really worried. The skies of this are just skyrocketing. Um and no one has visibility. Is it someone's agents? Which model is it? What were they doing? So all this token maxing is happening and there's no way to actually set limits on everything holistically or rate limited. So that's problem number one. Problem number two is that there's no way to control what the agents are doing. How do we make sure that the agents actually have access to the appropriate data that we can audit them that we have guard rails for their inputs and outputs? And how do we manage their identities? Who are they be acting on behalf of? How do we track this? And then finally, and this is the most important one, as I said earlier, how do we do this with choice without lockin?

33:08 Because the state-of-the-art models now have a lifespan of one month. November last year, we were super excited about Gemini came out from Google and it was like the best model and we all were switching over to it and we said, "Okay, that's that's it. This is the best model ever." Then February time frame, January, February time frame, it was like clearly Opus is the best ever. And then OpenAI came out with GP55 and I was the greatest model ever. And then last week, uh we had Mythos or Fable and it was out and we were all going to switch to it, but now they canceled it and then you know it's back and forth. So uh how do we make sure that we can actually have flexibility uh so that we can pick any of them and that uh we will have choice here. Okay. Uh so those are three

Unity AI gateway centralizes AI spend and governance

3 Highlight · Unity AI Gateway centralizes multi-model governance and cost control

33:48 problems that are really important and that's why we're super excited to announce as part of Unity catalog uh we are breaking out one place where you can manage and control all of your agents all your AI spend. We call it Unity AI gateway. Okay. So excited about this. So what is Unity AI gateway? It's a single pane of glass and there you can actually have your one entry point for all of your agents, for all of your AI, for everything that happens. Any of the harnesses that you're using can go through this. Again, uh this is part of Unity catalog which we open sourced. We don't charge for this. And also the AI gateway is part of ML flow. So it's also open sourced. Um, but one of the most important things that we do that not maybe everyone in the room knows is that we will actually provide you capacity uh for the frontier models. So let's say you committed to pay $100,000 to data bricks. You might use lake flow for that to consume things. You might use our lakehouse but you can also just use

34:50 tokens for that $100,000 directly from us on OpenAI's models or Antropics models or Gemini. And you can do this on any cloud whether you're on Azure, GCP or AWS. You can just use this. So this is available on the platform. Uh so we actually offer that. Two, you can

35:06 actually get perfect uh observability on all of the spend inside the organization with dashboards that give you the observability of how that spend is happening. And you can set limits. You can set budgets. You can set budgets on different groups, subgroups, all the way down to the individual level. And when someone exhausts that budget, uh you will get an alert. uh or you can actually just stop it and rate limit it so that you uh you can make sure that nobody's spending too much or that they every day get an email saying how much they've spent. So this is super important. And then finally we can enforce safety, compliance, auditing and identity management of all your AI.

35:43 Okay, this includes harnesses. This includes models and agents. So very excited about this. So that's uh what Unity AI gateway is. And now you can manage all these asset types. MCP tools, models, agent, skills that you have in the organization. And I just want to emphasize this is not just MCP or the tools that you have in data bricks. These are any MCPs that you have in the organization by any vendor. You register them in unity gateway and you will track all the usage and you will see how it's going. And in fact, it has a really awesome capability where you just authenticate once for the MCPs and it'll do it for all of them. So you don't have to authenticate again and again again.

36:19 So this is really awesome. Check this out. Okay, so now we have uh also control and we have cost. So now we've dealt with how do we get your data ready? So hopefully we have a data ready in an open format that's fast for the agents. Um we figured out how to do control so we can do governance and how

Live agent retrieval loops suffer from latency, cost, and low accuracy

36:39 do we do cost management with Unity AI gateway. Okay, so that brings us uh to the context layer. So as I said AIS are plenty smart. AI does not have an intelligence problem. It has a context problem. So how do we actually get that context into the AI? So this is the most important thing. Um so what do I mean here? So let me give you an example of how it works today. Today you're using these agents. You might be using a coding agent. And if you look at the best agents out there, they kind of work like this. There's a big graph of all the documents and everything that everyone is doing inside of your organization. And what they're doing is they're in these red dots. You can see they go when you ask a question for instance say how is my product revenue um last two weeks in Europe it goes and it looks at one document and then it reads that document and it says okay is there anything interesting and then oh there's a link and then it goes opens up an MCP server to another document it reads that document and it's sort of for looping live through your corpus to get the data okay but there's big problem with this approach that all the agents

37:40 today are using all the frontier ones uh problem number one is that it's very timeconuming ing this live loop of going searching for your data live uh is just going to take very long time. Okay. And we all see it like you ask a question, even a simple question, and it takes it 10 minutes, 15 minutes to just come back with an answer because it's live going through the MCP servers. It doesn't know where to search. It's just trying live to do this. It doesn't have the context. Number two, the costs are super high.

38:07 The joke is you tell it, please rename this file for me, and it takes 10 minutes, and it charges you five bucks. Okay, so that's kind of how it works because it's again it's going through and it's doing the search and it's using up a lot of tokens to construct all of this. But these two are not even the biggest problem. This this gives you really bad user experience and it's expensive. The biggest problem is that the quality suffers. The reason that is is that all of the documents and all the data assets in your organization is gigantic. Okay, so you're really just doing a small random walk of a small subset to get to the answer. It's remarkable the answers it gives you.

38:38 It's gamechanging. We're all excited about it and we all were using a bunch of it and the costs were going up. uh but actually if you look at the accuracy it's uh quite problematic. Okay so um how do we solve this? Uh so we are very

Genie ontology constructs enterprise knowledge graphs using OntoRank

38:51 excited to announce and this is one of the most important things I'm going to announce genieontology. Okay so what's geneontology? So let me explain. So geneontology what it does is that it actually connects with all the data that you have in an organization and in the background so this is not when you're asking a question this is happening in the background uh it's constructing a graph of the most important knowledge that you have inside of your organization and it's making it ready so that we can give that context to any of the agents. Okay. And one thing that's really important is that it's not just everything that you have in the lakehouse and everything you have in data bricks. It also connects to your Google drive. It connects to your uh SharePoint. It connects to your email, Google calendar, anything that you set up. And it can actually branch out and access all of these things that you have there. And the research team developed an algorithm uh called onto rank. Okay,

4 Highlight · OntoRank algorithm for enterprise knowledge graphs

39:48 I know it's a bad name, but you know this the research team came up with it and it's like ontology rank. Um but what what is onto rank? So what on rank does it's it's kind of like page rank for those of you who remember. So page rank was how Google uh 25 years ago or 30 years ago figured out how to take the whole web and create an index for what are the most important sites before Google Alta Vista Yahoo these sites um they were just looking how many times is this word mentioned on a site it led to these C SEO uh search engine optimization where people adding the same word thousand times on a web page to get higher ranking but page rank it could just figure out from the graph itself what's important okay so onto to rank does a similar thing on all of the assets that you have in an organization, but it's not page rank. This problem is harder because it's not just web pages with links. It's all the assets that you have, but they're actually different asset types. So, it needs to identify, you know, source code is different from a Google doc. Uh, also here we have

40:49 users and users inside of your organization, they're part of an org chart and it actually matters how often they access things. So, we have this information as well. So we take all of these different asset types, all the people in the organization and we construct this ontology graph that then

41:02 helps us actually um do amazing things with this for the agent. So this is the context that we're going to feed. Now you might say, hey, I already put together uh context. I already built semantic layers at my company. I already have a business glossery that I built over the years. Um Genie can take that from any of the sources. So you can actually bring your own uh semantics as well. So as part of uni catalog we have something called uni catalog semantics where you can bring from our partners whether it's from Atlan or at scale or from um uh existing BI tools that have semantics or any business glossery that you already have and you can have humans curate that you'll happily take that and it'll help it get smarter. So depending on whichever camp you're in, do you think this should be completely autonomic and happen by itself?

41:45 Geneontology does that. If you want to have lots of manual curation, you can do that with model semantics. They both work well together. Then this these are fed into our agents. So we'll talk about the agents that we have and we'll give you demos but there's three categories of agents that are important. I'll talk about them. Genie one, Genie agents and Genie code. And they all are fed this context which makes them faster. It makes them uh much more cost effective and it makes them much higher quality.

42:14 Okay, so you can much faster get the quality answers you want without paying too much. Okay, so that's what this is. Um, if you're using other agents, you want to use other agents, they can use Genie as an MCP server and they can just connect to that agent Genie and they can actually get the semantics from Genie. So, you can also do that this way. You can bring any client, any agent you want. You can connect to this ontology that you have there. Uh, so that you can get that semantics and the context into any existing AI that you have. Um, so this is what it looks like in the product. You can actually click on the graph and you can sort of explore it. In this particular graph that we're exploring here, this is just one instance of data bricks for ourselves.

42:51 This is our own data bricks that we have for data bricks employees. It has over 4.5 million ontology data snippets that it has collected and constructed. So obviously nothing you could do live like it would be infeasible for any of these agents to live go one by one and look through 4.5 million ontology snippets. Uh but it's done that and it's constructed this so that you can

Genie suite delivers tailored agents for business users and engineering

43:12 actually have this. Okay. So um that's super awesome. So this is now fed to our agents. Uh the most important agent that I'm excited about is Genie 1. Okay. So let me tell you about Genie 1. Some of you might already be using Genie at Databix for the last uh 3 years. Genie has been part of the platform. This is not that Genie. This is Genie 1. So it's the one place you go to access all of the data assets in your organization.

43:38 Okay. So Genie that we had previously that we're using, it's still there. Um but that's confined to a certain domain and you have say marketing people asking just marketing questions in a genie uh space or room. Genie one is just one place any of your business users in your organization can log into. So you should just enable automatic identity management for all of the employees if you have 100,000 employees. They then log in. They just get genu. They don't get the data warehouses. They don't get all the rest of the data bricks infrastructure. you know, they're not going to be able to go and spend spin up data warehouses and spend a lot of money. They're just getting a simple interface that looks like this and they just ask their questions there. So, it looks like any other agent that you might have. Uh, but there's some differences here. First of all, it now can get data from any of the sources that you have at the bottom of the screen. So, whether it's Google Drive or SharePoint or whatever it is, it of course has all the goodies that other agents have like you can have skills, you can schedule tasks and do co-work uh um uh activities for you. Uh but the key thing is it has access to it genie

44:39 ontology. Okay. And that changes everything. So very excited about Genie1. Uh please enable your organization to have it and also use it. Uh my favorite way of using is in the phone. Most of us use it on the phone on our iPhone or uh uh Android. Uh and you can see it's visual. You see all the graphs and it's integrated with our dashboards. AIBI. So it's basically like a BI tool or a dashboard. Okay. So that's GI1. Uh we're also announcing uh Genie agents. So what's Genie agents?

45:06 You can take any conversation that you had with Genie1 and you can say hey turn it into an agent. You can take that agent you can now make it available to the whole company. So for instance you can say hey here's all my HR documents and create that agent have a conversation make sure that it can answer questions about your HR documents. Now you can make that available to the whole organization. Anyone can ask any HR questions they have from that agent. You can put it in Slack you can put it in teams any modality they want. Um but it also can do more. It can actually completely do uh autonomous work for you. It can go uh every day fetch information from say Salesforce, prepare it, put it in workday, uh send you a report, uh work with other agents. Uh so very excited about Genie agents. And then we have Genie Code. Genieode is special because um Genieode is similar to other coding agents that are there, but it's really good at two things. Okay, it's really good at data engineering. So you know 13 years of data engineering on the platform we made it really good at writing pipelines for you, writing code

46:06 for you, doing data engineering. So that's super awesome. Uh it's also really really good at machine learning and data science. Okay, so it can go and build you a machine learning model. It can train it on GPUs. It can test that the accuracy of those machine learning models is really high and it can really democratize access to machine learning inside of your organization. So that's really really important. Uh so these are two skills that it's you know uniquely capable of on the platform. So check out Geni code and um we also are launching something I'm really excited about. I think this agent actually is going to be the big biggest transformation transformative agent for most of your

5 Highlight · Genie Zero Ops automates middle-of-the-night pipeline fixes

46:41 organizations. We call it Genie Zero Ops. So Zero Ops as the name implies. What it does for you is it runs in the background. It looks at all your data pipelines or if you have data science, machine learning models and it sees that okay, this pipeline is down at 2 am. It will then go investigate why is this pipeline down? Oh, the pipeline is down because it's there's an error. It's saying that you know this data I'm not familiar with it. There seems to be a new column in this data. It'll go see okay, what's the name of that column?

47:08 Can I add that column? It'll actually experiment with writing the code in a separate environment without affecting your production pipeline. And once it's ready, it'll prepare it for you and it'll send you a notification on your phone and you can see and you can review it and if it looks good, you can just click accept and it'll fix your pipeline. Okay, so you're still a human in the loop. Of course, you can just click do it automatically, but I would not recommend that. Yes, people are excited.

47:33 Yeah, I'm personally really excited about this. No one wants to wake up at 2 a.m. You know, we have all these folks that are on call and you know, this pipeline broke. Why did it break? go you have to go investigate and you spend you know two hours in the middle of the morning and everybody's yelling uh this is the world of data engineering uh so with zero ops we're really excited that you can actually make this much simpler and then finally we'll talk about apps building your own apps to democratize access to data and AI inside of your

47:56 organization uh but now we make that really simple to vibe code your own apps you can of course use our partners you can use lovable replet any of these but we also now give you uh on genie genie app builder uh which is which can do that for you okay that's genie Uh and key thing here is geneontology that we automatically construct for you. That's really the secret sauce. Um but the cowork and accessing the data outside of the lakehouse is also really important. A lot of people think of data bricks as okay this is the data I have my lake house. No this is any of the data that you have in your organization.

48:27 But what about developers who want to do low-level coding? They want to do coding agents. They want to write software. Um you know they want to use different harnesses. Maybe they're using cloud code. Maybe they're using codeex. Maybe they're using open code or pi or something else. So that's where agent bricks comes in. So this is for developers who are doing development work and uh we've expanded that developer platform so that we now have sandboxes so that you can run your agents in really fast sandboxes that come up and isolate them. We have agent memory uh and we announced this weekend a meta harness called omni. So mate announced that and he'll be giving a talk on this and explain what that is and how that works. But it's a harness of harnesses. So it can leverage all the existing harnesses and it can actually have them play against each other. It can have cloud code fight uh codeex and actually get a much better result than either of them. So he will give a talk on that as well today. So very excited about that. Okay. So uh that's what it looks like. So hopefully uh I've shown you how we can now take all the data that you have in your organization.

49:28 Okay. So the first half of my talk, how do we take the data make it ready for AI? uh how do we actually make sure that we have cost and governance control and security uh figured out and then how do we get with geneontology this context into the agents um so that's great but I think actually something more profound is happening in the industry something is happening with software okay and people are talking about it uh so what is that so what is the classic software as a service SAS stack look like over the last two three decades so it kind of looked like this you had a system of record which was a database um that's you had this most important golden records about your HR data or your CRM data or whatever it was and you had some custom workflows uh inside of that uh somewhat rigid but you could customize them if you brought in uh contractors that would augment that for you would take a while and it was a UI it was customizable no one really loved those UIs but most of us learned them and then once we learned them we didn't want to

50:28 switch to a new one and we had one of these for each of the domains inside of the organization. So you have one for sales, you have one for ops, you have one for finance. So there's one of these for each. But what's happened in the last two years is that each of these vendors are now providing you an agent of their own. So they're saying, "Hey, we have an agent for this. Here's use my agent to access my system of record data here. You know, we're headless. You can just access it with the agent." But the problem is when you go to these agents and you start asking questions, you never have just questions about the data in that system of record. your question spans things. It need to access data that's elsewhere. So, it needs to go and access systems that are elsewhere. Okay?

51:05 And this gets quickly very complicated and most of these vendors don't want to play well together anyway because they're competing with each other. Uh so, it's sort of very messy inside of the enterprise right now and it's very confusing. How do you make that actually work and go across these different systems? So, the big question is what does the future of software stack look like? You know, or the SAS software or the apps of the future. Uh what do they look like? What are they going to look like? Okay. And we don't know but we think it should look something like this, right? You have all your agents and they should just access um agent system of record or a gentic system of record. They should just be there and you should be able to access that seamlessly. Should be very fast. You shouldn't have to go through lots of other agents. You shouldn't have to uh try to go to a production database.

51:48 That's too risky. You should just be able to have it at your fingertips and let these agents do that really fast, quickly, costefficiently, securely. So, uh, what would the agent system of record look like in the future? What do we think that's going to look like? Well, we think it probably looks something like this. You have all your data and you can access it all in one place in an open format. Okay, you see where this is headed. Okay, so I can just click through the next few slides.

52:14 Um, on top of that, you want to have unified governance, right? So, you want to be able to have control. You want to be able to do cost management on top of that. And of course, you don't want these agents to go and go one dock at a time and spin up tokens and not know what they're doing. You want them to have the enterprise context so they can quickly do that. And the context should span all of these different organizations. Okay. So, we think that the new system of record is actually the data and AI platform that uh we've been talking about all along. We think that the world will look more like this.

52:43 Whether it's data bricks or someone else, but this is going to be the structure of the gentic system of record of the future. That way you can get the choice. You get choice of running this on any cloud, any SAS vendor, any of the AI models, you can do the cost controls, you can do the governance control and of course you should have the context for this. So that's what we think the future looks like. So let's augment the stack with that as well. So this is what I already presented to you those three layers. But now on top of that, we also get aentic apps. So databix apps is very exciting. Uh we'll have a whole talk dedicated to this. Databix apps is how you democratize access to data and AI inside of organization to hundreds of thousands of people and it could be custom um apps that you've built. Maybe you vibe coded these um there's a huge proliferation of this. This is the fastest growing actually product for us.

53:28 Uh so that's what apps look like and we actually now introduced a marketplace where you can actually get these apps. Uh so anyone can go and access these apps. And one of the things that we're also excited about that we're announcing is that you can actually transact and pay for these apps so that you can buy them from those vendors that are actually offering those. Of course, you can have your own, your company can build your own. Uh but all of these that also we've listed here exist as apps that you can actually get access to uh or work with. So that's a very exciting development. Okay, so that's what that

Security Lakehouse and Lakewatch revolutionize enterprise threat detection

53:55 looked like. Um, but there's a few apps that we're really excited about and we want to build ourselves that we think that we uh uniquely uh have a point of view and where we can really add value where we think it's really data intensive and it's kind of our really forte. So the first one is security and we announced two months ago that we're launching a SIM. Uh it's called Lakewatch. There's an agentic sim uh we call it the security lakehouse. It looks kind of like this. Let me explain what this is. It's built on the agent system of record that I mentioned earlier. So you collect all of your security data in a lakehouse. We have to do it that way.

54:31 The current approach of taking data and injecting it into these proprietary SIMs uh is super expensive. So people can't afford and the data is exploding. So you don't want to actually ingest all these expensive data into these classic legacy SIMs. So a lot of people what they're doing is they're filtering out the data before to make sure that you know only the most important data is in there. Well, this is not going to work because we're getting now massive attacks from automated agents. In the last 6 months, the best hacker in the world is a Gentic system called Expo. You can look it up on the hacker one um top list. U people are finding exploits using things like Mythos and even before Mythos and actually after mythos, we're going to see lots of lots of exploits. Uh so we need all of the data. It needs to be very cheap. it can't be skyrocketing costs and we should just store it all in any means we have in an open secure lakehouse and then on top of that we've built agents in lake watch those agents what do they do they automatically make sure that they create detections they when you wake up and there's like

55:32 hundred alerts of what's going on with your data they go and triage it for you automatically very similar to zero ops but this is really for the sock sock is security operations center so the sock analysts uh wake up and can do that. Um, and then finally, it can do threat hunting for you. It can actually find out that there's a new zero day attack and it'll add the pipeline to it and automatically do all of that work. So, we're very excited about Lakewatch. Uh, so check that out. And we are very excited to announce a piece of news uh which is that databicks has agreed to acquire Panther Labs. So, yes.

56:11 So, Panther is a really amazing um ahead of its time sim that's based on Python. Okay. So, what Jack Nglier and team that we're very excited to welcome to Data Bricks and you'll actually see him on stage here tomorrow. Um they really bet on everything should be Pythonic. So, they were way ahead of their time. Actually, I remember meeting him five years ago and he came from Airbnb. He was a stock analyst himself and he said, you know, we can't have these complicated drag and drop UIs. it needs to be Python based which turns out to be really preient because today um all the AIs want to speak Python. So that's really awesome and it has hundreds of connectors that they've already built that we're all adding. Uh so now we have all these connectors and we have all these pyonic detections. Uh so very excited about this and they have really marquee customers. So Entropic is using them, Coinbase is using them, Plaid is using them. So very excited to welcome that to the family. Um so that's the Gentic Sim. Okay, there's one more space that we want to we're excited to get into which is the marketing stack. So,

57:13 um what is that? So, we're very excited to announce what we call customer lake. Okay. Yeah. So, what's Okay, so what's customer lake? So, it's an agentic CDP customer data platform or customer 360 if you will uh built on the lakehouse. Um it looks like this. And so of course you can get now all of your data into a lick house. Um but it's unique. So it has two agents that are really special. One agent which is a profile agent. What it can do is it can automatically do all of the identity ddupe that these uh frameworks in the past have done but it can do that uh using LLM. So it's really accurate. It also has a campaign agent which implements what we call a new concept called infinity campaigns. And infinity campaigns basically let you tell your personalize every interaction that you have. So in the past what people would do is they would say the whole world let's split it up into 100 different audiences and then let's market to each

58:14 of the hundred audiences. But now we can really personalize with really cheap small uh distilled LLMs onetoone uh continuous campaigns. So, we're very excited about customer lake as well and we've developed this uh in partnership uh with all the vendors that you see down here and you can see at the top uh the logos of our development partners uh that are helping us actually develop this. So, very excited about this. There'll be a talk by Taso on this as well. Okay, so that brings me to the end of my keynote. Uh we have an exciting program. This is what I kind of showed you. So, let's simplify this a little bit and zoom out. What have I told you today?

58:50 So I basically said that we have this data, we have all these processes. If we can just capture all of that and make it AI ready, we can get the context and we can give it to the AI um then we can have huge impact in the organization. So really it's about how do we get choice so that you're not locked in. How do we get governance? How do we get cost control and how do we get context? Okay, so the first one is really about any data, any model and any cloud without locking. The second one is really one place where you can do all of your governance and all of your cost controls. And the final one is about how we get enterprise context to your agents. So that concludes my talk. I hope I've shown you that uh data bricks data and AI platform will help you do that. Thank you so much.

59:44 Here we go.

1:00:25 I see everything. welcome to the stage databick senior

Genie 1 outperforms leading AI agents through enterprise grounding

1:00:51 director of product management Ken Wong.

1:01:01 All right, good morning. So today I get to introduce you to Genie 1. Genie 1 is an AI co-worker that can connect to all of your data and all of your apps so that everyone in your organization can get insights and even automate action. You can even create a autonomous agents just by talking to Genie. But I know what you're all thinking. In 2026, why do we need another one of these AI agents? Well, the answer is actually very simple. The ones that you have today aren't very good with enterprise data. They either can't connect and reason about data at all or they're just not accurate enough for you to actually trust them to make real decisions or let alone automating your work. So, let me show you what I mean. So, yesterday we had our product advisory board meeting at Data Bricks. Um, shout out to the PAB. And I wanted to prepare myself for it by getting Genie One to help me with

1:02:03 a pro comprehensive profile of all the customers we have, what's going on in their accounts. So, this required Genie1 to connect to all the systems that we have, including my my personal email and Slack and my calendar to figure out who's in the PB and what we've been talking to them about. uh required Genie1 to connect to Salesforce to pull in the use case information, but it also meant that Gen1 needed to connect to data bricks in order to pull all the consumption data that they had because I had a lot of nuance questions that I wanted to include in the brief about what they're using Genie for. And this is what Genie 1 was able to come up with. Um as you can see, it gave me a comprehensive profile. I had to blur some of it so you don't see the details.

1:02:45 Um but it after you know querying data bricks and querying these systems they able to give me a comprehensive and visual profile of all of our customers. So what I wanted to do was try this exact same prompt with five of the leading AI agents that were out there. I tried two of the AI assistants that came bundled with some of the software we had and three of the leading coding and co-work agents and this is what I saw. So, the first AI assistant is actually one that I used a lot, one that I used to write docs and things like that. And it it came back with a response to my prompt very very quickly. Actually, a little bit too quickly. And there was something a little bit off about it. And the first thing I noticed was that it said that we had 24 customers in the pab. And I knew this was not true. So, I asked it and say, "Hey, like where did you get this 24 number from?" And guess what it said? it confessed that it just completely made it up. I was stunned, right? Because this is actually I use this thing a lot. Uh not just for at

1:03:46 work, but I also have a personal subscription. Um but it just shows you goes to show you how important enterprise grounding actually is. Now, the other AI assistant that I use actually did a lot better. It actually searched existing documents and pulled out account briefs. Um but they were all sort of stale. you know they were all from a quarter ago information that was no longer relevant to the conversation I wanted to have. So the results of this thing completely unusable. The good news it was very very quick. Now the a the coding agents and the co-work agents actually made a genuine attempt and connected to all the all the systems and the structured source in order to pull live data out. But if you ever used one of these things, you've probably experienced what I experienced, which was that it went off and tried very very hard to pull this information out. It actually hit its time limit um where I said asked for permission to continue burning Lord knows how many tokens in the process. And then after I allowed it to to continue, it went away and it

1:04:47 iterated again and again and it came back with some cool ASKI art. Um but then confessed again that actually we were far from done. You know, we had only gone partway through. So we're 30 minutes into this thing. I had a partial report. It's like a huge contrast with what Genie1 was able to do. So you might say, hey, this is sort of a one-off test. Um you cherrypicked an example, but you know, it's actually very very consistent with the research that data bricks research had been doing on the

6 Highlight · Comparative benchmarks show standard agents fail on enterprise data

1:05:17 ability of coding agents to answer novel questions. Actually, what we saw, we cured a benchmark out of real questions that our employees were asking of Genie, and we asked coding agents to solve the same problems. Now, these weren't run-of-the-mill simple questions that could be e easily, you know, uh um solved with some semantic layer. These were like nuance questions like, hey, of all the p all the customers who use Genie, what percentage also use a third party agents? Questions that, you know, hadn't been asked before. And we saw that coding agents were able to do it, the best of them, about half the time.

1:05:50 And it took minutes. And remember this chart, we'll come back to it. But 50% of the time for a for a real data question is basically unusable, right? You cannot make you literally can flip a coin. Um, and if you know how these agents work, and I'll explain this a little bit already, you're it shouldn't be surprising because they basically go through this process of the agentic loop where where really getting to the right answer is a matter of it, you know, kind of making sure uh happening upon like the right sequence of reasoning steps to chance upon like the right context it needs to ask answer the question. So, it kind of goes through this walk where it it just does its best. And there is a

1:06:32 very real trade-off between the accuracy that you get and the cost and runtime of these things. And and this trade-off results in what we see. Now, our current best approach to solving this problem is to inject some context, right? Like to give it some information that gives it hints on the right thing. And if you come from the AI world, you might call these things skills. And if you come from the BI world, you might call these things semantic layers, but they're really the same idea. It's like, hey, we can just tell it the things that it need to do and then it can do a better job.

1:07:02 And the problem with this is that well, while it's totally reasonable and in fact best practices to do it for the most stable business concepts you have, it's unreasonable to do it for the breadth of things that you do within your organization. Think about how each and every single marketing campaign defines leads a little bit differently or how every single scrum team uses all the fields in Jira a little bit differently. You just can't realistically write all this stuff down.

1:07:29 And before you say, hey, can't we just use AI to solve this problem by generating these models? Well, it actually just doesn't work. Actually, Anthropic published a great blog on the limits of this talking about that the fact that if you do this, what you end up doing is just encoding or like anchoring the AI on one specific use

Genie Ontology Uses OntoRank to Automatically Inject Learned Context

1:07:48 case and then that actually negatively result affects the results overall. So what is our solution to this problem? Well, I mean Ali stole my thunder a little bit. It is the genie ontology, right? The genieontology is an automatic context layer that adds a learned layer on top of any modeled context that you have, whether that's in Unity or in one of your existing semantic model or uh modeling tools. Um it and it basically allows Genie and any agents attached to the ontology to do a much better job of accurately answering questions with

7 Highlight · How Genie Ontology Extracts Context with OntoRank

1:08:25 data. Let me tell you how it works. So first with the genie ontology what we do is connect to the systems that you have that contain knowledge. So for example using the data bricks system you have a ton of information in your pipelines, queries, dashboards and things like that that tells an agent how they can reason about data. And we extract all of those things, things like expressions and relationships, but also expertise like the the idea that this person is really knowledgeable about a specific topic.

1:08:56 And we extract all of this into our internal knowledge store. And then we go over this knowledge with our an algorithm which you know this is how you know the engineers still run a show at data bricks. We call onto rank. And it's a page rank al like algorithm that helps us determine which snippets are actually authoritative. And then with with this information when a when a question comes into Genie we're able to look up this information apply permissions because we also extracted out the permissions of the underlying source access so that we don't have to you don't have to worry about leaking permissions and then we just inject that context into the agent loop so that Genie is able to answer much better uh questions much more

1:09:40 accurately and the results of this is so that if if I ask a question like how many people are registered um without ever having to model it. Genie knows about the nuances of how um how we captured registrations in our internal systems because it was able to extract that information out of some source asset. And if you want to know where it came from, you can just inspect it and see actually learned about this these calculations from our you know underlying dashboards. Uh marketing team's a little bit mad at me about this because I took this video from two weeks ago. Um but the numbers are a lot higher now. So coming back to this chart. So we saw what happened uh with generic coding agents. How does Genie with the ontology do? Well, in our internal testing, what we've seen is a consistent 30 percentage point plus improvement in in accuracy and roughly half the the runtime of the leading agents, right? And we're really confident that as we continue to tune

1:10:41 the onto rank algorithm and expand the the the breath of the ontology that we're able to push this accuracy much much higher. So with this level of accuracy enabled by the Genieon ontology now we can tr truly create a data smart co-worker one that you can give to all of your employees to actually make decisions with data and even automate decision making. Right? It's the it's the foundation. So, I'd like to welcome now Elise Joris onto the stage to show

Genie 1 Computes Live Operational Data to Build Executive Review Docs

1:11:12 you what you can do now that you have this foundation and what you can do with Genie1.

1:11:26 Hello everyone. My name is Elise. I'm a product manager here at Data Bricks for Genie and I use Genie in a lot of my day-to-day work. We actually have an exec review for our Genie OKRs that's coming up and I'm going to use Genie to prepare for it. I'm starting here in Genie 1 and as you can see it's a pretty simple interface. I have my chat here in the middle. We have some buttons that we can use to draft a doc or create a skill. On the left side, I have my prior chats as well as all the agents and the assets that are available to me. But more on that in just a second. I'm going to start by asking Jeie to create a document for my review. Every team at datab bricks uses the same template.

1:12:14 It's got metrics, highlights, low lightss. So I've given Genie a link to that. I've also asked to poke the people who have open Jira tickets so that we can keep those moving. Now, as Genie is running, we can actually click into what it's thinking. So as you can see here, it clearly found something interesting in the Unity catalog glossery. seems like it found some potentially relevant ontology snippets. And this is actually all from data that I do already have access to. Genie isn't forcing you to create a separate permission system.

1:12:45 It's enforcing the permissions that you already have through Unity catalog. Looks like it also grabbed some Google files and found those Jira tickets. It used MCP to comment on those tickets just like I asked. And now it's actually running some SQL across both data bricks and BigQuery to get the latest ground truth. And that's actually really important. A lot of generic agents are just reciting what already exists in a document. But there's no document for this week's metrics. That's why I'm doing this. Genie understands the numbers and is actually able to compute the results based on our live operational data. And to that end, it looks like I already have a doc for my review. Looks like Genie pulled the metrics and broke them down by region.

1:13:33 Gave me a bunch of nice visualizations. Grabbed the highlights, the low lightss, anomalies, and notice that it actually did that fairly quickly rather than just grinding through an agentic loop of trial and error and eating up a lot of tokens along the way. Genie just went straight to the ground truth. And that is really the genie ontology at work. So, how do we know that we can actually trust this? Well, for that, we're going to pop down here into the citations.

1:14:01 This is everything that Genie actually used to answer my question. And again, we have tables, foreign cataloges, tickets, docs. Looks like it did actually use the definition from the glossery. You can see here that it's verified, but I actually want to click into the ontology. I want to make sure that GI inferred everything correctly. So, what you're seeing here are ontology snippets. These are again the facts about our business that Genie has learned and evaluated automatically.

1:14:29 It looks like there's a snippet for how we define engagement, a snippet for what qualifies as a blocker in Jira, and one for the SQL that we can use to pull engagement by region. I'm going to click into this first snippet and see here we see the uh full definition. We're looking at sessions that lasted longer than 30 seconds. And it looks like Genie pulled this from our mobile KPIs dashboard which was authored by our colleague Chung. It's also got a high authority score. This is a function of things like how often the snippet is used or the asset that it was pulled from. It goes back to that onto rank algorithm that Ken was talking about.

1:15:09 And I can actually click into Chung's profile. And here I see that he's engaged with a lot of the domains our team cares about. And yeah, it actually looks like Genie has pulled quite a few snippets from the assets that he's created. Now, while I'm here, I might as well schedule this to run every week so I don't have to manage Jira myself. Great. And now I'm just going to take this one step further. I'm going to ask Genie to add a forecast to my doc. One of my teammates already has a forecasting skill, so I'm just going to use that. And she didn't do anything fancy. She just used the built-in forecast function and added a few instructions for how we like to review these forecasts.

1:15:48 And again, this is really only possible because Genie is able to compute these results based on our live operational data. And it looks like it added my forecast and it also gave me a pretty nice visualization here. So now I'm going to ship this. I'm going to send it over email and Slack. Now this OKR review isn't a one-time thing. People are asking me for updates on the mobile app literally all the time. So, I'm going to create an agent that can answer questions using context from this conversation.

1:16:22 I'm going to name my agent. Great. Now I have an agent. This is an expert co-worker that can answer questions and take action related to the Genie mobile app. And unlike an OKR doc, which is probably going to get stale pretty quickly, this agent stays up to date. And again, it can compute results on the fly. So, while I'm here, might as well share this with the other PMs at data bricks. And now I'm going to click into this agent.

1:16:57 So, as you can see, I'm actually chatting directly with my agent. I'm not going to have to specify what product we're talking about because it's already baked into the agent. It's customized to the mobile app. And you can also see that in these agent details that Genie has automatically generated. And more broadly, you can probably imagine all the different teams that might use something like this. Think about any function that has to answer a bunch of questions. HR, IT, finance. Now everyone in the organization can create and collaborate on these types of expert co-workers in one place. Now from here I can go in and manage my agent. You know tighten the scope, sharpen the instructions, add some context or I can certify it. And this is going to appear on our organization's agent list page as a certified agent. And the geneontology is also going to prioritize it. Now, I don't do all of my work on the desktop.

1:17:55 I actually have to do a lot of work on the go, which is why I have the Genie mobile app. And if I open my app or my phone that is, it looks like I just got a notification. Uh, engagement with the app is spiking. It's all very meta when you have to work on Genie with Genie. And I can click into the app to learn more or maybe ask a follow-up question.

1:18:20 And to be clear, I didn't actually have to set up this insight. Genie knows that I care about engagement with the app because of my activity, and it can actually proactively notify me because something interesting happened. And I think that's actually really powerful. Genie is no longer just a reactive Q&A tool. It's now a proactive AI co-worker that can bring me things that matter before I even know to ask. So, in just a couple of minutes, we moved the project along, prepped for an exec review, got some colleagues questions answered, and even learned something that we didn't know to ask about. That is a super high quality AI co-worker, and it's really only possible because Genie understands our business through the ontology. It can compute results rather than just reciting documents, and it can enforce our organization's governance along the way. So, we'd encourage you to try this on your own data and share your feedback. And with that, Ken is going to come back out. Thank you all.

1:19:24 All right. What do you think? So, Genie1 is a data smart AI co-worker. It has full co-work capabilities. You can create documents and again compute them live. not just summarize and recite existing things that you have, but net new analysis. You can connect them it to all of your systems thanks to the connectors that we have. You can schedule it to work with your background in the background. And you can schedule it without having to keep your laptop open. Um, and you can integrate it to all of your systems, even custom ones because we have custom MCP tools, all managed through uh Unity AI gateway as well. And you can develop custom skills that you can share and create with your co-workers. Even create a dedicated domainspecific agent just by talking to Genie. So you can improve the performance of that focused agent over time.

1:20:18 And Genie again works on all of your data thanks to the large and growing ecosystem of connectors that we have through federation and ingestion and customer and it's all managed by Unity. And we'll have a deep dive on all the governance features of the Unity AI gateway tomorrow. And what this means is that customers like Warner Music Group is able to use Genie for both data bricks and non-data bicks uh uh uh data and and customers like General Motors is able to deploy hundreds of Genie agents into production to improve the performance in every single one of their teams. And it allows customers like Foot Locker to be able to think about how to transform their business to a world where every employee is armed with the ability to make decisions with data. We really want every single user to be able to leverage data to do their jobs, which is why we're striving to make Genie available everywhere. So, I'm pleased to announce that Genie is integrated now into Teams and Slack available today and

1:21:21 it'll be available on your mobile devices on both both major platforms on Android iOS again today. So, as Alli mentioned, this has already become the most popular way for data bricks employees to experience Genie and I'm pretty sure it's going to be that way for your for your C uh for your employees as well. Now, for those of you who are using one of these existing AI agents or have developed your own and would like to benefit from the super act superior accuracy and data smarts of Genie, we're also pleased to announce the availability of the Genie MCP app and that gives you your your existing agents or existing tools the ability to tap into Genie's a ability to compute results accurately and all and and that is also available today.

1:22:10 So we really believe that Genie1 is unique with the ontology. It's the only AI agent that allows everyone in your organization to actually consistently make results and not just summarize summarize information from existing documents. And we want to make sure it's available to to absolutely everybody. So to in order to kind of encourage that to happen, we're giving $10 worth of tokens um for every single user inside your organization every single month. So there's really no reason not to get started with with Genie Genie 1 and we're going to make Genie One generally available today.

1:22:54 Right? So please connect Genie1 to every

PepsiCo Transforms Massive Supply Chain Operations Using Genie

1:22:56 everything inside your organization and give it to everybody. Well, next I'm excited to introduce to you PepsiCo, who's going to tell you a little bit about the work that we've been doing together. One two three.

1:23:42 Welcome to the stage PepsiCo's global chief data and AI officer Magesh Bagavathi and Datab Bricks co-founder Arcelan So, oh my gosh, we saw a pretty cool demo out there talking about Genie, but here we're about to talk about all the amazing things that Pepsi's been doing.

1:24:18 Um, I'm sure everybody out here is familiar. they just saw the video, but you guys are pretty massive and complex uh business, but best-in-class when it comes to AI transformation. So, how about you tell me a little bit about that?

1:24:30 Well, thank you, Arlon. It's fantastic to be here. Great crowd, very talented crowd, and I'm here to represent team PepsiCo. And uh you know, this journey started off for us around six years back as we were really initiating our full-blown end-to-end digital transformation and process transformation as well. And the focus has always been faster for our customers and consumers, stronger for our people, products, and our businesses, and ensuring we're better for the planet.

1:24:58 So, that's been the core mission of PepsiCo. And as we talk about the size and scale of PepsiCo, sometimes it amazes myself, right? Um, we have 320,000 employees globally, operate around 200 countries, and this is continuous non-stop operation. We have I don't know some of you might not know we have one of the largest fleets in North America. So when you have this level of complexity you also want to most importantly be able to service your consumers and one of the things we measure how we service the consumers is occasions.

1:25:33 We do around 1.4 billion occasions a day and occasion is you saw all the great products in the video. It is really how you all consume our fantastic products every day. So that's what we call as an occasion. So when you have to service these occasions through six million retailers and also your EB2B and E and and D2C avenues, it's a pretty complex endeavor. So for us, what we truly believe is an AI transformation is a data transformation and that's what we've undertaken. So what the team has done really is fundamentally over the last six years really moved us from a legacy architecture where we had 60 plus data lakes to one lakehouse right now running on data bricks and also from uh a monolithic architecture to a truly pluggable Lego based architecture. That transformation for us is really how we start unlocking the value of the data that's in this platform because now 90% of PepsiCo's universe data universe is

1:26:34 now on the platform and this which is what we call as enterprise data foundation. So for us this is a living breathing platform that we double down on.

1:26:42 That's awesome. And look you were one of the early ones uh that talked to us about Genie. I know you had efforts internally and you basically came to Ali and I you're like we have to build something like this. Uh why has Genie been such a game changer for you guys? Uh yes we did come to you guys a few years back and this was when we were actually as we started onboarding all this data into the platform we saw that there's going to be so much richness this platform can offer for us in terms of the core capabilities we call this AI forbi and our ability to dialogue with data

1:27:13 whether it's a warehouse operator whether it's a frontline employee or whether it's a knowledge worker we want to be able to dialogue with data just as Ken showed and for us as we were looking at this AI for BI consoles it was taking us anywhere between three to six months as you want to vectorize it you want to rag do do rag patterns and try to bring the data on board and really then start executing this as we having conversations we were starting to see you know the accuracy was not where we wanted it to be so that's why it was taking three to six months to productionize and this is when we reached out to to you guys so at this point in time having almost 95% uh accurate and and catalog data uh in the enterprise data foundation for us, Genie is all about unlocking the value from the data bricks environment.

1:27:59 So that's kind of an amazing story. Now bring that home for me. Make it real, right? You guys actually have actual use cases of how you've been using it. And even though you're getting started, talk to me about some of the value and how you've been using it.

1:28:11 Yeah, it's not real until you productionize it, right?

1:28:14 And um what we have in PepsiCo as an ambition is we want to identify PepsiCo. And as we think about identifying PepsiCo, we think about concepts such as the supply chain AI brain. One of the components of the supply chain AI brain is the procurement brain. Very strong uh sponsorship from our chief procurement officer to really start looking at insights for all things procurement, both direct and indirect spend. We're a $95 billion revenue company. So there is a decent amount of direct and indirect spend. So um within this platform called spend wise where our procurement leaders and our key senior leaders operate to understand what their overall indirect spend is we've had dashboards capabilities and reports. But as we unleashed Genie within that that space it was truly a big gamecher. Just in the first few weeks of launch, we almost had 30,000 queries, engagements back and forth and all of a sudden you start seeing this user population shift from reports and dashboards into a genie type

1:29:16 engagement. One of the things I love about Genie, by the way, is a little bit of what Ken showed was that ability for you to go into agent mode and really then start looking at deep research. So you have tremendous amount of flexibility within the platform, which is what me and a lot of the leaders within PepsiCo really like. And now what we're looking to do is we're now looking to expand this use case across all parts of PepsiCo, including commercial reporting for Europe and also for Latin America as well.

1:29:44 Fair enough. And then I know they're going to yell at us because we're running late, but I'd be remiss to not ask you, you know, what's next for you guys? You guys have such ambitions of what you want to drive. So what's next? You know, for us it's all about how do we get to an agentic PepsiCo and we've got huge uh ambitions. For us it's really about going from insights to action to outcomes for our end user and for our population. So our goal is now to go from these 30,000 plus reports down to 50 plus AI for BI or Genie consoles. That's a big outcome for us.

1:30:18 That's how we're going to drive our our business going forward. This will really help us agentify PepsiCo. This will help us uh identify our 320,000 associates.

1:30:27 Awesome. Well, it's been an incredible journey and excited to see what comes next. Thanks so much,

1:30:31 Ron. Thanks for the partnership.

1:30:32 Appreciate it. Thanks.

1:30:33 Thank you all. Welcome to the stage databri senior

LakeFlow Simplifies Messy Data Engineering Into Open Declarative Pipelines

1:30:50 director of product management Ballam.

1:30:58 Good morning. Good morning.

1:31:02 I'm going to talk to you about leg flow. But just a little quick note first. I wore this I wore this purple jacket two years ago. Talked about leg flow. I did not wear it last year and the biggest piece of feedback was bring it back. So here it is. Uh my name is Bilal Alam. I'm a senior director of product management at data bricks. Super excited to talk to you about LakeFlow. Now uh first of all thank you to Magesh and Arlelan and Ali and Ken and everybody who's come here and talked about AI and agents. Now uh super exciting future. Uh but if you're anything like me, if you're a data engineer, uh you're probably sitting there thinking this is great, but all of this needs data. uh all these apps, agents, AI, everything needs data. And you might be thinking, my job got a little bit harder. Uh and that's because uh with data engineering, the destination is simple. The destination is get to the right, get to the business outcomes we want. We want agents, applications, we want real-time operations. But we're actually starting from the left. We're starting from raw data sources. We have data in Kafka, Salesforce, Netswuite. We have on-remise databases like Oracle and SQL Server.

1:32:03 And the messy middle, that's data engineering. That's our job to take that raw data, turn it into insights. And over the years, we have built architectures like these. I put this up here because it's representative. You don't have to look at every box here. It's a lot of logos. It's a lot of boxes. It's a lot of lines between boxes. And that's just to get data from one place to the other. And you know, we've made peace with complexity. Sure, there's a lot of tools, but you're probably sitting in here thinking, "Okay, at least it works." uh even though you know instinctively that this complicated not unified architecture is missing a bunch of critical things we want to version control everything as data engineers but in this stack you can version control some things not the others uh you can version version version control your spark code but not your Kafka infrastructure you want governance everywhere but it's pretty much impossible you're reading and writing to too too many different places and there are parts of the stack as we will see soon that are just locked up and finally you want everything to scale and you want to save money uh so you can build more applications and outcomes for your business. But not all parts of the

1:33:04 stack scale equally well. I'm here to tell you that your job and my job as a data engineer is about to get quite a bit harder. What you're seeing here is Genieode. It's a purpose-built coding agent for data science, machine learning, and data engineering. Uh if you're using it, that's awesome because uh 60% of LakeFlow pipelines, this is a crazy statistic, are already written by Genie Code in just three months. We just released this. It's pretty awesome. On the other hand, this statistic scares me a little bit as a data engineer. And that's because I have a crush on agents.

1:33:39 Agents are great, but all agents mean new data, new pipelines, more to build, more to manage on that fragile infrastructure. We're going to get swamped. So, what do we do? We start by removing the moving pieces. And actually, it's super important. We don't just want to remove complexity with more complexity. We want to replace that with open formats. open open frameworks, open tools, we ultimately need to simplify the data stack.

1:34:05 I'm here to make a pretty bold claim and my bold claim is that LakeFlow is that unified stack that it gives you that open foundation for AI and agents so that you can build the future that your company deserves. And now we work with hundreds of partners, lots of tools. They're all interoperable with LakeFlow and the lakehouse. You can use them as well. So let's get started. I'm going to do a little bit of a speedrun through this. Uh how do you simplify data transformation? ETL. This is where you're spending most of your time and money. Uh you have Spark, you have lots of legacy Spark jobs. You may have a Spark distribution on EMR or some other vendor. I'm going to replace that with Apache Spark declarative pipelines. If you're not using them, please try them out on LakeFlow. They're open. You can run them anywhere, including on your laptop. And they're declarative, so you can just focus on the work to be done.

1:34:52 And because they're open, agents are really, really good at writing these pipelines. They unify batch and streaming. Many years ago, DBT was the only game in town when it came to SQL. I like writing ETL in SQL, but Spark declarative pipelines unifies that as well. It has SQL and Python. You don't have to choose. And finally, if you're using Flink for real-time streaming, that's probably because actually low latency streaming was really hard on data bricks and spark. But now, last year we open sourced real-time mode, which is a brand new engine for low latency stream streaming. And I'm super excited to share that it's now available inside Spark declarative pipelines. So you can get millisecond streaming on an open framework that runs anywhere.

1:35:30 Awesome. So we're taking care of this complexity. We're getting rid of this.

LakeFlow Designer Allows Analysts to Build Declarative Pipelines Visually

1:35:33 We're building an open foundation for agents and AI. Well, what about all the shadow IT that's happening? So, you see, you may have 20 data engineers in your company, but you have hundreds of analysts who are building pipelines in drag and drop tools. These pipelines are built on their laptops, proprietary formats, proprietary tools, and if somebody leaves the team, you're out of luck. Super excited to share that LakeFlow Designer, the noode data prep tool powered by Genie, is now available.

1:36:02 It's generally available. You can try it today. You can try it right now. Thank you. Awesome. And by the way, one super cool thing is that LakeFlow Designer doesn't build any proprietary pipelines. It just builds Spark declarative pipelines under the hood. Again, you can just run them anywhere. I think you're starting to see a pattern. All right, we're going to go a little faster. I'm going to go ahead and simplify data ingestion. If you're using a SAS that has hundreds of connectors, that's great. But it's probably landing data in proprietary formats. Your agents don't know how to operate this SAS. You cannot automate this. You can barely monitor it or observe it or govern it. As Ali mentioned, I'm super excited to share that LakeFlow Connect now has more than 100 connectors. If there's a connector you're thinking of, we've either built it, we're building it, or we're going to build it. And there's a new community that's building open source connectors.

1:36:54 You can even build your own. And guess what? I think you're going to guess what I'm going to say next. This isn't a special type of pipeline or ETL. It's just Spark declarative pipelines under the hood. That was pretty awesome. All

Zerobus Ingest Provides Kafka-Compatible Streaming Directly Into the Lakehouse

1:37:05 right, let's talk about Kafka. As data engineers, we use Kafka day in and day out for high volume telemetry. We're bringing that in. Kafka is a buffer, but we also know Kafka is a real pain to manage. Okay, super excited to share that zerobus injust is a fully managed service that is now 100% wire compatible with Kafka. So you can take your coffee producing code, point it at zerobus ingest and you can land data into the lakehouse in open formats ready for AI.

1:37:34 So no small files, millions of small files at 12 GB a second. So it's pretty awesome. So you can get rid of Kaka for your stack. Great. Uh okay, so we're almost there and you're going to see we're still building everything on an open ETL framework that's declarative, but you see you see this little uh Airflow icon hanging out there. That's your orchestrator. You have to use that to trigger things, to schedule things. Let's simplify that as well. So, you're probably using Airflow because you like writing workflows in Python. Well, LakeFlow jobs supports writing workflows in Python. Now, it's just pure Python.

1:38:08 You can also build point-and-click DAGs. It's fully serverless, so you don't have to manage any infrastructure. And one interesting fact is that we did a survey of our customers and 80% of our customers are using old Airflow distributions. So, they're managing them, maintaining them. they have security vulnerabilities and they're not even getting all the new features. Okay. Uh and I'm the last thing I'm really excited about because for the longest time, let's face it with jobs it was really good at orchestrating data bricks, not so great at orchestrating other systems. So super excited. This probably one of my favorites in this conference is we're releasing 50 plus integrations so you can orchestrate everything. You can think of these as Airflow operators. They're open source and you can orchestrate everything including Snowflake. Okay. So where does this bring us? This brings us to a an open stack. Everything is pretty much building open source declarative pipelines.

1:38:58 And this is not a brand new product. We've been at this for a while and it's ready for the world's toughest data workloads. I want to share some statistics with you. Spark declarative pipelines now process 200 trillion rows of data every single day. LakeFlow jobs is running the world's biggest data workloads. It's about 1.7 billion job runs per month. And finally, serverless compute is really popular. You don't want to manage clusters. You don't want to set up VPCs. 50% of our customers have now opted in to use serverless compute. That's pretty awesome.

1:39:33 Great. So, we've I hopefully I have

Data Engineering Operations Fail Silently Because Code and Data Intertwine

1:39:35 convinced you to some degree that we made it easier to build. And now you might be thinking, okay, we have an open stack. I can start building. AI can write the code. Maybe you're going to go try out Genie and it's going to build something for you. But what about operations? I haven't really solved that for you. Actually, I've kind of made the problem worse, right? Because now you're going to say, "Great. Now more people can build more pipelines, more outages, more problems." And here's the reality. Simplifying pipeline creation is actually the easy part, although this was pretty tough, right? It's data operations that are really, really hard. Here's a survey that uh is by one of our partners and it shows that data engineering teams spend more than 50% of their time on maintenance. So that's where your time is going. It's only going to get worse.

1:40:21 And even with this time spent on maintenance, you still see up to 60 hours of downtime every month. So this sucks. Now you might be thinking, uh, but Bal, you just told me that this is a unified

8 Highlight · Why Data Engineering Operations Differ From Software Engineering

1:40:32 stack. Surely it has APIs and endpoints and tables. Maybe a coding agent can just fix it. And unfortunately, not really. Let me walk you very quickly through why. And the big idea here is that data engineering is not software engineering. It's something a bit more. In software engineering, your code is self-describing. You It's just literally described itself. And in data engineering, it's data and code. In software engineering, your tests either work or they don't. They're fully deterministic. And data engineering, you have code and data. And you can't just test data. It's statistical.

1:41:06 And finally, in software engineering, and this is the big one, failures are loud. When something goes wrong, you're going to see a trace. You're going to see an exception, and then you can roll back your deployment. In data engineering, failures are silent and permanent. You're probably going to get an angry phone call from a CEO or a stakeholder saying, "Hey, how come this data is bad maybe a week later?" Uh, so why do coding agents trip up here? Well,

1:41:29 that's because when it comes to detection, they're just missing data. They're missing telemetry. And you can give them spark logs for example but these are multimemegabyte traces you have to be very careful to manage the infra the the context window there similarly when it comes to assessing the root cause they're missing lineage now you have to export your lineage out of the system and to some degree you can do this coding agents are getting better and better and context windows are getting better and then you have to remediate they have to write the code fix but it's actually the final step which I'm going to be a little pedantic about and call it verify you can't with data you can't just run your unit tests, you actually have to take your code fix and you have to run it on production data. Okay. And if this is giving you shivers, that's exactly right. It really should. So, what does your agent need?

1:42:16 Your operations agent needs to combine code and data. That's tools and skills. To some degree, it can do it. To figure out the root cause, it needs read access to production data. Uh who's comfortable with that? It's an agent you didn't write and you don't trust. I I don't see a single hand raised here, but actually your coding agent also needs right access to production data. That's the only way it can verify the fix. Like who's here wants to give a random agent right access to their production data?

1:42:48 It's got to be one person, okay? There's no one who wants to do that, right? Because it can cause chaos. It can drop everything. And this is the big insight. So we've been looking at this problem. It's a tough problem. And the big realization we came to was that your operations agent actually needs to live in the data plane. It cannot live outside of the data plane. And if you think about it, that makes sense because the data plane has the data. It has the lineage. It's also the right governance boundary. So I'm super super excited to announce

Genie Zero Ops automates end-to-end incident management

1:43:21 Genie Zero Ops, a new background genie that puts your data and AI operations on autopilot. Great. I'm going to give you a demo, but let me quickly walk you through how it works. So, for detection, Geniezer Ops autonomously builds per table machine learning models and it continuously fine-tunes them and it has native access to metrics, events, and logs. So, you don't have to build this plumbing yourself. We already have tens of thousands of these machine learning models in production with Geniezero Ops to assess. It does graph ranking on data lineage in Unity catalog. I'll show you how it walks the lineage forward and backwards and then to figure out the root cause and this is where you spend a lot of your time. The root cause is basically what went wrong. It actually has a supervisor agent and a fleet of sub agents who do research and then come to a consensus on what is the likeliest cause for remediation. Genie zero ops works with Genieode. It cooperates. If

9 Highlight · Safe Verification Using Production Data Shallows

1:44:16 Genieode has access to your code, to your ticketing system, to your version control, it uses all that. So it can also update tickets for you as it goes through the life cycle. And finally, verification is the toughest step to verify. This is the 10 years of building an open lakehouse. It builds shallow clones of your production data. These are super cheap, super fast branches similar to what we do in Lakebase. And then it utilizes native network and code isolation and data bricks. All right. So I'm going to show you a quick demo of what Geniezero Ops actually does.

1:44:48 Great. So let's see. All right. So

1:44:53 you're going to notice a couple of things here. First of all, this does not look anything like a dashboard. We're not flooding you with metrics, events, alerts, light, uh, you know, red lights and all that. What you see is something that looks and feels a lot like an email inbox. It's actually we modeled it after a prioritized email inbox. So, for example, here, one cool thing is that you get to see severity. So, every incident is ranked by severity. So, you can only you have you can just focus on what really matters. And also, this isn't just a list of jobs and pipelines and failures and things like that. Over here, for example, here's an alert and Geniezero Ops figured out that, hey, 16 jobs are failing. They're actually the same incident. So, natural grouping saves you time. All right. So, I'm going to go ahead and look at an actual incident. And what's happening here is that there's an upstream table called top fan voters, and something's going wrong with it. About 10 minutes or so ago, uh the row count dropped quite a bit. And what genius your ops does is by the time you log in your background

1:45:54 agent has actually done all the thinking for you. It has done all the investigation for you. And as a data engineer I can start looking at things like what's the impact. Now when it comes to impact not all tables are important not all pipelines are important. So it walks the lineage forward. But in this case it figures out that hey top fan voters is a pretty important table. Uh downstream tables depend on it and this fan fan engagement dashboard depends on it. So it marks it as critical. Then it deploys the root cause agents to figure out what's going on. And what these agents do is this is they're saving you hours of time. They start from this table and they fan back using lineage and they go investigate potential causes upstream. So over here this is a fairly simple uh DAG for example. You know uh it could be any one of these tables but really it finds this one table this fan interactions table.

1:46:43 That's the one that's problematic. Okay. Then it went ahead and autonomously wrote a fix for you. But this fix is not deployed to production. You're in control. What it did do, and I want to share this, this is super cool, is that it went and created a shallow clone. It's like the lakebased branch of your production data. By the way, you have to give it permission. It won't do it automatically. You have to tell it you can do it with this pipeline. And this could be pabytes of data. It's a shallow clone. It deployed the fix and it actually verified that the right number of rows are returned. This is awesome.

1:47:14 This is hours if not days worth of work. And now at this point I'm just going to go ahead and create a pull request and I can follow my uh software development life cycle. Uh let me show you one other super cool thing that Geniezero Ops can do. So one other thing Geniezero Ops does is if you give it permission, it will autonomously scan tables for things like PII. So in this case our application is unwittingly exposing the PII of about a thousand users. Okay, that's pretty bad. And Geniez Ops creates a report for me. It finds out what the PII, the personally identifiable information is. It gives me a per table breakdown. It looks at lineage to figure out this is important.

1:47:54 Yeah, this is pretty bad. And then in this case, it comes up with a proposed fix. That's not a code fix. It's a uh it's a Unity catalog policy fix. And I can deploy it. Okay, so that's Genie Zero Ops. We're super excited about it. And for data engineers like me and hopefully for you, it means far less time building, far less time maintaining and much more time building on a unified and simple data stack. Thank you very much.

1:48:26 Welcome back to the stage. All right, super cool. I'm very excited

Unifying the known data realm against agent complexity

1:48:41 about zero ops. It's going to make our life so much easier. Okay, so now we're getting to really exciting part of our uh show where we're going to actually talk about a lot of the announcements that I mentioned. We're going to see Reynold introduce a lot of those things that I mentioned that we uh have innovated on in the last year. So, but before I do that, I want to put it all in perspective because it can get kind of confusing uh in the data world. There's so many different technologies, there's so many moving pieces. So we wanted to simplify that. So we call this um what you see here on this slide the known data realm. Okay, it's kind of like the game of thrones, okay, of data infrastructure. Okay, this is like the troubled waters. So I'm going to explain it. What does this look like? Okay, so on the west of this planet, we have the world of OLTP databases. Okay, so this realm of old TP databases was invented in the 80s and what it was is people needed a database behind every piece of software. So this software for instance you might have an ATM you're going to go withdraw money from it that needs to hit the database it needs to do that really fast we want that latency to be milliseconds we want it to never go down

1:49:42 and we want it to be super reliable it can't be that it ever messes up the transactions okay and over the time we've had these niche databases that also have appeared we have key value stores we have now vector search with AI uh so that's this island of OLTP okay on the right hand side of this map. What we have is the world of data warehousing. It appeared shortly after the 80s because people wanted to ask you know how many how much data volume went through this ATM here or how much money has actually transacted on all the ATMs.

1:50:16 Asking that question uh on those planes of data warehousing or uh OTP on the left uh would be too complicated and it would take down those databases and it would affect the ATMs and we never want that. So we want that to be up and running. You want it to be isolated. Okay. So on the right hand side we started doing data warehousing and now you could ask these more analytical questions. Okay. All right. But very soon in the last 15 years people wanted to ask much more advanced questions. Not just what's the total transaction volume on the ATMs but how much is Ali going to withdraw today? Can you predict the future? So we started doing data science and more and more uh advanced statistical analysis. Okay. And then last few years, we've now had real-time analytics appear. And real-time analytics, that's, you know, agents or applications or dashboards that want really low latency. Um, they can't quite answer complicated questions and doing big joins of data and saying, you know, let's do this analysis of all this data, but they can give you dashboards that give you some analytical queries. So,

1:51:17 for each of these, we needed different things. And we needed to move data from the left hand side from the OLTP databases all the way to the right. Okay, so for that that's where data engineering came in and that's where we had all the complexity of moving data and shuffling it from the OTP databases all the way to the OLAP engines on the right hand side. Okay, so this is what the world looked like. All right, you just heard Bilal. So this world of data engineering that's what he was focused on how you actually get your data from all these systems ready uh so that you can start doing operations on it. So we just cleaned this up. Okay, these planes he just cleaned them up and added roads.

1:51:49 But now what we're seeing is that there's this onslaught of agents coming, right? We're having agents that requires us to ship way more data between these different islands. Okay? And we're going to get crushed by these storms that are going to come and we're going to lose ships. Okay? Our infrastructure is super brittle. We've already seen many of the very important infrastructure sites that we depend on go down in the last year. This is going to happen to every organization. It's almost like everybody just hired double the number of employees. Okay, because the agents are submitting queries, they want more data stuff, more data movement is happening. So, we need to simplify this. Okay, so how do we simplify this?

1:52:26 We kind of did that already uh when we started data bricks and what we did is that um we kind of merged data science and data engineering. Spark already could do big data movement and it allowed you to do machine learning with things like Spark ML. And five years ago, we announced Lakehouse, which lets us do in one engine data warehousing, data engineering, and data science. So, we've already kind of accomplished that. That's kind of great. Um, but we still have this real-time analytics uh island separate. And we need to make a separate copy of the data if we want to have really really low latency because the analytics stacks today, all the data warehouses that are there, they can't go down to a fraction of a second. No matter what you do, there's a hard trade-off. they can't do the complex queries and the fractional uh second queries that realtime analytics has. So we need separate engines for all of that. Okay. So for this I'm excited to announce our latest innovation. I'm going to let my co-founder Reynold Chin

Raiden engine breaks the analytical latency wall

1:53:24 do that. So please welcome him on stage. All right. Thank you Ali. Um I'm super excited coming on stage today to talk to you about this new effort and honestly it's probably the largest single largest innovation we have done since our introduction of lakehouse and as Ali already mentioned um with the lakehouse we were able to actually unify data science data engineering as well as data warehousing and uh our data warehousing business have taken off exponentially since its introduction five years ago it was recognized by the Garner and Forester both as a leader in data warehousing I think over 60% % of the Fortune 500 uses the lakehouse for warehousing. And I joke about we actually scan exobytes of workload depending on which time zone you are before you have breakfast. The um and over the course of the last few years we worked fairly hard to improve performance at every single data and I summit. I think me or Sean somebody else would come up on stage introduced to you

1:54:26 some massive performance per performance improvements we've done. But the reality is kind of the warehouses or the lake houses hit a wall at about a second. And what do I mean by hitting a wall? It doesn't mean you can never get a response from the lakehouse or warehouse in less than a second for a query. It just means if you have a very stringent workload that you want to have very tight SLAs is in the range of milliseconds or hundreds of milliseconds, it it would be very very difficult to accomplish that because you might actually get latency spikes. And as a result, I think for many organizations actually require um workloads that have very stringent else SLAs. They started separating setting up a separate serving stack. They would copy their data into the data warehouse or the lake house and copy a portion of it that's required for serving into a separate stack and this caused a lot of problems. It's actually data pipeline to maintain for every data set. um this serving stacks tend to perform super well on very simple queries but doesn't actually work well for the more general workloads. It's a governance nightmare.

1:55:27 You now have to worry about hey did I forget to actually secure my data set in this specific platform but I actually did in my underlying warehouse. So if the whole theme here is unification, what if you never had to move your data? What if you never have to copy it? And we asked ourselves that question a couple years ago and uh it turned out it's actually fairly fundamental. It's very very difficult to tackle this problem. Um as a matter of fact, Michael Stoner who's a touring award winner and some of you might know him as the original creator of Postgress wrote a whole academic paper about it called one sizefits all an idea whose time has come and gone. And in this paper um stonebreaker actually argue it's very very difficult to engineer a single database engine that's capable of running a wide variety of workloads. And ultimately it's because of the increase in complexity makes it very very difficult to actually engineer the system.

1:56:19 And the fundamental reason for that is if you look at how database engineering were done in the last four decades or even five decades um every database engineering team started working approximately this way. um data bricks included. we go targets of hey here's a new workload we might want to actually uh study and then understand and then actually build a system for um and the way to do it is let's study all the latest and the coolest academic papers about this specific type of workloads and some of the papers actually go back 40 50 years some of the papers are published in the last few years you try to study them understand hey in my might uh so apply in real life and you spend a lot of time implementing it and at some point you try to roll it out to uh test how well it works with real world workloads And inevitably this happens. So the new technique you implement or new it could be a new algorithm could be a new data structure. It works remarkably well for a subset of the queries and it works terribly for a subset of the queries. So the database ends up hey we're doing

1:57:20 better here and actually regressing. And many of the uh algorithm or data structures actually work really really well for very very low latency to guarantee maybe a millisecond response time tend to actually backfire when we have larger data sets and vice versa. So this is one of the biggest problems. Um so we two years so decided hey can can we break the status quo? How can we challenge it and do the best we can? And uh we actually went back to drawing

10 Highlight · Rebuilding database engines using workload ML models

1:57:46 board. So with amazing engineering team um we actually flipped the way how database engineering were done. instead of starting with ideas and techniques went back to the workloads. Um so luckily on data bricks we have collected over zetabytes of data were scanned traces from all this data um which in the range of quadrillion naturally traces I have to learn how to pronounce that word um and all coming from trillions of queries that were executed and based on all this um traces we're able to actually build a machine learning model um that give you very high fidelity estimation of what might happen in practice given a specific new technique. And based on this machine learning model, we can do two things.

1:58:26 The first is we can select the right algorithm to dispatch very quickly at runtime. But it's not an LM because LMS we have too high latency. We can actually pick the right algorithm to uh dispatch at runtime. But even more importantly, we can decide and predict what algorithms to even implement before it hits production because there are millions of algorithms and data structures out there. it would be virtually impossible to actually find on mall. Knowing what to implement is one of the biggest advantage one can get.

1:58:56 Right? So the resulting engine of that is Raiden which all knows a reference to

1:59:00 Mortal Kombat. The um so Raiden um and the resulting engine is actually pretty remarkable and you might have actually used some of the other serving stacks out there. um you have so and if you have you run into many of the issues you're talking about for example some of them can't run queries longer than 5

Raiden delivers massive concurrency and low latency

1:59:20 seconds almost all of them require you to copy the data into a very specific format that's very engine specific they can't run complex joints and all that but raen can actually do it all right and the way raider will show up is going to come in the form of a new SQL warehouse and um this is kind of abstract so let me give you maybe a quick demo for you to help understand what Raiden actually All right. Can I get a demo on stage?

1:59:46 Okay. So, here I have a very simple setup here. Honestly, just a very simple query of New York taxi data set query. You don't really need to understand what it does, but for those of you that actually know what the data set is, it's fairly small and this runs. I first of all, I ran this query in Lakehouse today. All right, it finished it. It's not terrible. It finished in about a second. All right. And on the right hand side I have the Raiden setup which we call break house RT. Let's try to run this.

2:00:16 Okay. I don't know if you can see this number. Uh let's try to run it one more time. N did we hardcode it 007? No. No we didn't. Um okay. But that's just one query. All right. What's the fun of running just one query? Um, here I want to show you a demo of what would happen in practice when you start hammering the system. And we built a simulator simulating real dashboard application workloads hitting against the RAN engine. And I'm going to show you start with just one active session here. So just imagine this is an actual dashboard running. We're just going to load this dashboard. It loads fairly quickly, right? It's probably what we expect. But what happens if you scale up to 10, 100, or why don't we try a thousand? So now we're gonna simulate 1,000 active agents hitting the same engine at the same time. Each of the agent, by the way, is generating more

2:01:16 than one query. The dashboard actually takes about eight queries for it to load. Um, and they all hit at exactly the same moment. Let's see what happens. Done. All right. So, we executed 8,000 queries at 6,000 queries per second with a tail latency of 37 milliseconds. This is incredible. None of the existing systems can do this.

2:01:42 That's a very simple demo. Okay. But when I told you, hey, none of the existing systems can do that. Let's put it into the perspective of existing systems. So here I want to show you a benchmark measuring latency versus throughput. Throughput here means queries per second right. So we have the y-axis which is the uh p90 90 percentile latency and the x-axis the number of queries are executing and we're running a very simple query t6 which is a canonical query of scanning a little bit of data running some aggregation and let's see what happens when we test the different systems out there. So um for a competitive vendor um using their latest warehouse we're able to actually push the warehouse um it started fairly low latency but as you start pushing up about 160 queries per second latency started climbing up and hitting a wall. So the same band that actually announced a fairly innovative offering last year called interactive warehouse um is so interactive they

2:02:43 can't actually read the same data with a general warehouse. The only way to actually have it work is to copy the data into a separate warehouse. Um, and the interactive warehouse does do much better than their latest generation general warehouse. You were able to push to like 300 queries per second before the latency starts spiking up. Um, and we also tested sort of a uh open- source u query engine that's known for being able to run very simple queries very very fast with the yellow color. Um, and it's actually running much better on this specific workload compared with the uh um Gen 2 and the uh interactive warehouse. It's able to push all the way to close to actually I think 15,000 queries per second. That's pretty good, right? But then it started crashing right after. It doesn't actually time out. It just crashes. So, how can Raiden do right? How does Raiden do on the same workload? Raiden could actually run all the way to 12,000 queries per second while keeping the tail latency still below a second.

2:03:51 So there's one other benchmark I want to show you um which is what if we run all of PPCH which is a standard data warehousing benchmark. Um as you can see on the screen when data set is small actually all systems run reasonably well. I mean it's like a few seconds running 22 queries. I mean some a little bit faster than others but they all run pretty well. But as you scale up for example to 100 gigabyte you started seeing the yellow systems uh runtime started spiking. And this is actually I'm trying to point out the very specific problem I talked about earlier which is a specialized serving engine doesn't actually work very well on the more complex joins which will show up in the full TPC queries. And of course uh when it was scaled up to actually bigger amount of data both systems actually started crashing. The interactive warehouse started crash uh can't even finish because it cannot run any queries longer than 5 seconds and the yellow system couldn't actually complete because started running out of memory and RAM was actually able to complete all queries at all scale factor.

2:04:49 You might have benchmark fatigue at this point. So many benchmarks. Um the who cares about synthetic benchmarks. Um so one of the thing that we're really excited about in the process of developing Raiden is the design partners we've been working with some of which I've shown their logo on the screen here. I want to share with you sort of two results from all these design partners. The first one is Amberverse. Emmeris is a global um leading data and AI platform for the energy sector and they tested Raiden with 11 of their most representative production queries. And what they found compared with the existing serving stack is that with Raiden the same extremely low latency queries remain subund milliseconds whereas the longer running queries can now actually dramatically shrink their runtime and across the board on average they're actually getting a 16 times speed up. So Paul from Amoris was super excited to see that they're able to now execute queries in tens of milliseconds but also shrink the runtime of the longest running queries by almost 100x

2:05:53 and meta also tested Raiden and they found very similar things they were able to have the typical queries come back in 10 seconds and the most important thing is you don't need a separate system on the side just for your serving so what Raiden does it can give us the best low latency possible at extremely high concurrency. Um it can actually run real work workloads really really well and it can also handle the complex EDL um EDW workloads super well and today very excited to announce the first product offering powered by the Raiden engine. It's called the lakehouse RT and Lakehouse RT is a new warehouse they're showing up starting with read only workloads. They will support millisecond um performance and massive concurrency. But the most important thing is directly on the lake governity catalog on the same delta or iceberg data.

2:06:48 So what can you do differently with lick house RT where RT stands for real time? Um challenge your conventional wisdom. If you have a separate serving stack, test it out and you might be able to collapse the two platforms into one and that will give you millisecond performance, massive concurrency directly against your data lake in open formats and beta starting today. Try to talk to your account team to get early access.

2:07:18 Um and there's also two talks at this conference starting soon today on uh Raiden and Lick House RT that you can learn more about. And that's it. And next I think I would like to invite Nikita on the stage to talk about Lakbase. Not quite. Um so Reald refuses to say what Raiden stands for. So I'm going to get up here and embarrass him a little bit. Okay. So Reynold runs all of the data platform stuff. All this innovation you see that's part of his team. So his team actually called the engine Raiden because it stands for Reynold's dream engine. Okay, it's true true story. Um you know a I think they wanted to embarrass him. Uh so that's why he doesn't want to call it that. But secondly we I was actually talking to one of the team members and he said look there's so many projects at data bricks and you guys always you know kill the projects that you know you think you don't want us to be spread out thin. So we thought what if we renamed this after a co-founder.

2:08:20 Okay. So uh that's the radon engine. So this is the map that we had of the data world. And then you saw that now we had real-time analytics on a separate island. Now we can shift these tectonic plates and merge that in. You get the same format of the data under the hood. You get the same Unity catalog governance and you can now do all of your low latency uh processing as well as the big data processing in this one island using the same technology stack under the hood. Uh so it comes at as part of lakehouse RT which stands for real time. Uh and we've actually priced it super competitive. So it's essentially the same price. So very excited about this. Okay. So now we're going to turn our um uh eyes to the left and we're going to look at this island

Lakebase brings serverless Postgres to the lake

2:09:00 of OLTP. And I'm very excited to welcome on stage uh Nikita Shamgunov who was CEO of Memsql later single store and then Neon which was acquired by data bricks where he leads all of this and he's been doing OLTP for a very very long time. So he'll tell us about Postgress. So let's welcome Nikita on stage.

2:09:27 Thanks to AI we now generate software. We're not writing it by hand. And we're looking at the trends for the first six months of year 26. And now we can confidently say we're going to have more software generated in the next year than in the history of humanity. And every application still needs a database. So what do next generation application and agents need from a database?

2:09:58 So the first thing is it needs to be familiar and the reason to that is the more agents know about the systems and they know by scanning the internet and scanning all the stack overflow forums and Reddit the better they are at running and operating it. So it needs to be open source. It needs to be popular and it needs to be extensible so you can consolidate all your main and also niche workloads on the single database platform.

2:10:26 It also needs to be nimble with this onslaught of applications that are coming at us and all the various dev test and staging environments. Uh the system needs to be serverless and it needs to be branchable so you can easily create environments for your agents in which they can safely operate. And finally as you run those things at scale and at volume uh it needs to be cost effective.

2:10:52 The final thing it needs to be mission critical. Operational databases run your business. And so um that means is it needs to be infinitely scalable. It needs to be fast and it needs to be extremely reliable. So familiar, nimble and mission critical. So we decided to start with Postgress the most advanced open-source database in the world. It has a largest uh ecosystem in the world and also lots and lots of extensions. Again, you can consolidate all your additional workloads to the main one to the operational database system one into the platform. It's also open source and understood by every agent on the planet.

2:11:40 But Postgress is a monolith and the compute and storage in the monolith are

11 Highlight · Decoupling Postgres storage and compute onto the lake

2:11:45 tightly coupled. So if you want to make it nimble um you need to do something. you need to rearchitect that system. The main idea we had is we can decouple storage and compute and move storage into the lake. But it's actually, you know, much harder than you might think. So for that, we need to rearchitect storage from the ground up. The lake storage is, you know, has a lot of very very good properties. It's inexpensive. It's very easy to scale but it's also slow and transactionally inconsistent.

2:12:21 So as we are rebuilding uh the storage for posgress to run it on the lake we had to introduce two additional services one for reads and one for rights for rights and transactional consistency consistency uh we built safekeepers they implement a consensus protocol called paxos. So they give you low latency rights. And then we introduced another service called page servers that serve pages to posgus compute. Those systems are called page servers and they deliver on low latency reads. Then we put it all together and integrate with the lake and that gives us uh lakebase the fully

2:12:59 managed serverless posgress that runs on the lake. So what does lakebase give us? Well, um it certainly is nimble, right? So, we can provision posgress in under 500 milliseconds and as you have lots and lots of developer uh staging um environments for every potentially PR that you you want to move into GitHub.

2:13:25 Um we can scale that also to zero, right? So, with lots and lots of environments and they're serverless, we need to deliver in TCO. So we automatically shut down environments

Lakebase enables instant branching and disaster recovery

2:13:36 that you don't use. We introduced two more patterns that are incredibly useful in this new agentic world. The first one is branching. Every Postgress database you can easily branch and it also uh branches in about 500 milliseconds. So you can create an additional environment for you to create dev test or staging. Another another pattern that kind of emerged in that agentic development is uh snapshot restore with any with a simple mouse click or an API call. You can snapshot your database then unleash your agent to do work uh to build some software, change the schema, change the data, whatever. Uh if the work is to your liking, you can proceed. If not, you can instantly roll back to the previous snapshot.

2:14:28 So now uh it's built for agents but is it mission critical? Can it support your business? Let's start with scalability with Legbase. All you need to set is uh error bars for your compute. You can say well don't scale my compute below this minimum or over that maximum and you want to do it for potentially cost controls. But within two those error bars, you actually can uh um scale up and down automatically um as your workload changes. So we'll scale the system up at at your peak hours, scale things down um at maybe on weekends or nights.

2:15:09 Storage of course is on the lake and that gives you infinite scalability. you will never run out of storage and you will never run out uh you will never need to run management operations if you're approaching you know your disk size or whatever. So now you might be wondering like so you changed the architecture of posgress at the storage level so is it fast uh how does it compare with the industry implementation of posgress as a service so this is a very similar um graph to what rail was showing for Raiden where we measuring latency as we scale throughput obviously here it's an operational system so it runs much smaller queries than an analytical systems, but they run it at at very high concurrency. And we're comparing it uh with uh the first cloud vendor, which is quite popular. And that cloud vendor taps out about at about 130 uh operations per minute. And by the way,

2:16:10 we're running a fairly standard industry benchmark called Trock. another cloud vendor with slightly different architecture was able to push um uh its throughput to about 350,000 operations a second but after that of course latencies started to spike as well and I'm very excited to to to showcase some of the incredible performance work that we've delivered and legase can scale under 10 milliseconds for each transaction for each operation all the way north of 600,000 um operations per Second lakes base is is ready for almost any workload. Uh mission critical also means a lots of features, right? And those features are security, compliance, encryption, you name it and we have it.

2:17:03 But I also wanted to um showcase something that we are in a very very unique uh position to deliver on. So cloud outages don't happen every day. Uh but when they do happen, they're they're devastating to your business. And now that a lot of your business is increasingly automated and run by agents themselves, cloud outage could could be really really disruptive uh to your operation. So I'm incredibly excited uh to introduce first fully managed crosscloud disaster recovery.

2:17:45 This is what we mean by truly missionritical. With fully managed crosscloud disaster recovery, you can set up your system to run cross clouds. So you can uh provision lakebase, let's just say US West uh uh AWS and then a a replica US East Azure. And in the case of AWS outage, you can instantly fail over from one cloud to another and continue your uninterrupted business operations.

2:18:18 So this is Lakebase. It's fully managed serverless possess that runs on the lake and it is mission critical. We've only been on the market for a year, but we already have over 3,500 enterprise customers that trust us with

Mastercard deploys secure AI agents rapidly using Lakebase

2:18:34 their emission critical workloads. One of them I would like to invite here on stage. Please welcome Federica Cohen from Mastercard. How are you doing? Thank you for having me.

2:18:57 Doing good. Uh excited to have you. So, Mastercard um aims to have data and services account for over half of its revenue. To get there, you consolidated um 80 different services is the one I platform. So, what didn't work in the past and what's working now? Well, it it's a great question and maybe just to level set just to give a sense of the dimension that we're operating in. We're operating in about 200 uh over 200 countries. We have billions of card holders and we process more than 150 billion transactions every year. And so at the scale that we have to operate, bringing together our services was very critical. Now, Lakebase helped us accelerate that by creating a shared foundation where we can bring in differentiated insights, our governance and isolation when we need it to basically bring together uh agents that can reason in real time, but more importantly can work together with that

2:19:57 shared context and that architecture. So you've built a a reusable architecture um here using uh Unity catalog and lakebase to help with the creation of a virtual seuite. How does standardizing on lakebase help?

2:20:15 Well standardizing on lakebase has been pretty critical. Uh for example in March we announced a new solution called virtual seauite. It's designed to help small businesses operate with executive level decisioning. Think about the average small business owner. They're very busy. It is very difficult for them to take decisions on cash flow, on forecasting. And so what we've done is we've created a suite of agents that will help each as a digital executive aiding in a number of digital responsibilities. The first one we're starting with is the virtual CFO. It brings and helps uh make decisions on cash flow, on payments, and how to make working capital. All of which are make or break decisions. And so we use Lakebase to create a foundation that allows us to take in insights from their data, from their performance, bring it all together. And the benefit is because we're using Lakebase, we're able to have

Ensuring secure multi-tenant data isolation

2:21:08 that shared understanding. So as one agent takes action or generates insights, it becomes instantly available for the next one.

2:21:15 So let's talk about multi-tenency. With thousands of issuer banks on a single platform, um multi-tenant isolation is a high stakes challenge. How did you use legbase to approach a secure bank level data separation while enabling AIdriven insights that can continuously learn and improve over time?

2:21:34 It's a great question. So there is another solution that we've been working on that's coming soon called uh performance pools and for some of these solutions we have to support thousands of issuing banks all on a shared platform. Now think about the challenge that we have to have for us. Data isolation isn't optional. It's foundational to the trust that our clients have with us. uh and it helps us manage things like data residency and other requirements that we have. And so this capability that is coming soon that helps us bring always on insights then to action to measurement in a continuous loop requires us to keep our issuing data isolated and separate for protecting each of our clients. But at the same time we have to have the ability to consolidate insights in a secure and transpar and anonymized way.

2:22:18 So to draw it, Lakebase allowed us to basically create this solution by establishing a single foundation that then can apply to thousands of issues at once. And so for us, it's enabled us to have that trust and that embedded foundation within that we can create for one client and then expand and scale horizontally to all of them.

2:22:39 Your team went from concept to a scale ready MVP demo in seven weeks. Uh my mind is kind of blown. I've never seen anything like it. Um and and for a company with with Mastercard's regulatory bar, this is kind of wild. Um so how did keeping um agent memory, model serving, governance inside of a single platform allowed you to move fast without sacrificing trust?

2:23:04 We we're incredibly proud of the speed that we've been able to achieve. But part of the key piece is that we are uncompromising in our approach to how we manage and govern our AI and our data. And so what we were able to do with Lakebase was to build all of those foundations from the start. So when we start developing a new MVP, what we what we have the ability to do is embed that governance, the guard rails that we need and a number of the other isolation and foundations that we need to have in place from the get-go by building this in with the lakebased architecture that allowed us to move much faster than usual because all of those trust components become non-negotiable. So as opposed to accelerating by cutting corners, we accelerate it by building the foundation from the start into something that frankly now we can reuse every time we build additional agents.

2:23:50 So looking ahead, what becomes possible as your agents get more autonomous? What are you looking forward to building uh on top of data bricks platform?

2:23:59 Well, I'll go back to something you said at the start, Nikita. Um we aspire to have our services power more and more of Mastercard. Well, what that means in an agentic world is that over time we're going to have an increasing set of agents that our clients and the ecosystem uses, but also it means that they have to be trusted and that they have to have that shared foundation within. And so as we start moving from agents that not only provide insights and information, but can help you take action, help drive the ecosystem, that shared foundation is what's going to help them learn from each other and drive their power even further. So when you think about some of the things I mentioned like the virtual sea suite like performance pools like capabilities we're driving in agentic commerce bringing that level of trust for us is the currency innovation and we believe it's going to let us scale much further thanks to that shared components that we'll be able to build in.

2:24:49 Thank you for being an incredible customer fed.

2:24:51 Thank you for having us. Thank you

OpenAI accelerates model cadence and agent utility

2:24:52 everybody. And now a co-founder of data bricks Patrick Wendell. Welcome to the stage Open AAI's president and co-founder Greg Brockman and Datab Bricks co-founder and vice president of engineering Patrick Wendell.

2:25:35 So, you'll sit here. Um, well, my guest needs no introduction. Uh, but I will I will do one anyways. Uh, um, Greg Brockman is co-founder of OpenAI and, uh, we're thrilled to have him here. Uh, Greg, I'd actually love to start by talking about your everexpanding role at OpenAI. uh you're known as sort of the godfather of the model production process, the infrastructure and talent and everything needed to to power that core engine, but uh but I know you've recently started taking on a much broader role uh as a founder and I'd love to hear the motivation behind that and where your focus is these days.

2:26:16 Well, first of all, thank you for having me. It's amazing to see so many people who are passionate about data bricks about where technology is going in in one place and I at OpenAI you know I've been involved in basically every aspect of of our technology that I can you know help with since our very early days and the way that I tend to operate is whatever the most important pressing problem of the day is I'll try to go and and make a difference there. Um there was a time like for GPD4 where you know I was up at 2 am every night like you know getting the the model back up and running trying to figure out like you know deep binary search through our nodes to go find uh which one was was you know causing a problem things like that built build a lot of the training frameworks and the these days one of our most important problems is that we've made we have such a great model production process right that we really have spent a bunch of time over the past couple years to uh be able to innovate at every single layer and connecting those models to the world is actually becoming this most important critical

2:27:17 challenge. And so I've been spending a lot of time really helping the teams focus on what do we need to do to bring all this amazing research to bear building the products um having a really focused effort to have one way of building agents uh that are able to add value to you know people people in this room.

2:27:34 Yeah. And certainly uh allowing these models to do useful work inside of an organization is something everyone here cares about a lot. Um and I will get into that. But before we get into it, I do want to talk quickly about the models themselves. Um you've been on such a tear in the last 6 months or even really three months. And if I have this right, you know, 3 GP 3.3 came out in February, 3.4 in March, sorry, 5.3, 5.4 in March, and uh 5.5 in April. So it was almost once a month. And and as a user of those models, I use them largely in codeex.

2:28:04 Like each each model was just a huge leap from the prior one. Um I'm curious what accounts for this like really rapid development in model quality and and how do you see things moving forward?

2:28:14 Well, it's very much the the you know the duck analogy where it's like you know it all looks very smooth and seamless and you know here's these models on a six week cadence that kind of thing. Under the hood it is a lot of work. It is a lot of teams coordinating together. And the interesting thing for me as a technologist, you know, I did have, you know, before open I was I was working on Stripe and there, you know, you sort of see what it's like to build a technology company and, you know, you might think that building these models it's like totally different, but it's actually pretty much the same just in a very different domain um that requires a lot of like really sort of, you know, parsing apart this complicated space into a bunch of different layers. So there's pre-training, there's post- training, there's um a bunch of just like dealing with like you have some race condition, you got to go find that, figuring out how to get the inference into good shape. Um so there's a lot of technological problems that all stack together, thinking about the safety, uh how we actually are going to release this pricing, packaging, the whole thing. Um and that we basically have a flywheel internally that I think it just

2:29:15 goes faster and faster in terms of just like how does every team contribute to each of these releases. And so at a technology level I think that we've just made huge strides like we were kind of you know two years ago I think that releasing a model or even just like training a big model was a huge amount of pain and that the teams at every single layer infrastructure the research everyone has come together in order to make that process smoother better um and to be able to just work together as one to get these models out. So it really is a repeated process where we're building a muscle.

2:29:44 Like we at data brick certainly saw when we rolled out 5.5 in our development process there was like huge wins and uh and I think our engineers certainly saw it as like absolutely at the frontier around those types of workloads.

2:29:55 Yes. And the the exponential continues, right? And that is the wild thing about this field, right? It's like every single time a model drops, you try it, you're like, "Wow, like mind is blown. Here's this problem I couldn't do before. Now it's easy." And you just project that forward, right? And it's like every single problem both in software engineering, but also now in broad knowledge work, right? Any sort of work you do with a computer, it's starting to be something these models can accelerate, right? It's like this rocket ship that just moves you forward on anything. And the exponential, it's not going to stop. And I think that's something that's worth pricing in as people think about how they're building their AI strategy.

2:30:29 So, am I hearing that you're going to be able to tell us something about the next model timing wise, capabilities wise? How much how much can we'll keep this just between us. Don't worry about these guys.

2:30:39 Of course. Of course. I mean, look, like the the we we are continuing to work super hard delivering the next level of capabilities. Um, I think that the way to think about the moment, right, they were going through is something like GPD4 that that was the that was that was a model that was great for chat because it was the first time that you had this interactivity. It was worth reading the output of the model when it was something very crafted for you. Whereas in the current moment, we have these models that can really do useful work, right? That they actually are able to accomplish things. They're able to use tools and so you want those to be brought into your workflows, right?

2:31:11 Software engineering is the first place that I think really felt it, right? of wow if you actually utilize these models it goes from 20% of the work of the you know sort of grind work being done by the models to 80% suddenly it changes the tools you're going to use and I think we're going through that moment right now for workers everywhere right for any kind of slides presentations powerpoints like all of these things um that you do or just connecting data between different systems and all of that is going to get much much better and so the way I'd look at it is like we're going to keep having step function better models that are more useful to you across anything that you need to do in your Um we're we're figuring out how to make these models broadly accessible. We think that's part of our mission is to bring these models to the world in a safe way. Um and that that is something that we're doing kind of in partnership with many companies and uh and bring bringing to bear. So uh that pace of progress the you know some model release cadence those are all things that we

Data quality underpins autonomous agent capability

2:32:01 intend to keep up.

2:32:02 Yeah. Fantastic. So, you know, this conference is the data plus AI conference. And so, you have folks that are data practitioners, folks that are focused on AI and and the synergies and combinations between them. Uh, one thing that I think is sort of underappreciated about uh OpenAI's work is the degree to which you use data in a way to inform the improvement of quality in models and in your own products. And it's been um it's also been a a privilege to be uh a part of that. You know, data bricks is uh publicly used by OpenAI for certain things. And I I love seeing when a gra when you or Sam tweet something and I see a graph, I'm thinking, "Oh, I know that's a data bricks graph. I'm always texting you about that." Um, but yeah, I'd love to understand a bit um what role does data play even just for your own development of products and models at OpenAI and uh and yeah, any anything about also how data bricks or other pieces of infrastructure are important in that? Well, fundamentally I think that data is almost the core ingredient upon which I don't know if you want to think of our models as like

2:33:02 a cake or something, right? You know, flower work data is probably on the bottom.

2:33:07 Exactly. Or something like that. Uh and that this is true both for how we actually produce the models and the amount of return on just like actually making sure your data is clean is like not to be understated. Um but also more broadly understanding how are people using our products, right? really understanding, you know, chat chatbt is the most broadly deployed AI chatbot and that there's a huge diversity of things that people do with it, right? And that a lot of what I really try to that I really care about is thinking about what are the use cases, right? And there's a lot that we dig into where people are using our models in like these surprising and amazing ways we never would have anticipated. And sometimes you know it's applications like uh in healthcare people using using AI to get information uh about you know you have a medical report that you upload you get an insight from that that you wouldn't have gotten any other way you don't have access to a doctor and just being able to find and understand the different ways that people are getting value from these models is something we spend a lot of time thinking about and doing. Um on the agentic side, I think that that they're just like even just really looking through um just understanding

2:34:09 how is an AI actually solving a problem. It becomes much harder to just read a trace, right? You need some better way of looking at these things analytically. But the amazing thing about where we are is we have this technology that's capable of sort of ingesting information at a much larger scale and rate than than humans alone could. And so sometimes we can actually use our own models in order to understand how the models are are operating and behaving. And this is true on the insight side, but it's also true on the safety side, right? For example, one of the, you know, sort of most painful parts of using codecs has always been approvals.

2:34:40 And now we have an AI layer that's able to do those approvals for you.

2:34:43 Yeah. Fantastic. Any data bricks feature requests, you know, I I can try to make it happen if you've got

2:34:48 I mean, for us, it's always scalable. Uh we want it faster. We want we want better better latency. Um, so I think it's just in general the big thing we want is just like we we have such major workloads and everything is scaling and and falling over constantly and so just like the the kind of basics of infrastructure reliability is always going to be our top one.

2:35:06 Better, faster, stronger.

2:35:07 I mean the the usual.

2:35:10 Um, uh, great. So I'd love to talk also, you know, data bricks and open last year announced a partnership together. Um that partnership was a hundred million dollar partnership initially focused on uh allowing our customers to have access to the GPT family of models to combine with their enterprise data. Um and in you know the last six months as as you've continued to innovate around codecs. We've added support for that in our AI gateway and and we've also worked together on uh allowing Genie RII to be used natively from inside of codec. So lots of exciting stuff happening there.

2:35:40 I'd love to hear how OpenAI thinks about partnerships in general. What what roles do partners play as you kind of are expanding your presence and then anything specifically? You know, we have thousands of joint customers. A lot of them are in this room. You know, what what do you want them to hear about our future work together?

2:35:56 Well, number one is that we view what we're doing is that we're kind of a a spark in the overall engine of of the economy of what you know this AI revolution that we're all building together. And so partners are a critical part of that that we really want to be building with the ecosystem. You know, there's there are certain things that that we're able to do in house and we'll double down on, but I think that the only way that we get there is through the leverage of what everyone is doing and working together with companies like data bricks is absolutely essential for us. And so, it's been really amazing to see how people are using systems like Genie powered by our models in all these different contexts. Uh, you know, people using it internally within their own companies. And so, how many people in this room have have tried codecs?

2:36:38 So I think that everyone in this room should absolutely try it because it's like we have been evolving what Codex is in a very significant way. Like I think that that first of all the codeex app is something that's very differentiated. It's something that is worth trying. It's like there's nothing like it right now. It feels like a very different way to use a computer. Um and and Patrick maybe you could talk about your own use of codecs like yeah

2:37:01 yeah I mean we've like rolled it. I would consider us early adopters at data bricks and uh and not only are we seeing huge productivity gains for our developers but just the pace at which it's improving is remarkable. I mean I come back you know either a new model release or an update to the desktop software and like everything is just getting better very very fast. So, you know, we're always encouraging our own customers as we do to look at multiple AI solutions, AI vendors, kind of empower your engineers with the best models and tools that they that they can find. And, you know, certainly it's become a very critical part of our of our development stack.

2:37:34 Yes. Yeah. And I think it's it's been very interesting to watch kind of the competitive dynamics where um you know I think we were relatively late to the game in some ways for building the you know codeex type form factor but we have focused extremely hard and the rate of improvement is so high and so we're bringing this technology to make it more accessible available to everyone. Um one thing people actually underappreciate is codeex itself is open source right and that that is like in our DNA is how do we actually make this be something people can hack on and use in all the context that they want.

2:38:02 Yeah. So it sounds like your message is try codeex out and my message is you can do it with data bricks. It's super easy especially you know if you're already uh working with openey and data bricks together. So uh so yeah we'd love to to have everyone try it out. Uh there's also many talks at this conference about different folks through our partnership building cool applications and internal tools uh with data bricks plus openi. So a lot of exciting stuff uh for folks here.

2:38:25 Yeah it's never been a better time to be a builder.

2:38:28 Yes. Like I think that that to me is the most amazing thing about this moment is that creativity is being unleashed that anything you want to do suddenly it's possible. If you can imagine it you can build it.

2:38:40 So we're almost out of time. I think for my last question I will ask you you know earlier this morning Ali was talking about AGI and in his mind AGI is here in the sense that the types of tasks that you know our our partners our customers care about are almost already perfectly executed by uh by AIS. Now AGI means different things to different people. What's your take on you know AGI? Do we have AGI? Is it very far away? Is it nearby? Is it undefined? You know how do you where do you sit in that one? I I I do think it's kind of wild that AGI is this like very personal thing. It's almost like AGI is a feeling, not a defined thing.

2:39:15 It's the friends we had all along.

2:39:16 Exactly. But I do think that there is something significant that is worth thinking about. And I think maybe one of the category errors that that I think about is that AGI is almost a spectrum, not a not a not a moment, right? Or it's like a AI progress is not a thing that has an end point, right? It's going to continue. And I think of the AGI moment is something we haven't hit yet. Like the way that I think about it is when you really do have an a you know sort of assistant that is able to really autonomously go and and work on any task for you and accelerate you and and do that at the level of of the top humans.

2:39:49 And we have this jagged intelligence, right? It's something that actually really accelerates people in certain ways, but it's also like very limited and stunted in other ways. And I think this is a great opportunity because one thing we really stand for at OpenAI is keeping humans at the center, right? that we really really want humans to be setting the goals to be in control and to make sure these these systems are in benefit of humanity. And so that that means that we have this opportunity to sort of leverage these systems in cases that we really want, but also have a lot of time to think about where do we want the humans to fundamentally remain and to remain the drivers of of everything that happens from here. And so um I think we're not I think I think we're we're not nearly at the end of this exponential. It's going to continue. And again, everyone in this room, I think, has something to contribute to to that change. So in a sense is it fair to say the you know the framing of AGI versus not AAGI is like not a helpful framing at this point or or how do you feel about that?

2:40:36 Because because everyone has a different definition. I think it's not the right framing. I think that the right framing is what are the capabilities? What do we want these AIs to be doing? How can we benefit from them? How can they help us? Right? And I think that keeping that at the center and thinking about what we're building together to me that is number one.

2:40:53 Awesome. Um well I think our parting recommendation is try out codecs. Uh if you're a daticks customer, you can use it with AI coding gateway extremely easily. It takes about one minute to set up. Um I think Ali will come on next to uh keep the program going. But Greg, thank you so much for being here. And everyone, please join me uh in a round of applause to thank Greg.

2:41:16 Thank you for having me.

Eliminating fragile CDC pipelines with LTAP

2:41:17 Yeah. All right. Okay. That's awesome. We love partnering with OpenAI. They're awesome. Uh you should check out Codeex. Uh they're also using data bricks. Uh you know they have all these you know almost billion users and they put all that data in data bricks and they analyze to make sure that you know nobody's jailbreaking or nothing you know weird is happening.

2:41:46 Um so let's bring us back to the data realm. We have the most uh exciting actually announcement still left. Uh so you don't want to miss that. Okay. So what does the data realm look like? We already talked about the right hand side. This is where you're asking your analytical questions, data warehousing, data science, and then Rainald showed us that now we can get super low latency, world's fastest engine uh with Raiden and Lakehouse RT. Um and then what we did is we went over to the OLTP side and Nikita who's really our expert on OTP databases started focusing on how can we actually get all of that to be fused so that you can do everything Postgress on top of the lake really fast really cheap with disaster recovery across the clouds. So how do we deal with this onslaught and the storm of agents that is coming our way? We still have these waters that we have to shuffle things between OLTP and Olap. So, let's welcome back Reynold of Raiden fame back to stage.

2:42:46 Thank you again. So, you heard from below simplified data engineering from myself about how we're going to unify analytics and uh Nikita from um modernize OLTP. But honestly, one of the biggest problems, there's still one big thing remain, which is we have this giant continent and OLTB databases where applications start putting their data in first. And if your applications become successful enough, you want to reason on that data and analyze on your data. And so we end up actually building CDC pipelines that ship the data from ORLTB databases over to your analytics continent. And how many can I get a show of hands? How many of you love your CDC pipelines?

2:43:28 All right. Oh, there's some like I would love to learn from you how you actually maintain them, right? CDC pipelines are basically change data capture pipelines. It reads the bin log of OLTB databases and it gets all the delta little delta changes and ships the delta over to analytic systems and reconstruct the state of OLTB databases in them. They're super fragile. They're very annoying to maintain. Many data engineers been waken up at 3:00 a.m. um actually because it caused data corruption. As a matter of fact, some of people at data bricks actually joke about CDC doesn't stand for change data capture. It really stands for continuous data corruption.

2:44:05 And to solve this problem, the industry started introducing um zero CDC, zero ETL, mirroring and all this are just fancy terms of let's do a managed CDC by hiding the pipeline. So it automatically happens. But in practice, it's still a pipeline under the hood. It suffers from exactly the same issue CDC test right so some of you might have heard of another term called HTAP hybrid transactional analytical system um or processing and HTAB is sort of the holy grail of database engineering the idea is you create a single database system that's capable of handling both your OLTB workloads and your analytics workloads in one system um but HTAB has I think most of you probably haven't actually used HTAB system in practice and the reason is HTAB as a category had largely failed. There's very little adoption of it and all the HDAP systems out there are kind of very proprietary. They don't have a big ecosystem and by building a single system they started compromising on both the performance for OLTP and the performance of analytics.

2:45:08 So the question is how do we truly unify the two right? If HTAP doesn't work and CDC is very annoying and causes corruption all the time how do we do it and the solution to this actually goes back to the very fundamental sort of problem or dichotomy of row oriented storage and column oriented storage. as Ali and Aikita both alluded to earlier. Um, OLTB systems needs row oriented storage because it needs to do a needle and a haststab lookup and needs to actually be doing updates on those rows very quickly. Whereas analytic system does a lot of scans of large amount of

12 Highlight · Solving the HTAP dilemma with storage-layer transcoding

2:45:42 data and really benefit from column oriented storage. If you look at architectural diagram already show you where we have safekeepers and pace servers that write data in an actual row oriented format to the data links. Um the solution actually lies right there. When we looked at and profiled the systems of safekeeper and page server, those are storage services, it turned out they're very underutilized from a CPU perspective, but very IO bound because all they're doing is local disc reads, local disc rights and network reads and network rights. So it actually give us the opportunity to leverage those idle CPUs on those storage services to do a transcoding at the very moment to turn the data from row oriented format to a column oriented format. Right? And it turned out once you convert the data from row oriented format to column oriented format, it doesn't actually increase the CPU utilization that much but actually dramatically shrink the data volume because column oriented format has a better compression ratio. is often

2:46:43 somewhere between 10 to one to 100 to one. And because of all the services IO bound by having a better compression ratio, we can now actually benefit from this transcoding instead of hurting the performance. And with that, we can actually write the OLTP data into the data lake in column oriented format. And once they're in the column or format delta and iceberg, you can actually apply any analytic compute directly against that data. So how does this work in practice? Imagine inserting a new row into the lakebase and the lakebased compute eventually sends that row to the underlying storage system and the storage system will do the transcoding right there and actually write the rows directly into the columner format in Delta iceberg. And now you can actually file up a query against your lakehouse and the lakehouse will read directly the delta and iceberg format and return your query on the freshest copy of the data.

2:47:42 So we think this is actually revolutionary. It's probably as at least as big as the uh raen talk I was talking about and the whole concept of lakehouse plus lake base. We can actually have unify the storage for both of them. And we're giving this term uh the storage technology a name called LTAP. And LTAP stands for lake transactional analytical processing. We think LATAP is HTAP done right and accomplish the goals of HTAP without actually having a single query engine for it. But we're able to actually unify the storage which is by far the most important part. And now you

2:48:11 have one copy of data for OLTB databases and one copy of data to govern. So, LTAB can truly unify your data infrastructure. You start with OLTP databases for your applications, but you don't have to copy your data. There's no pipelines to maintain. And most importantly, there's actually no compromise in performance at all for both your OLTP systems and your analytic systems. And all of this are built on open interoperable formats, Postgress, Delta Lake, Iceberg.

2:48:46 So with ELAP we can finally actually combine the separate continents of OOLTP and analytics into a single giant continent. But we want we don't want to do this alone. So we'll be open sourcing actually a very fundamental library that allows converting Postgress data directly into columnar format in parquet. The logo is actually it's kind of a cute logo that has a parquet elephant. Um so indicates what they do and this should be coming uh fairly soon.

2:49:16 So again, same question. What do you need to do differently if you want to leverage LTAB? Actually, nothing. Just start using lakebase. We'll be rolling out the alpac capability in coming weeks. And once it's rolled out, every single one of your lakebased table will automatically appear in your lakehouse. There's nothing you need to do. And we can do only do that because there's no pipeline to maintain under the hood. It's available for every single table, right?

LTAP delivers real-time analytics without performance compromise

2:49:47 And in order to uh sort of demonstrate to you, this is the last demo of the day. Um we invite Holly onto stage to actually show you a demo of Lake Base combined with

2:49:55 Elapitt. I'm going to give you a tour of both Lakebase and LTAP and why you need to combine your analytical and operational data. I have business banks across the Americas creating billions of transactions a day. And I want an agent that spots potential VIP customers um and matches them to an adviser based on their qualifications and local regulations all in time for a teller to tell them that information before you've left the door. Previously, this would have been impossible. uh the delay would have been too slow and the intervention would have come too late but not anymore. So let me show you data bricks and let's go ahead and start with lakebase.

2:50:48 So lakebase is open source postgquest fully managed by datab bricks with separated storage and compute and it's really easy to get started. I'm going to create a new project. I'm going to name it and I'm going to stick with the defaults and that's it. It's made. The compute size isn't fixed. So, uh, I can scale that up and down within the UI and set a minimum and maximum size. And on the left, you can see we've got branches here. Um, and I can click in it and I can go and create a new branch. Uh, we can name it, we can have it auto delete.

2:51:20 We can make a branch of production. We can make a branch from a point back in time, but we can also branch from a branch. And it's created almost instantly because there's no need to copy data. Speaking of back in time, we also have backup and restore. This is configured to retain seven days of history by default, but you can do more or less if you like and pick any time window to restore to. Or if you wanted, uh, you could restore from a snapshot, which is scheduled on a regular basis.

2:51:51 Now, all of this is empty. Here's what I set up earlier that's processing all of our data. And we can go into the tables, and I can see all of my transactional data here. And I'm going to need a query to scan across all of these historic balances to calculate who is an outlier and potentially a VIP. And for those queries, we care about some metrics. So first is TPS. So this is the line chart in green that is we're going to care about on the right hand side. If this number drops, our business is impacted and if it stops, our branch staff are going to have an outage. The second metric we care about with our queries is how long our analytical queries take.

2:52:31 And this is going to be the top left number in blue here. And finally, how fresh or stale our data is. And this is measured in seconds. And it will be the top right number in purple defines how up to date our data is. And if I'm quering in real time, I better get real time answers. So earlier Reynolds said there were three ways of running a query like this. And the first is what you might an agent might try by default. and that's to submit the complex analytics query to existing lakebased compute. I'm running this from the data brick SQL editor. It's possible to connect to lakebase from there. And when I'm running this query, I can see that this is not quick to come back. Uh this is not good of a one-off analytics query, but it's terrible for our agents who are going to need answers a lot faster. And we can also see that it's starting to impact our TPS. We can see that this green line is going down and it shows a bit of a worrying trend because realistically it's not just going to be one dip here. It's going to be thousands of them. You can barely do analytics

2:53:32 like this uh let alone flag in real time who would be a VIP customer and it's still going. Okay. Um, well, you all know my name. Looks like we have time for everyone to go around the room and introduce themselves. We'll start at the front, maybe. And oh, it's done. Okay, finally. Maybe we'll do that later. And in blue, we can see that this took nearly over a minute to complete. uh we can say that the sailness is also quite bad as well and we risk taking down production. Okay, so that clearly didn't work. So what if instead we took a copy of this data using CDC?

2:54:36 I'm using the lakebased change data feed feature which is meant to be for audit workloads not upto-date analytics. Um, and with this I know that there's going to be a bit of staleness with at least 15 to 30 seconds of data, but this is within the data bricks ecosystem. If I was using a different Postgress offering um, or a different CDF tool, this could

13 Highlight · LTAP query performance and zero operational impact demo

2:54:56 be into multiple minutes, not to mention the additional bill and the additional team to maintain it. We can see that the TPS is flat. Um, but it's finished running and it took 13 seconds to run, but it was out of date by 25 seconds. So finally the LTAP way of doing this and with LTAP these tables are available in Unity catalog and here it is looking like a manage table but we can see from the icons that yes it's a lakebased table but it's also available to both Delta and iceberg readers. I didn't have to set up any additional pipelines to make this happen. I just specified a catalog and it appeared. And you might notice a slight difference in our UI here and that's that we have an S3 path.

2:55:40 And if I go to this S3 path, um, I could be able to see all of my tables in here. This is great for openness. I can query this with open-source Spark if I wanted to. Uh, that's a bit of a tangent. Uh, let's head back to Elab. So, here is our query again, and I'll be quering it with a new type of warehouse, a real time warehouse. And so, now I click run, and it's done. It ran in milliseconds and we could see that this is very real time data without any staleness and it had no impact on our TPS. So we can throw as many agents at it as we like and because this ran almost instantaneously. It also means it's significantly cheaper in compute costs too. It is also cheaper to maintain because there were no intermediary tools to manage for all of us. This is huge.

2:56:35 We have a single system of record where your data can be used for both your operational and analytical systems without having to compromise on performance. All of this happened automatically. I didn't have to set up any steps in the background. And finally, we no longer have a divided view of the world between operations and analytics. We no longer have to choose. Thank you.

2:57:08 All right. So that's it. We were able to

Day 1 keynote summarizes the agentic data stack

2:57:12 unify all these different islands, remove all of the data movement into just this one big pangia. All right. So we have the with the laptop or lake tap, you can just, you know, on tap get your lake data out. The underlying data is just stored in iceberg. Okay. So it's just open source iceberg delta and you can do transaction processing and you can do analytics. You can do real time on it. You can do data science. So that's a game changer. Okay. So just to recap, we already unified data engineering and data science with spark.

2:57:40 Then we had data warehousing with the lakehouse that we unified there. And then we were able to with the raiden engine or lakehouse RT unify all of these into one uh OLTP databases and then that got us uh LTAP. Okay. So that's what that looks like. So to zoom out and recap the day quickly, uh we have our agents, but they're lacking context. We want to give them all that context on the platform. So we did a bunch of announcements today. So uh we introduced the Aentic data foundation.

2:58:10 We had the lakehouse RT that's super super fast. We had the lakebase with disaster recovery across the different clouds, LTAP, LakeFlow with all the connectors. So super exciting. Then we went to the context layer. Their genie ontology is extremely important. That's your context graph. Uh we have the Unity AI gateway for all your control cost governance of all your models and agents. Um open sharing and then we had omnient which you're going to hear more about tomorrow. And then finally we had the agentic dev work. This is all the genie agents. There's three main ones.

2:58:45 There is genie one, genie code and genie agents. Those are the three main ones. Uh and then finally we talked about the apps that we're getting into and we're very excited about those. Lakewatch for your SIM and customer lake uh for CDP. So that's a wrap uh for the keynote session today. 11:30 we have the sessions and also tomorrow morning don't miss Mate's keynote on Omniant. Uh have a great show for you. Thanks for coming.