When Data Lakehouses and DataOps Collide: Open Source Nessie and Lakehouse DataOps | DataOps Day

Hey everybody, this is Alex Merced, and welcome to, uh, this keynote presentation as part of Data Ops Day. Um, we're gonna be talking about data lakehouse and data ops. So these are two very important trends in the data industry, um, that are addressing very important problems in Martin data architecture.

Um, but before we go there, let's just talk a little bit about who I am. My name is Alex. I'm a developer advocate here at reio.

I've worked in the data and webspace for several years. Um, basically, uh, you know, whether it is educating people about creating data pipelines and archite their data or training people on web development and web development practices, uh, I've spoken at many conferences like day-to-day, Texas, O s a Con, P 99 Con. Also, I'm more of the authors of the upcoming O'Reilly book, uh, Apache Iceberg, the Definitive Guide.

So, um, that's a little bit about myself. I'm also the host of several podcasts, and again, also the author of Apache Iceberg, the Definitive Guide, which you can get yourself an early copy by scanning, uh, that QR code. Okay?

And again, the host of many podcasts. One is our weekly, uh, our weekly, uh, broadcast every week on, uh, Demio at gnarly data waves. Uh, select star from Day Lake, which I do with my counterpart at Dremio, uh, Dipankar Mazumdar, and, uh, my other show at Data Nation, where I just talk about data topics and data concepts and helping people kind of understand what's kind of going on in the pulse of data today, all available on Spotify and iTunes.

Now, let's just kinda understand, first off, what is a data lake house so we can understand what is this trend of a data lake house? And for that, we need to understand sort of like what the modern challenges are for data today. They kinda really fall into three different categories that lead to sort of three different trends that are going on in the data world right now.

Okay? The first being that data's growing faster than ever before, okay? And because of that, the use cases for data are growing faster than they've ever had before, and that's leading to no particular tool, no particular platform being able to really serve all your needs anymore.

You really do need to use multiple tools and have, you know, uh, the ability to switch tools when you need to for different use cases. So just kind of putting all your data in the data warehouse just doesn't quite work. So what happens is there's one place that generally most tools can access your data, and that is the data lake, the central repository of data that we generally have.

Um, but in the past, the data lake has always just been sort of read only. Basically we would land a bunch of data and like parquet files on, let's say an on-prem Hadoop cluster or, you know, an Amazon an S three, and we would run ad hoc queries using different query engines like Dremio, Trino and others. Um, but that's about it.

Okay? Um, we, but we wanna do more because again, we wanna be able to satisfy those use cases. So with the recent advent of data lakehouse table formats, that allows to create a metadata layer on top of the data in our data lake that allows all those compute engines to treat our data, like database tables that allow you for full asset transactions, uh, that allow for, um, you know, things like time travel and all the really great features we formally only associated with the data warehouse, we're able to kind of robust the best of the data lake world, that open sort of environment where anything can use our data with all those patterns that we apply on data warehouses and create something called the data lake house.

So the Data Lake House addresses, uh, that particular trend. But, you know, having to move data across so many tools and having so much data also means that the quality of that data and getting that data where it needs quickly matters more than ever. Okay?

And this is kinda what brings us into the world of data ops. Okay? Uh, you know, sometimes you also hear referred to as like data as code.

We're borrowing from sort of the DevOps world in the software development in, in the software development industry, and borrowing a lot of those practices and applying a lot of those practices to how we manage several different aspects of data. So again, the goal of DataOps is to improve communication, so that way we know what's going on with the data, where it's going, why it's going places, and just be able to provide information for the purposes of making sure that their data has better quality and reliability. We can trust what the data is, but not, but to do it in a less manually intense way.

So we wanna automate that process so that way all these things kind of just happen, and we make sure that we always have nice, fresh data available, and we wanna be able to do this in a way that we can rapidly iterate on. So as basically our use cases and needs change, we're able to kind of adjust it, okay, we want. And then at the end of the day, all of this automation, all of this data quality is for the purpose of data delivery, making sure the data is available where it needs to be available for our data consumers and for us to not get any regulatory fines, right?

Because different industries, uh, we have to follow different rules as far as what data can be available, what personally identifiable information can be, and when do we need to get rid of it. So we wanna make sure that the right people have the right access to the data that we're complying with all those rules. Um, and that's part of this whole data ops.

How do we operationalize all of this in a way that's sort of maintainable fast? And basically, we have a lot of tools in the data ops toolbox that allows us to do this. We have version control, which is gonna be what I'm gonna be talking about a lot today with, particularly with a project called Project Nessi, uh, which really kind of helps bring a lot of these data ops practices, uh, to the data lakehouse, uh, C I C D.

Okay? So basically the idea of just automating us, integrating data, so CI for continuous integration. So us merging in new data with the existing data, continuous deployment, us publishing that for access to sort of our data consumers automated testing.

So being able to kind of test that data. Um, you know, there's a lot of popular tools for that. Stuff like, uh, grid expectations monitoring.

Okay? So many different tools out there for like data observability to kind of monitor what's kind of going on in your data pipelines. Uh, containerization being able to take a lot of these processes and put them in containers so that way they're easily, uh, replicatable.

Um, and, you know, basically they don't affect each other. So basically you have a more robust sort of infrastructure and orchestration, making sure all these things happen at the right time. So if I have a particular step in this entire process that requires three other steps to occur, first, orchestrating that using orchestration tools that basically say, Hey, long, you know, if job C depends on job A and B, job A, it won't run until A and B are complete.

And something like, and tools like that. So using like dags directed a cyclic graphs. So again, we have the goals of data ops, which is again, to basically make it easier to deliver data and high and fast quality with low, um, uh, low manual work or less manual work.

And we have these components that kinda help us do that. And, you know, if you decide to adopt a data lakehouse pattern, which many people are adopting because of the challenges I've mentioned before, you're still going to need to have all these data ops patterns. So before you might rely on, you know, tools that were built into your favorite data warehouse platform to kind of apply some of these practices.

Well now you need to kind of build this robustly from point A from end to end in your data infrastructure. Um, and Nesti is a great tool for doing that in the data lakehouse world. And what is nesti?

Nesti iss an open source project, and it's an open source. What it is, it's an open source data catalog built for the data lakehouse built for this kind of environment where we kind of need to be operating from sort of our, our data lake and maintaining that quality and automating these processes and so forth. So what does it do?

Like any data catalog, it tracks what data we have and where does it exist. Now, primarily what it does, it catalogs Apache iceberg tables. That's one of those table formats that allow us to interact with our data as database tables, but it allows us to version that data catalog.

Okay? So in that case, any time any table that's tracked by our catalog changes, we create a commit ending nesty catalog, which you can see sort of here. So here's a commit one, okay?

Table A, table B. And basically what we can do is that, let's say we wanna isolate, uh, some new data that we're gonna bring in. We can create a branch which doesn't create any physical copies, okay?

So a branch can also be used to create zero copy clones for data experimentation. But we can also use it to isolate changes to our data. So we can integrate new data on a branch, allow us to run all our data quality checks, and then after we've tested our data, we've run transactions on multiple tables and all of our data's kind of where it needs to be, we can then merge that branch back into our main, uh, our, our our main line, which is where all our data consumers are gonna be consuming the data.

Okay? So this allows us to kind of practice what's traditionally known as the right audit published pattern where you would write the data somewhere you would audit, and then you would make it available to users. But we do it in a much easier way, because instead you just write it to a branch and then you do all your auditing and then you publish it by merging it back into that main line, okay?

And basically making all those transactions atomically available, uh, across multiple tables. So basically you're, you're having multi table transactions, which before was really something you couldn't really do on the data lake. You would need something like a data warehouse to be able to do, um, or have to stick to one single platform to have transactions occur on multiple tables and get published simultaneously.

So nesti really brings a whole, uh, lot of heft, a lot of possibilities to what you can do on the data lakehouse when you use that to track your tables. So some of the benefits of using nessi versioning, as we mentioned already, is isolation. Okay?

So we can isolate the work of bringing in new data into our data. Lakehouse no copy experimentation. So essentially when I create a branch, it's kinda like creating a branching gi, it's creating a alternative path of changes without necessarily duplicating the physical data beneath.

So that way if someone wants to experiment with the data, they can do so without harming sort of the mainline production data. Multi table transactions, multiple transactions can occur to multiple tables on one of these branches. Okay?

Um, and then you can again publish all those changes atomically to your main production line by just running a merge transaction, okay? Rollbacks, okay? So while Apache iceberg and other table formats, uh, allow you sort of to roll back the table to a previous snapshot from within a single table, but by having a catalog level abstraction, I can roll back the entire catalog.

So if I have a problem with multiple tables, instead of having to roll back each table individually, I can just roll back the catalog to an earlier commit and go back to my data looking like it did at that point in time, reproducibility. So typically we would do this with something, a feature called tagging. So in Git you would tag certain points in your code, and now you can quickly look up those points in your code.

Same thing with, uh, Nessi. You could tag your catalog. So that way, if you need to, let's say you want to tag your data as is, is at the end of each month, you can create a branch at the end of each month or just tag that particular commit.

So that way you have a name. So you can always say, Hey, how did my data look like on this particular, at this particular point in time? And easily refer to it without having to look up that individual snapshot.

Again, you can do that with individual tables, uh, 'cause it's built into the Apache iceberg standard, but then you would've to tag each individual table, nesty makes it where you can just tag the catalog and get the same benefit in a much easier to consume way. Okay? And then also allows you to do consistency, quality, and data validation, because you can isolate your work, you can make sure it's consistent 'cause you're only publishing the changes once you're done doing it, okay?

You can do your data quality checks on that branch. You can validate your data. And again, all of this happens before you publish the data.

So by having all of these benefits of being able to version your data on the data lakehouse, what then happens is we're meeting a lot of those benchmarks and providing a lot of those tools that we saw in those components of data ops. Okay? We see the versionings, we're versioning tables, okay?

We can automate these processes. So we have that C I C D, okay? We have the continuous integration of integrating the data.

So we can automate the idea of generating a branch and then automate our ingestion jobs into that branch, automate our data tests while the data's on that branch, and then we can continuously deploy that data by then merge automating the per when those tests pass, automating the merging of that data into a branch. Okay? So basically you create this sort of nice pipeline that again, allows us to, you know, say, Hey, if there's a problem, if one of those tests fail, we'll, we'll, we'll, we'll go there and manually kind of take care of it, or, but if they'll pass, we can automatically integrate that data and deploy that data.

That's the benefits of versioning. And again, the benefits of nessi is oftentimes doing this at the catalog level. So again, you're doing all of these things at the catalog level, not the file level, not at the table level.

So when you're branching, you're branching the whole catalog, you're merging the whole catalog, you're tagging the whole catalog, you're rolling back the whole catalog, okay? This makes a lot of operations a lot easier because you're not having to do them granularly table by table. You can, but you also have that flexibility of being able to do it at the, the catalog level, it's open source, okay?

So that basically you can change it, you can work with it. Um, there's a cloud managed service in, in order to use a Nesty catalog called Dremio Arctic. So if you don't wanna deploy your own Nesty catalog, you can use the Dremio managed service that also provides automated table management.

Okay? It's cloud agnostic if you want to use, uh, because nesti ISS an open source project. It doesn't matter what you use as your cloud store or as your cloud vendor, whether it is a w s uh, Azure, Google Cloud, it doesn't matter where you store your data.

So if you're sing your data on an on-prem Hadoop cluster or in an object storage on a cloud provider, you can use what you want. So basically, it allows you to design the lakehouse you want because it's an open source project. It gives you that flexi flexibility.

It's not marrying you to any particular vendor or tool, okay? And you have access control. So there are governance features in nesti and cleanup features to help sort of clean up, you know, access files and things like that, uh, that might occur after you as you expire, snapshots and whatnot.

So it's a really great, uh, open source project for really kind of enabling more data ops type practices on the data lakehouse. Okay? Now, Dremio Arctic is a lake house.

Um, basically is this nessi as catalog, but distributed as a service. So basically it's a cloud managed service from Dremio that gives you all the benefits of nessi without you having to deploy it on your own. Again, you can still use it however you like.

Um, really designed to work with sort of iceberg, uh, Apache iceberg tables. Uh, basically it integrates with the Dremio software in a way that makes it really easy for you to create a new catalog of multiple catalogs for your use cases, which just makes it really good to handle data match. So remember I mentioned there was like three challenges kind of going on in the modern industry.

We mentioned two of them. So the idea of DataOps being a solution to sort of maintaining data and, and automating data processes, data lakehouse for basically making our data more accessible to multiple tools. But the third challenge is basically making sure that the informational context of our data to curate that data in a timely way is available.

And the problem is when you have centralized IT teams, as the company expands, the scope of your data gets so large that it's really hard for that centralized team to kind of deliver the data and understand the data in a way to deliver, make, meet those SLAs. And this is where data mesh comes in and comes along, allowing you to sort of break up the, uh, division of labor when it comes to producing that data. So you're saying, Hey, people who are experts in their domains put together different data products.

And with Dremio Arctic, which is, again, nesti delivered as a service, allows you to kind of sit there and say, Hey, here's a catalog for our supply domain. Here's a catalog for our marketing department. So that way each domain can have their catalog that's managed, taking advantage of all those DataOps practices that Nessi enables, but in an easy to manage way, okay?

Also basically offers even more enhanced ability to access control and govern through a easy intuitive ui. Um, and governance features, uh, automated tap table optimization. So basically one of the things about when you're working with a data lake and data lake house versus let's say, using a tool like a database or data warehouse, which auto generally automates the sort of cleaning up of your tables under the hood in a data lake house, you kinda have to do that yourself.

But, you know, with, with Romeo Arctic, uh, using that nesti catalog, it can automate, automate the optimization of those tables for you, okay? And table cleanup, so that way it can clean up those excess data. So you're getting all the benefits of nesti and this nicer, easier to use package.

And you get the benefit of the isolation, the version control. So if you wanted to kind of see that work, see, like actually like see me run through a bunch of queries with nesti and see how that all kind of comes together, uh, when you branch audit the data, merge the data here, you can, here at this QR code, you can go watch a video where I, I walk through that step by step. Um, we'll walk through the code that I, I do during that demonstration in just a moment.

But just so you know, if you wanted to see that, you can go see that at this QR code right here. Okay? Now, basically in the example that I do in that video, what's happening is that we are a company that handles virtual assistants, okay?

And basically what's gonna happen is we want to ingest some new data. So we have some new virtual assistants that have come on staff. We have some new jobs that we ran with those virtual assistants with some new customers, and we wanna add them to our data.

But of course, we don't wanna accidentally publish that data to our mainline data consumers until the, the data's ready. So what we're going to do is we are going to create a branch, and we're gonna do all that ingestion work on a branch, and that's what we do right here. So what I'm doing is I'm creating a branch and then switching over my operations to that branch, okay?

And this particular SQL syntax is specific to a Dremio sql, but Nesti catalogs are supported by Spark, they're supported by Flink, they are supported by Trino and many other engines. So basically the benefit of branching can be used across multiple tools when you're using that open source nesti catalog. So again, here I'm creating a branch and switching over to it, and then in this case, I'm creating a table.

'cause I'm doing the setup. But then what happens, I'm creating a staging table where I'm staging the data that I'm gonna integrate into, um, my existing data. So then what I do here is I run a up search statement or a merge into statement to take the data that I'm bringing in that staging table and match it up with my existing data.

So we're gonna update any records that need to be updated, add any records that need to be updated or that need to be added, and that's what that's gonna do. So wonderful, I've now added the new data, okay? So I'm well under the way of integrating the data.

And again, these are all things that can be automated. This SQL can all be, you know, automated, uh, through, you know, scripting and whatnot. Okay?

So then I'm gonna start first running some checks. So things like checking for referential, uh, integrity, making sure that for every record that I have of a sale, that there's a matching customer and that there's a matching virtual assistant. Okay?

So I'm gonna run my checks, and again, I don't have to worry about anyone querying any inconsistent data because this is all occurring isolated in our branch, okay? Where basically nothing, basically my, my mainline consumers aren't accessing the data. So I can take my time, run these quality checks, knowing surely no inconsistent data or unfresh data or low quality data is being queried by my data consumers.

Now, in this case, I might be doing all these types of, uh, data quality checks. So referential integrity, we just saw a moment ago, I might be checking for duplicates, might be checking for missing records, might be checking for incorrect data or checking for hidden data. You know, I could be checking for all of these things, but if I notice that there's a mistake and I need to take my work back, I can then just basically move the commit on that catalog back to previous commit and just basically undo basically an easy undo button, okay?

No weekends having to spend back filling records due to, you know, problematic ingestion jobs. Okay? And on top of that, if I need to do transactions on multiple tables, I can.

So in here, what I'm doing is I'm inserting a new assistant and a new, uh, and a new sales record. Now, normally without multi table transactions, without the ability to kind of branch them off on the data lake, I'd have to make these as two separate transactions. And depending on what order I do them, if I add the sale before I add the assistant, there's a chance that someone might run a, a join on those tables and realize, hey, there's a missing assistant.

Where is it? Because I hadn't added it yet. They would be seeing inconsistent UNRES data, but hey, nope, this is happening on the branch and no one's gonna see it till I'm done.

So I can make both transactions and then I can go back and merge those transactions in. I'm making those all available for everyone at the same time is one atomic commit of multiple transactions on multiple tables. Okay?

And that's essentially what we see here, where I'm running just a simple merge brand statement. That's a, that's all it takes to publish the data, A simple merge statement, okay? Making a, a much simpler paradigm.

And again, nessi also allows all of this to be, to be done through pure SS Q L. Okay? So making these sort of practices and automating these practices and understanding these practices much more simpler to a, a wider set of people using an accessible language like SS Q L.

Okay? Now, just showing you an example of how you might wanna automate this process, okay? Because again, there's a C I C D aspect of it and what makes it so the I integrating, we see how we bring in the data, you know, by just basically making a branch and ingesting the data and we see the d how to deploy the data, we merge it back into our main production branch.

But what about the c the continuous meaning we automate it so that we can continuously run Well, there's different approaches. You could do that at the end of the day if you're using Dremio, but again, it could be any platform, could be uh, could be Spark, could be Flink, could be trino. As long as you can deliver the proper S Q L for it to do the right nessi operations, you can have these sort of branch, uh, you know, branch audit, merge sort, sort of practices.

So in this, here, I give you sort of two examples. One example is we use like, uh, an orchestration tool. And that orchestration tool can then just like deliver the proper S Q L, um, to Dremio.

Or you could create your own custom service that delivers the right, like a p I calls to Dremio or whatever query engine you're using to then, you know, create the brand, do all the right tests, and then make the call to, to branch. So here what we're doing is we're saying, Hey, like let's pretend like you were ingesting the data via Spark. So I ingest the data via Spark, okay?

And then let's say I'm ingesting it into like an A W S S three bucket. So in that case, I could trigger an event using an A W S S three, an a W S Lambda, and that function will then make an a p I call to my service. So in this case a custom service using like Flask and Python, okay?

I can make a automated call to that service triggering saying, Hey, this ingestion job happened. Maybe include some metadata for it to, to, to verify the job. And then it can then make the proper a p I calls to run the proper queries, see what the results of those queries are, and if they're successful, then you know, make another a p I call to, to do the Merc transaction.

All of this can just be automated. And again, either through your, your own custom service, through an orchestration tool that can just orchestrate, uh, you know, cask over to Dremio or really in any way you want, um, because you have those flexibles. And again, Dremio is just one example of a tool that works with that open source NESSI catalog, but again, also Spark Flink, Trino and other engines also work with that.

So basically it's all about automating the tool that that is the center of sort of your ingestion process. Now, if you wanna know more, I have some tutorials I can walk you through step through step exercise on how to use nessi so that way you can see, you know, create a similar environment to maybe what's your environment to actually try it out and see sort of like that branching, um, seeing all that versioning happening, uh, firsthand. Okay?

So in this first, uh, QR code, it takes you to a tutorial where we use Flink, nesty, and mineo, okay? And you'll basically set that up using Docker containers and it walks you through the exercise of basically ingesting data with link into an e catalog and basically being with the actual data files being stored in a a a, a Minio, uh, instance. Similarly, the, the middle, uh, the middle tutorial is the same thing, but just with Spark, Nessie and Dremio.

So basically what you're doing is you're ingesting data into Apache Iceberg via Spark into that nesti catalog. And then Dremio can read that nesti catalog and make it accessible because that's the other big benefit of using a catalog like nesti. It just makes your data portable.

Any tool that can connect to Nesti can just immediately discover my tables. I don't have to reconfigure the tables, they just connect to the catalog and my tables just exist among all these tools, okay? And that's the beauty of a Lakehouse being able to use all these tools so that way I can satisfy whatever use case I want because again, no one tool is gonna satisfy all my use cases.

It's one of those challenges that the Lakehouse so solves Nesti helping us fill a lot of those components spaces when it came to data ops so we can make sure that we, you know, make the process of having good data, uh, quicker, faster, and less manual. And then the third QR code is another one that shows you an example of how a lot of that would work with, with Mio Arctic, which is again, just a, a cloud managed, uh, nessi catalog, uh, service that also includes automated table optimization from Mio. So if you wanna learn more, uh, hit up those QR codes and read those articles again.

Uh, my name is Alex Merced again. You guys have a full day of a lot of great talks today at DA Data Ops day. So I hope you guys enjoyed, um, leave any questions in the chat box.

I should be there, uh, answering your questions. And again, feel free to reach out to me on Twitter at am data lakehouse or follow me on LinkedIn and subscribe to any of these podcasts to hear more what I have to say. 'cause I have a lot to talk about when it comes to, uh, especially, particularly versioning on the data lakehouse.

I have a lot of thoughts on it and just the data lakehouse in general. So check it out. I hope that keep in touch.

I hope to see you around. Have a great day, and I enjoy the coming presentations here at DataOps Day.