Data Lakehouses: The Foundation for the Next Era of AI – Sendur Sellakumar, Dremio

This is Textron tv. Hey guys, thanks for the throw. We are here with Sdo Kumar, who's the newly appointed c e O for Dremio, and we're talking about data lake houses and how the world is gonna evolve because, well, we got more data than we know what to do with sender.

Welcome to the show. Thanks for having me, Mike. I appreciate it.

Alright, Maybe you could give us a little bit of perspective, but back in the day we all kind of got excited about Hadoop and then we went off and we built these big data warehouses and then we were gonna build data lakes and a lot of them turned into data swamps. And now we're all trying to figure out what the end of the horse is up as we combine the attributes of warehouses and lakes to create something better. But where are we as an industry we're, it seems like we kind of made a detour in this journey and maybe we're getting back in the main road.

Yeah, yeah. You know, it's interesting. So as you probably saw from a history I spent about, you know, a decade at Splunk and I saw this journey, uh, certainly of the Hadoop and big data world.

I do think we're getting back on track, uh, as you're spot on there. I would say the big changes, you know, when we started this journey on data Lakes, it was very much a technology first conversation. Let me deploy Hadoop, let me deploy.

We're gonna put a data lake, we have a data lake project. And too often that was the sort of tip of the spear. Well, the tip of the spear should really have been, we need this for marketing.

If marketing needs this data to analyze, do a customer 360 and that's the principle driver. So the technology choices, choices be what they are, but ultimately that's the goal. And I think we inverted that early on in the, in sort of the big data world because we got excited about these big data technologies, all great technologies that came out of these, these sort of large technology, um, uh, houses that were deploying Apache projects.

And I think over time we started to see customers get mature about, about what they think is the, the right value prop business outcome that they want. And now we're seeing customers look at that seriously and say, yeah, I wanna do SQL on this data because I need to make business decisions on the data. Which is I think the trend you're seeing.

We started, we saw what's starting with Impala, but ultimately I think what we're seeing with Lakehouse is really a more maturation of that to to enterprise level workloads. Yeah, of course every new manager of a baseball team gets asked the same question, so I'm gonna ask you, but it's Hey coach. Yeah, coach, what do you like about your new team?

Oh, yeah. Um, well I would say, you know, I research this prior to joining the company and I think it's in validated further. Number one is we have a great customer base.

We have a great team that supports a great customer base. Uh, and I would say that the customers are excited and, and eager to continue growing with us, which is awesome. Uh, awesome to see one, because you're providing value, but also they see broader opportunity within their cu within their, uh, their own ecosystem, which I love.

Like I get up every day saying, am I delivering value to customers and are our employees happy doing that? And I mean that's sort of the, I think the oxygen if you will. So that's very exciting to me.

The other is we've doing a lot of innovative things on the product side. You know, know, as you can probably tell from where we started the conversation, I'm very much customer outcome oriented. And so we're doing a lot of things on product, but where I think that really comes into play is we believe we're gonna achieve some pretty significant T C O benefits for customers, use benefits, performance benefits, talk ad nauseum about that.

But I'm excited about those two things and eager to see our customers get value from it. Of course, everybody and his brother is talking about AI these days. Ah, Yes.

Will AI be kind of be the best thing that ever happened in data lakehouse? Uh, I think it will be core fundamental use case, uh, of, of it, right? You know, back to kind of AI I think is interesting, but what's really more interesting is can we accelerate decision making for enterprises critical business decisions through ai?

And I think that answer is emphatically yes. And that's where you're seeing a lot of vertical applications. Pick your, you know, your customer support portal, pick your, uh, your marketing tool, adopting those AI tools.

And we wanna be an engine that allows customers to be able to take advantage of the data. The core of all AI is data at the end of the day. And so what we do see customers do is say, yes, I've got a business need where AI can get value.

I'm gonna get a set set of open source tools that can, that can leverage, that can essentially build models, manage L L m, et cetera. And I need an underlying engine that can give, that can furnish data to me. And that's where we come in.

We furnish that data to those, to those AI workflows. So I do think it'll be important, but I think we're gonna see it in very concrete ways insofar as translating to business value for customers. Alright.

When you and I are mistaken, we're just flat out wrong when a machine is mistaken and apparently is just the hallucination. As you kind of think that through though, do you think that there's gonna be a greater appreciation for the quality of the data that we're putting together? Because it seems like that has a direct impact on the outcome of the AI model?

Oh yes, for sure. You know, in fact, one of, uh, yes, the short answer, I think the quality of data is important. How you generated the model, how did it get to the decision is important.

That sort of visibility is gonna be a critical piece of the puzzle. Um, I'll give you an example. You know, uh, one of, one of my friends is a radiologist and she is looking at, uh, using AI to assess, assess, um, you know, diagnose medical images, right?

That's not formally approved yet by the F D A, but companies or startups are starting to push that model. But then who's liable when something's wrong? How do you what model what, what data generated that model to then create that outcome?

Those are all questions. We as a society need to get better and forming for ourselves. And I think we're gonna play a part in that, but ultimately it's gonna be the, the customers that decide how we can do this effectively.

And that is still something we're maturing as an industry for sure. Mike. There are a lot of pieces to this puzzle.

On the one hand we have the people who are building the data lakes, then we have the data engineers that are driving data into the data lakes, and then we have the data science teams creating the ML lops platforms that consume the data lakes. Yeah. And finally, but not at least are all the DevOps folks who are trying to embed those models into something that looks like an application.

Yeah. Is there some way to streamline all this? We seem to have a lot of workflows and processes involved in this.

Yeah. But how do you kind of think about this? Yeah, you know, I'm a big believer in self-service.

You know, one of the reasons Tableau and, excuse me, power BI took off so well, and frankly why people love using chat G P T is because it has abstracted all the problems away and gives you a much simpler way to self-serve your, the outcome that you seek. And so I do believe the more we can take those parts of that chain and push a lot of it to be one self-service, but two of course automated the better. So in the world of, you know, in the world of sort of traditional bi, you see the conversion of self-service, I think that'll happen with ET t L as well over time.

Um, and I think the, the generation of of L L M, so using LLMs, that workflow will start to become more and more self-service where the business users will start to be able to do more and more of that, of that data pipeline, if you will. And certainly with our customers, when we see, when we think about Dremio, we see customers say, yeah, I don't need to do a separate e t l drop in the data and let me use, let me actually use a, uh, a, a lakehouse engine to be able to do E T L, which you're currently seeing with other solutions, right? That trend of D B T coming in and taking over the chance form side of things, we're gonna see that sort of use case happen across, I think, the AI chain as well.

Hmm. So what's your best advice to organizations as they kind of try to get their arms around this whole thing? Because, um, frankly it's a little intimidating when you look at all the peace parts.

Yeah, of for sure. I, I would say the first thing is have a true North star of what business outcome you're seeking. Uh, that's probably the, the number one thing I would say.

And people lose sight of that sometimes. 'cause you think about, oh, this technology, that technology, et cetera, and the technology is an enabler for that outcome. So I would say customers that have the best, get the best value out of any technology skill set and certainly out of Dremio, is to be able to say, I'm solving for this cust this problem we have in our business.

Here's the underlying data I need, and then what's the technology tool chain to make that happen most efficiently, cost effectively, and with the fastest time to value or fastest time to insight, I always say. And so those things I think are critical. So that's how I would start the, the conversation with customers and that has proven to work well.

And, and when you think about what outcome we're seeking, Do you think organizations will be able to standardize on one data lake or are they gonna have multiple data lakes and we're gonna have to figure out what the interoperability is between them? I think there will always be multiple data lakes. I hate to tell you that.

I think it's true. Why? Because organizations make independent decisions, right?

When you look at a big organization, the insurance department makes a different decision than the finance department makes a different decision than, than the folks in the field. Why? Because they have context for their part of the business.

And so I think we'll see independent decision making always be there in enterprises. And as a result, I think we will see, continue to see bifurcation of that. I do believe though, that there are technologies like jamir that can bring those worlds together.

We do massive federated querying across semantic, across a semantic layer, which allows you to bring those individual silos, allow the nimbleness that those organizations want, but then connect the dots together through a common semantic layer that GIA provides. And that is an example of where we can say we wanna meet your customer where you are versus changing the way you do business, which is to put everything into one place, which is not always, always optimal for customers. We Hear a lot about digital transformation initiatives, of course.

Do you think that there's a better appreciation for the value of data? I mean, honestly, I've been around long enough that I can remember when people treated data like it was a burden to be born and a cost to be minimized. And have we changed our minds about what data's all about?

I think people have gotten more mature about what value you can get from data and what you can't. Like, it's not the nirvana. You still have to make decisioning, you have to make judgment calls on the data, and you have to scrutinize the data.

You know, how often have you looked at a dashboard or a chart and you said, I have three questions that I now have to ask on top of the chart I just saw. Uh, and so you want, you want solutions that can get you answers quickly and allow you to make independent decisioning, which is why, again, back to self-service and ease of use is so important for these tools because you want folks to make independent decisions. So I do think we've gotten more mature and more practical about these data.

There are also a lot of privacy regulations starting to run around and other signals from governments around the world saying that they will hold you accountable for how data is used. Um, how will that play out? I mean, are we gonna see governance and compliance stuff start to merge with data management?

We've been talking about it for a long time, but it, are we on the cusp? Yeah. I, I don't know that there's a, a, a, uh, tipping point, but I would certainly say it will be the, it'll be there for, for some time.

I think it will be the sort of foundation of the, the cost of doing business, but also, frankly, you know, we need, you know, every organization wants to be a good steward of customer data. We wanna be a of good stewards of our customers data in the same way that our customers wanna be good stewards of, of, uh, of their end customer's data. So being a good steward means you've gotta do some of those foundational things.

And I think a lot of that align with the compliance and, and data governance world. I do believe, you know, again, back to sort of a little bit about Dremio, we do enable you to sort of on that semantic layer, we talked about separate governance so that you can actually say, Mike, you get access to this data, but not this data and this person gets access to this data and that data and the complexity of that, I do believe will increase as the regulatory environments change and, and customers and consumers demand more of this, of the levels of stewardship that organizations need to provide as incumbent on, on solutions like reio, uh, to be able to make that easy for customers to manage, for our customers to manage. So I do think it'll converge and I think it'll up the bar, which I think is frankly a good thing for the industry, At least in my experience.

Data management has been, uh, rough in a lot of organizations. Very few of them that I know of would get a good housekeeping seal of approval for the way they manage data. But, um, we see a lot of talk about using data to drive ai, but I wonder, are we gonna use AI to improve data management and maybe save us from ourselves?

Oh yeah. I, I think that will certainly be the case. You know, like you use AI to make vertical decisions, you will likely use AI to, you know, inform how you query data.

So for example, we have actually leveraged, uh, generative AI within our system to be able to query, uh, query data through raw text or, or actually basically asking questions in normal natural language versus writing SQL, for example. Again, that pushes more self-service to that end customer. So then you're able to broaden the use of data and access the data.

Um, but I think it's really important. I do believe M ML will also be used to things like optimize E T l, lower cost optimize, how we, how we do, uh, uh, how we do, uh, summaries and sort of materializations of data. So all that will be will be thing to think a lot of every industry will use some element of this to sort of optimize usage of, of what they sell.

Like it'll be part, part part part of the course of what we do. Are we gonna save as much data because we're kind of drowning in it and there's lots of different variations of it, but if we can get to the abstracts and the analytics, maybe we just save those and not necessarily save every piece of data that ever existed. There's a lot of data hoarders out there.

Can we kind of be smarter about it? Yeah, I think we can, you know, storage is extremely cheap, particularly when you look at, uh, the things that the cloud providers have, have already provided. Um, I think there is a, a level of aggregation that makes sense.

And, you know, back to where we started the conversation is everything we do has to be based on what, what business or customer outcome are you solving for? And so, you know, hoarding data where there isn't a real customer outcome or business outcome is, is, is difficult to justify. And I think organizations are getting better at managing that, to be honest.

Um, I do think there are tools that we can use. So for example, one of the things we do at Dremio is a capability called data as code, where you can take data and version the data. So you don't need to co make five copies of data.

You're able to just version it and alter it from there. So you're able to maintain the old copy as well as the new copy, but then actually not maintain two full copies of data, as an example. So sort of deduplication, but in the context of actual business workflow, big piece of what we do.

So I do believe the, the concerns around keeping lots of data will become a lot easier to manage as technologies like Dremio enable you to sort of manage and maintain data at lower cost. Who's in charge of all this these days? And I'm asking the question because, you know, back in the day, the IT people, they didn't think too much about who created the data and what it meant to the business.

They were just managing and processing it. And the business side was like, well, we're just creating data and you guys magically manage it. So have, yeah.

Have we matured and somebody's gonna, you know, be the adult in their room about this? I do think so. I, I think what we're, we're already starting to see with customers is that the data platform team who typically doesn't have context for the data, what they know is the technologies that manage the data, but they actually don't have context for the data, mostly because they're not in the business unit that's running that's sort of managing or owning of that data.

So often what we see is in the past, we would rely on the, on the data engineering teams to actually have, have context for the data so they could be effective for the end users, but through things like self-service, certainly in end tools like D B T or or TAB or Tableau, others, and through engines that are more self-service like Mio, we're able to bring that user closer to the, to the sort of earlier in the data pipeline because they have context for that data. But the role of a data engineer, I think becomes that much more interesting because now you're able to serve more customers in your organization effectively, and you don't need to be a bottleneck. You actually can focus on higher value things and let the customer self-service in a, in a much bigger way.

I do think that that's a big theme we're gonna see change in the industry. What's that one thing you see? You've been around the block a few times, it just Yeah.

Makes you shake your head and go, folks we're better than this. Uh, uh, I think there, you know, I'm excited about all the, all the hype around ai. I think AI is pretty gonna be a, a game changer for industry, but you sort of are going through that classic, you know, sort of hype cycle down to down to sort of reality.

And I think we're starting to see some good practical applications of LLMs, uh, which is great. I'm starting to see vertical applications come out with that. So I'm, uh, I think, you know, we, the, the tech industry always does this.

It's sort of like fashion model of a lot of hyper on a thing and then it sort of stabilize through really practical things. Uh, I'm waiting for that for LLMs and we're already starting to see examples of that, uh, where I think those that are in the domain spaces, uh, of markets that can leverage lms, we'll start to use them embedded in their workflows. And I think that's where we'll see the real value come up.

And so I'm excited about that. All right, folks. Well, you heard it here.

Things change, but they, they don't change. It starts with the data and ends with the data. All the stuff in the middle is what we gotta do to get some value out of it.

Send our thank you for being on the show. Thank you Mike. Much appreciate it.

And back to you guys in the studio.