The Art and Science of Data Engineering with Pentaho’s Maggie Laird

Hello and welcome to the latest edition of the Techron AI video series. I'm your host, Mike Bazar today with Maggie Laird, who's president of the Pentaho Business Unit for Hitachi Ventera. And we're gonna be having a little chat about well data repatriation in the age of ai.

Maggie, welcome to show. Thank you for having me. Good to be here.

When I was much younger, people used to tell me nothing good can happen when you move data. And yet here we are talking about moving data. So from your perspective, what is driving all this activity where folks are, seem to be moving, um, workloads and data, not just from the cloud back to on-premise, but sometimes the other way as well.

Um, I think there's more data moving now than I can remember in my entire years of doing this. Yeah, no, it's certainly an exciting time for data. I think gen AI have brought the core data management challenges to the front fold.

Right now, from my perspective, what we're seeing with our customers is, is the gen AI adoption experimentation, right? Companies are trying to figure out the right spot for the data, right? So, so things are moving around.

Um, some of it's coming on-prem. They want to bring the data, bring AI to the data, right? And be able to control it in a very, um, measured way, right?

And then other companies are, are try, once they know what they need, they're putting in the cloud and run and operating it. So it's, it's really this, uh, from my perspective, some experimentation, um, around how to most effectively and economically, um, take advantage of the VA opportunity that is in front of us. So a lot of folks out there trying to figure out what, what fits.

Some of this also feels like to me, to your point, they're building the first rev of the AI models in the cloud because that's cost effective. But when it comes to deployment, uh, and the inference engines, that needs to be either on premise somewhere or even at the edge. And now I've gotta take all the data that I use to create the model and get it out to the inference engine.

So is that part of this conversation? I think absolutely. So run, operate is what you're talking about, the AI ops.

Um, so once we're ready, once we know what we're doing, once we're convinced of the value, once we know the quality, right? One, once all of that is known, um, how do we do that at scale, right? And what does that scale mean, right?

And I think that's where the cost equation is coming in. Um, being able to be predictable around that cost and for some companies that are running are gonna be running a lot of volumes through this. That's on-prem control, visibility, predictability, um, again, is, is is coming to the forefront and being able to really wanna have those guardrails and understand, um, how, how it's gonna work in a very controlled environment.

Um, I think you're absolutely right. Aren't we using traditional kind of batch oriented processes to move that data? Or are we shifting to more of these streaming platforms?

And it seems to me a lot of this conversation is just trying to make sure the right data's in the right place at the right time. Yeah, absolutely. That's the name of the game.

So I think it's both, right? We're seeing real-time streaming data come in, we're seeing kind of the data at rest as well, because to unlock the value of, of a lot of this, you need all of that kind of moving. Um, so, so then it, it's about the core elements of data management.

Do I have the right data? Right? And we don't need to take everything, right?

So making sure that that right data, um, that has the right quality to that, I think that's where folks are learning a lot about the quality of their data because they put it in here and they're getting outcomes they didn't anticipate that doesn't look right and they're having to go back into the core data itself to say, Hey, did we bring the right data in because this isn't, you know, what we were expecting? So that, that real inspection of, of that source data again, is, is, is becoming real. So again, what did we bring in?

What is the quality of it? Um, do we understand it? And, and again, do we trust it to be ultimately making decisions now?

And it's, you know, at a, at a much different, um, um, scale than before. We have a tendency to be critical of the AI models 'cause they will hallucinate. But how much do you think the real core issue is just that the quality of the data that we're exposing to the models is somewhat flawed.

And, um, you know, issues that we've kind of sort of known about for decades are coming home the roost. That's a big part of it. Um, that's what we hear from our customers who have maybe jumped farther fast to, to try to get started to have a proof of value.

Um, and then they get caught back into the core challenge of data management and then they've gotta go back and start again because again, they, um, you know, been able to bubble gum and bandaid their way. Some companies have around that, these core data challenges. But AI is exposing that risk.

There is too much risk from not having, um, the quality right and the right data at the beginning of the process. So it doesn't, it, it does become a roadblock to production, right? Um, you can learn some things with some sample data, right?

And you can manually clean some of this in a, in a, in a very experimental way. But when it's time to go big, right? And to put this into production, you've gotta have, you know, that scalable, um, an underlying quality of that core data, um, you know, understanding that and, and ready to go.

Is there a greater appreciation for data management? And we've been talking about this now for years and years and years, but I felt it was always around the structured data and the unstructured and semi-structured data we kinda overlooked a little bit. Let's say very few people would get a good housekeeping seal of approval for the way they manage that data.

Um, so is all of this bringing, you know, a focus back on the fundamentals of data management? Yeah, now I think that's exactly what this is bringing to the, the forefront. There's more of a burning platform, right?

It's always been a good idea to have a good quality data, right? And to, to maintain it well in a cost efficient way. That's good hygiene.

But you know, today's world, it's, it's actually required to be able to do, um, the innovation that folks wanna do on, on that data. So, so it's, it's back to reality, back to the basics, get it right, do the work, um, because the work's gonna be exposed if it's not, um, if it's not done in the right way and in a way that can be, can be scalable. So I think all those, all those cheap data officers that, you know, getting an analy getting more of an AI remit now, um, it does kind of go back to the core.

Um, and so, so again, it it's, it's a good, if you, if you didn't take the steps at the beginning of a process to, to get it right now is the time, but the key messages, you're not gonna go very fa far. You may go fast, but you're not gonna go far until you, you know, isolate and really do the work at the foundation. And I think that's, you know, where Penta is really focused is helping companies get that foundation, right?

Because you gotta, you gotta deal with it now before you are able to take full advantage of, of the innovation off the backs of ai. You know, I was just thinking about this, but back in the day, the data more often than not was managed by somebody who we generally refer to as the storage administrator or something who was an administrator. And now every time I turn around, um, there are data engineers that people want.

So has the nature of the profession changed and what's required? Because last time I checked anybody who was called an engineer costs a lot more than anybody called an administrator. Yeah.

I mean, we're seeing, um, you know, certainly the, the expansion of the need to have higher level data skills for sure, right? And also some of these roles are starting to blend inside of the, the IT organization as well. And then Mikey didn't even mention the, the AgTech workforce that may be coming online as well.

So, so here we are, and you, you kind of step back and you look at how this will be managed, right? And then who needs to manage it, what skillset they would need, and really what, you know, they're, we're, we're blending a lot of, um, automation inside of, of, of these skills as well. So, so I absolutely think that under that, a deeper understanding of this, of the engineering of the data is required, um, but also again, what can we look to and how can we make that data as most prepared for, you know, again, an adjunctive workforce to be able to help and assist, um, ultimately in the, the management of all this data moving forward.

And so to your point then, can we expect to be using AI to manage the data that we need to train and build the ai? Yeah, I think that's where we, where at least we see it headed, right? Um, it's gonna, we've gotta take the right steps to get there.

Um, but it really is infusing AI into these core data data processes to make them more efficient. Um, you know, you, again, we gotta step through it with the right guardrails to understand how that's happening and keep those humans in the loop as, as we begin the journey. But absolutely having, um, those automated ways to build brag pipelines, right?

How to, to do some of these things that again, took a very sophisticated, you know, skillset to do. You know, again, this is kind of the work here at pental is how do we get that into more of a data citizen world to be able to do more? Um, you know, with, with all that you have again, um, in a, in a user in a way that's very, um, can brief the outcomes but doesn't require all that, um, you know, all those advanced skill sets that, you know, that's where the, that's where the automation and the AI comes in and, and, and then you get to get the results a lot faster.

Is all this gonna lead to some need for increased transparency into what data was used to train what model and what was used to where to run it in a way that is much deeper and maybe more profound than we're used to? Yeah, I think absolutely. And you, you can see this with the AI regulations popping up, right?

It really is about transparency, transparency. Um, and so, so yeah, so companies are, you know, honestly having to happen to pay attention in terms of how, how they build, um, you know, what are those regulations coming, coming through to make sure that, again, they're, they're building the right hooks into the processes so that they can see, um, the data that's, that's coming in. Um, what it was there to do.

Was there any data that was injected in there that shouldn't be in there? So again, companies are gonna, if they don't already have, have tools, um, and systems like data catalogs that help to provide guardrails, to provide visibility. 'cause ultimately if you've got a regulation around something and you wanna be compliant, you've gotta be able to produce, um, the report or the valid, the justification that says, this is where we're getting that.

So, um, companies need to be mindful of that as they are bringing, especially as they're bringing things into production. Did they, are they setting this up, this right this up for that transparency that ultimately, um, even if the regulators aren't out there asking for it, I guarantee that the business users are gonna wanna know where this come from. Are the C-level executives more conscious of these issues than they have been in the past?

Where I think they tended to view this as some sort of, you know, lower level IT issue. But is do they understand now the value of the data and exactly the use cases for it and how it kind of drives a process? Is, is the conversation getting elevated?

Yeah, absolutely. I'd say, I'd say C-suite and board level as well. I think that's where, um, it is this kind of renaissance room for, for data.

Um, you know, having ai, having CEOs talk about AI processes, I mean, who would've thought about that 10 years ago, right? Um, and we, there is, you know, when, when I'm out there meeting with, with our customers leadership team that, that they are fluent, um, on the topics right now, the details, but they're asking the questions, right? And they understand that this, this is, you know, a source of risk, it's a source of innovation, you know, and so tho from, from those two pieces, then as they start to drill down and ask their organizations for, you know, plans and, and, um, you know, and, and proof of concepts and, and really point of views and governance, it, it's, I'm, it's, it's great to see from the data world this elevation and it, it matters.

Like this is a, this is a big deal and it's, it's very core strategically to company's strategy, right? Um, as they move forward. If you don't have AI in a strategy, I think your board's gonna be asking you why.

And that's a very different situation than we were in three, five years ago. As part of that conversation, do you think that data management and data security is gonna converge more than it has in the past? And there'll be more focus on how to management of data in a secure fashion?

Because I think those two things have been somewhat orthogonal for many years. Yeah, exactly. I mean, depending on the architecture of where the data was sitting and who was controlling it, right?

That we are seeing a lot of that merge, right? And the access, the access to the data, right? And the secure security of that access, those two things are, you know, front and center because, you know, I think, uh, in our consumer lives, we're all starting to realize, you know, when you are inter interacting with these models, you're, you're giving, you're getting something, but you're giving up something as well.

So the data exchange, um, around that, and you bring that into the enterprise environment, you've gotta be very, very careful about what you're giving to whom, and then how that's being secured in your environment, and then what, what are the open doors that you have out? So absolutely, I think the data security, privacy, access and control, it's, it is a convergence because it all has to be solved in order to, um, you know, kind of bring the right outcome at the right risk level to the business. So as you look in organizations that you work with, what are they the ones that are getting it, what are they doing right that others are not then kind of, you know, that you wish other folks would kind of crib?

Yeah, yeah, we're definitely seeing some best practices out there. Um, I think companies that are taking a strategic approach, uh, and really thinking through, they're not gonna know everything, but they, they've got a strategy behind their, their architecture, right? They've, they understand they're likely gonna be in a, a hybrid world, but then they ask the questions, what they get down to the workload level, right?

And they start to really plan around what, what needs to be where for what reason? So with the, the cl the more detail you can get into, you know, where does this data need to be to support the workloads? Then you can start to architect, you know, the right environment around your data with strategy in mind, but also cost in mind.

I think this is just one that is a red flag right now for a lot of people as they're starting to get their bills, you know, from their cloud providers or, you know, maybe even, you know, on the other side, they, they're examining their costs all the time. This the cost piece is really real. So bringing the economics into the strategy, you know, to the workload.

Um, and then also those folks that can look at their data and, and make sure they're being really optimized around the estate. 'cause this, again, it data is exploding. You mentioned the un un inter unstructured to the scene, right?

Okay, now, now we're really exploding. You gotta be smart about what you're maintaining. You don't need 50 copies of that same file, right?

So I think companies that can take that big picture, bring it down to the details, and then start to make, um, you know, the decisions from from that lens are the ones that you know, will wi are, are kind of winning right now. You, when folks, you heard it here, they say, Dave is the new oil, but what does it matter how much oil you have if you can't manage and process it? So there we are, Maggie, thanks for being on the show.

All right, thanks Mike. All right. And thank you all for watching the latest episode of the Textron AI series.

You can catch this episode and others on our website. We'd like you to check them all out. Until then, we'll see you next day.