Vikram Chennai on Building a Strong Data Engineering Foundation for AI Success

Hello and welcome to the latest edition of the Techstrong AI Leadership Insights series. I'm host Mike Zu. Today we're with Vikram Chennai, who's CEO for Ardent ai.

And we're having a chat about, well, data engineering 'cause it does drive all things ai. Vikram, welcome to show, Nice to be here. Data engineering's as a concept has been around long before we started this whole AI push and we didn't have enough data engineers then, but now it seems like every one of these AI models and applications is highly dependent upon getting the right data to the right place at the right time.

And to be honest, we haven't been very good at that historically. So I, I see all these projects and I see a lot of experiments and I see a lot of things not quite working and I gotta sneaking suspicion that it all comes back to data engineering or am I just hallucinating myself? No, you're absolutely right.

Uh, data engineering, at least the way I see it, is the core of pretty much every product that we use today. For example, you use Uber to get around, none of that functionality works without the data that actually sort of hydrates your app. And so if you don't have data engineering, pretty much, not even just with ai but with almost any application that most of us use on a day-to-day basis, like they just don't function.

And so solving this issue with data engineering and essentially allowing data teams to have enough scalability to actually meet demand, actually build products at the pace that they would want to, it not only allows, you know, new AI products to be built, but essentially allows almost every company in the world to just move faster. And fundamental way Also, also seems like there's a lot more finesse to this than people give credit to. It's not simply about loading massive amounts of data and exposing that to an AI model.

From what I hear from folks who hear the phrase context engineering a lot, which seems to be an extension of data engineering in my mind where I need to not throw too much data that's irrelevant at the AI model, but I gotta make sure that there's enough data with enough context so that I get some output that's meaningful. And this seems to require a fair amount of nuance and balance and skill. No, Yes, a hundred percent.

Um, and especially with with data engineering problems, you're obviously not dealing with small amounts of information, right? If you are serving an enterprise that has thousands and thousands of tables at minimum, there's no way you can just feed that into a language model and tell it, okay, well look at all my tables and go build this new data pipeline for me or go modify my schema. That just doesn't work.

And so you have to build a significantly more intelligent solution that can almost find the information that it needs just in time. So it says, okay, I'm gonna look at these three tables, come up with a conclusion and then move on to the next two, whatever, whatever, until let's say you then can modify a schema that can make sure that it doesn't break anything downstream or upstream. And it sort of understands everything as a whole without sort of overwhelming the system.

Where do I go find data engineers that have that level of skill or knowledge? 'cause it was hard enough to find them before ai, so, and it doesn't seem to me like there's more of them. So is this kind of what's holding up a lot of these projects in the first place?

Yeah, a hundred percent. If you had to go find them, um, I would say the first thing you would do is struggle a lot and then you'd have to pay them a lot of money. And then only after you do those two things will you be able to find really good data engineers.

They're usually sitting at Netflix or Meta or these companies with massive scale data problems and they're getting paid very, very well. And so for almost anyone else in the industry that's competing over this really, really valuable talent, it's near impossible to get both the level of talent that you want and the scalability. So you're just at this sort of crossroads, you're like, what do I do?

I have this data, I need a process, my application depends on it, guess I'll do it myself. And then you, you know, you sort of create this amount of landmines in your product, right? You don't, your pipeline isn't optimized to scale and so you quadruple the amount of users you have and suddenly your pipelines are breaking and users aren't happy.

And so now you have a huge mission critical fire that you have to put out versus if you had sort of had that knowledge baked in and that scalability from day one, then you could take a very different approach. You wouldn't have to worry about a scaling challenge, you wouldn't have to worry if this pipeline will handle, you know, significantly more volume or you want to add a new data set. You can just do that and it's as easy as just asking for it.

Does that mean therefore I don't need data engineers to the same degree? Can I just use mere mortals to kind of build these applications? Or where does this all go?

I would say the way we think about it is that you would probably still want data engineers on your team. They understand both the business side and they understand the actual implementation side. And so when they're operating a tool, just like you have tools like Cursor or Devon from cognition, the best people to use it are the people that understand the system well and therefore can distill requirements down in a way that makes it very easy for agents to sort of operate on that end to end.

So for example, in our domain, there's multiple ways to write a data pipeline, right? Even if you try to optimize it, maybe you don't want that level of optimization. For example, if you're using something like Spark, which is a massive scale parallel processing engine, there are startup times to that.

So if you want your pipeline to run really, really, really fast at humongous scale, you should be using Spark. But if you want it to run really, really fast with less data, you shouldn't be using it. And technically, you know, both solutions are correct to sort of the query of I want this data pipeline built, right?

That moves data from let's say point A to point B, it's an ETL, it's a batch ETL that's valid. Like it's a valid solution in both cases. But a data engineer would understand the nuance of what they'd actually want to do there and they can make those intelligent design choices and then use a tool like ours to simply fan out their expertise.

So now you can think of every data engineers worth 10 20 data engineers, which allows massive scalability for your organization. But then you also sort of retain that, like that brain and understanding of how this company works and what we care about. So I need somebody who knows more than say your average IT administrator who knows how to deal with an ETL tool.

But I also need somebody who if I do get them, I can scale them so that they can work on 10 projects simultaneously. Otherwise we're just gonna be backing these things up and the, the number of AI projects that are, you know, waiting to be done will be 10 times bigger than the ones we've actually done. Yeah, exactly.

And a very common problem that data engineering teams have right now is they don't have enough people. Therefore the only things that get solved are the mission critical things that maybe, you know, you have a CEO that's looking at an analytics dashboard, that dashboard is powered by some, uh, some data and the data engineers are in charge of delivering that data so that the analytics dashboard can pick it up and then that data pipeline breaks and therefore there's no data in this analytics dashboard that the CEO is using to make all of his decisions. That takes first priority, but there's like 15 other things that everyone on the team would like to do that just never ever get done, right?

And so unless we find a fundamentally different way to approach it, it's not just AI products or project, it's just every single thing that you would need data for in a company that isn't absolutely hair on fire just gets left behind. And so you just are looking at almost like keeping the ship afloat rather than actually growing and actually being competitive and actually being a company that's looking to win, right? Because you just don't have the resources to do it.

But with tools like this and the amount of leverage that it gives teams, you suddenly can flip the script, you can start looking at the other things. You can be a lot more aggressive to go chase new opportunities and new product lines and, and whatever you wanna build because now you're not bandwidth constrained by we only have three data engineers and there's no way that they could possibly keep up. I also feel like we're pressing people in the service to do data engineering who don't have much expertise in that space.

So they tend to be developers or data scientists who, you know, are now working on plumbing issues rather than models. And even IT admins get thrown into the mix. So as part of our issue now is, you know, a lot of these folks are, um, being handed fairly low level tools and told to accomplish something and they're learning on the jobs.

So maybe we need to just make that easier. Yeah, I a hundred percent agree. And I also think that, you know, when it comes to something like data engineering, what you really want to develop is really deep expertise, right?

And so we can allow these people to sort of gain their scalability, but I also think that with uh, data engineering teams and empowering them like this, people can actually work on the things that they're sort of skilled in already, right? They can develop a really deep expertise in let's say the machine learning models instead of going and fishing for the data that's not present so that they could go build their risk model or, or whatever they're trying to model or their next generation of LLMs or something like that. The problem is that most of those the time it's like, okay, I'm supposed to be training this model, but I don't have the data, so what am I gonna do?

There's no way to train this model. So now I'm gonna go fish for data. And so I think something like this, well yes, allow those people should they choose to, to go and sort of fish for the data, but they don't have to now they can just sort of deploy the task off.

But I think more importantly for the data team, it actually allows them to properly cover the surface area within the business that they need to. And I think that's actually what we see is that because they can't do that because there's just not enough bandwidth, everyone else has to sort of pitch in and help, right? And then you've created sort of this unhealthy mix where it is good that people understand what's happening with data engineering, but you want the people that really get it to be on the job.

'cause they can finish it a lot faster. They understand the nuance and everyone benefits from that. There you go.

Will we at some point use AI to help engineer the data itself for the AI applications? It feels like a lot of the work that we are doing is plumbing and maybe there's an opportunity to throw AI agents at that task so that it becomes easier to build the AI apps themselves. Oh, a hundred percent.

A hundred percent. Like the fundamental reason actually that language models even work to the degree they do today is because someone figured out that if we just scale compute and scale data, it just gets better. It just gets a lot better.

And so the number one task that they had to do was pipe in a lot of data, make sure it's formatted correctly, it arrives at the right time, it has the right distributions that we expect, it's just plumbing work before they can actually get to training this, this massive data set. And that's actually, so the reason that I started this company, my background is in model training and machine learning. And one of the biggest problems that we had at the previous company that I built is that we had to build these data pipelines to pull in all of this data and we just had to do that over and over and I was spending 80% of my time doing that instead of delivering a product to users and using that data for what we actually wanted to do.

And so that's, that's exactly it, right? It's like all of these AI applications or products that people wanna build are powered by data. That's the fundamental basis of why they work.

And so if we can get away from sort of plumbing problems, we can start doing the stuff that actually matters for companies instead of just sort of firefighting at the lowest level. What's your best advice then to folks? I think many of them have gone down the AI path and you encounter these issues pretty quickly.

So what are you seeing the, the folks who are kind of get it doing that others are struggling with? I think there are two main things. One, they're using specialized tools for whatever they're trying to do.

So for example, tools like Cursor are very, very good for general software engineering, right? They work extremely well for that. And so you should be using a tool like that instead of maybe copy pasting to chat GBT, right?

It doesn't have context of your code base, it doesn't understand how your files fit together or your functions. Therefore when you just sort of copy paste back and forth, it just doesn't work as well. And it's the same thing for something like data engineering and what we've built, we focus specifically on understanding like table relationships.

So foreign key relationships, we understand schemas, we understand data lineage. So we essentially understand how does your entire data ecosystem fit together and sort of what is dependent on what. So then when a tool like ours goes and makes a change, it is able to actually do that surgically.

It knows, okay, if I change this, these are the tables that are dependent on it. This is the application that's going to read from this table and this is what they expect upstream. This is what's happening.

And with all of that context and how everything fits together, then it applies a change. And so again, this applies to sort of any tool and people have seen a lot more success like that because the tools are built for the job instead of taking sort of a general tool and shoehorning it into whatever you're trying to do that just doesn't produce very good results because language models are just giant probability distribution engines, right? They've just been fed tests, they've been fed code, they've been fed everything.

And so your job actually is to almost sample out of that massive amount of learning. I want this point, this is what I want and that's what you're trying to extract. But if you don't give it the right inputs, it's gonna have a really hard time trying to figure out exactly what that point is that you're looking for.

But the more you give it, the more specific it gets. And it says, okay, well I wanna pick the, this part of the, the distribution and therefore it generates a response that's a lot better. It works a lot more easily into the things that you want to do.

It's just built for the job. And to your point about that, everybody's talking about AI agents, but it seems like that becomes the forefront of a data engineering challenge itself because that's really needs the right data and mm-hmm to accomplish a specific task. And it seems to me more people are gonna distill foundational models to create these smaller language models that will drive those AI agents, but all of that requires some data engineering or am I just imagining that as well?

No, I, I think you're absolutely right. Uh, that any advancement we we make with AI is powered by data, right? Actually, the way that they distill models is they have it just generate a bunch of data from the larger model and then retrain smaller models to sort of fit the distribution of those of those models.

And there's a bunch of other techniques they add, but that's one of the main ways they do it, right? So they essentially teach smaller models to get as good as as bigger models. Um, so data really is everywhere in, in this game.

It is actually the reason that the game even exists. Um, so I think, you know, sort of like looking forward to where anything or, or this industry sort of goes like that is the core. It's just if you don't have the data, you don't have anything.

And if you're slower on sort of your iteration cycle with the data, so maybe you have it, but okay, we need to move it from one place to another. It's terabyte scale. So we can't just write a simple Python script that takes too long and needs to parallelize well, but you need to understand and handle all these sort of contingencies before you even move forward, right?

And so it sort of is, in my opinion, the great bottleneck between where we are now and sort of moving forward, especially with ai. All right folks, you heard it here. Turns out much like any building, if you don't have the right scaffolding and the plumbing and the foundation, the rest of it doesn't much matter.

So that's the same thing that's going on here with or AI and data engineering. Hey buddy, thanks for being on the show. Yeah, appreciate it.

All right. And thank you all for watching the latest edition of the Techstrong AI Leadership Insight series. You can find this episode and others on our website.

We invite you to check all those out. Until then, we'll see you next time.