Revolutionizing the Management of DataOps with Astronomer’s Julian LaNeve

This is Textron tv. Hello and welcome to Textron ITSM video series. I'm host Mike Ard.

Today we're with Julian Nieve, who CTO for astronomer. And we're talking about, well, the need to modernize or perhaps revolutionized ops altogether. Julian, welcome to share.

Thanks Mike. It's great to be here. This is a conversation that kind of perplexes a lot of people 'cause they'll say, well, we've been managing data forever and we've been processing data for as long as anybody can think about.

It's core to everything we do in it. What is it now about data ops that's changing the way we need to think about the way we process, manage, and work with data in general? Yeah, I think it's really about the level of operational maturity that folks need to get to today.

You're absolutely right that, you know, the industry's been working with data forever and in some shape or form, but I'd say the level of criticality on that data is only increasing over time. And it certainly feels like we're hitting an inflection point today where, you know, businesses are starting to rely more and more on data as a source of competitive differentiation, right? They collect all this data, they put it in a data warehouse or a data lake, and they start to look for ways to get value out of that data, right?

Whether it's surfacing that data back in a product or using it to make more intelligent decisions, um, or even, you know, training LLMs or, or AI models in the kind of most extreme cases today. Um, and that's kind of put data teams at the really the center of, of the map, um, because they're, you know, the, they hold the keys to the kingdom, right, for, for lack of better words in terms of how you access data. And that's a pretty big shift from I'd say the last decade or so where, you know, the software teams have been kind of the, the core of everything.

Um, and you know, if you look at what's happened in the field of software, there's this whole field of DevOps. It's kind of been created and innovated on over, you know, the last decade or two. It's really around taking the work that these software teams are doing and make them more productive, make it easier to maintain existing systems so they can focus on building out new use cases.

Um, and we're starting to see a lot of that now with, you know, the world of data where data teams today are bogged down by, um, kind of the operational overhead of maintaining all of their existing data assets and data products, so they don't have as much time, um, to build out, you know, new use cases and, and new ways of finding value. To your point, DevOps is kind of a philosophy as much as it is anything else? Is data ops essentially also a philosophy or are there multiple ways to go about data ops?

I mean, what's the coer? Yeah, absolutely. I think, you know, there are very, very similar power parallels to, to the world of DevOps where, um, it is certainly more of a philosophy than I'd say, you know, a set of tools, although there's a bunch of tooling out there that can help you, um, with it.

Um, you know, I'd say companies traditionally have had to, um, you know, treat their data teams almost as a service center, right? You go to the data team when you have an interesting question that needs to be answered by data. Um, and, you know, then the data team ends up, um, you know, dealing with these kind of ad hoc requests all day long.

And I think that, you know, the shift to data ops is really about treating your data team more like a software team, where there'll certainly be, you know, ad hoc things that come up now and again, um, but really, you know, taking the work that they're doing, um, giving them the context of kind of where the business is going and, and where the company's priorities are, um, and letting them build, you know, interesting kind of data products and, um, analysis to, to help support the business. So it, you know, it is certainly more of kind of a mind shift, mind set shift to start treating data teams more like software teams. Um, and there's absolutely tooling that, that can help with that.

So I think it's, you know, it's a little bit of both. I Feel like the, it's not just the volume of data that's increasing, but the velocity at which it is being processed is fundamentally changing, and that requires maybe this different mindset that you're describing because, um, we have to make sure the right data is in the right place at the right time, much more these days than we did when everything was kind of more of a batching it kinda processing. And if it happened overnight and it worked great, but, um, is, you know, so has the whole mission requires now, you know, somebody who really is a data engineer.

I think you're absolutely right. I'd also add that not only the data, does the data need to be there at the right time, um, but it also needs to be of high quality, which is oftentimes what, you know, these data teams struggle with the most, you know, these, these data teams have to work with, you know, external systems like different marketing tools and, and CRMs where there's a lot of manually inputted data and that, you know, data can be, um, misshaped or, you know, have, have certain anomalies part of the time. And so, you know, as a data team, if you're having to both like fight, um, against the tools that you're working with to make sure that, you know, they're always, um, you know, working and you know, they're delivering data in the, the way that you expect them to, um, and you have to go continue building out new use cases and delivering new value, um, you know, it's almost an impossible task, right?

Like our, our data team wakes up in the morning and, you know, they look at all of their pipelines, they look at kind of all of the data that was supposed to have been delivered. And you know, more often than not, there's some small problem with the data, so they have to go spend time looking into it, fixing it, rerunning pipelines, um, and, you know, trying to both maintain kinda the existing use cases and tables and dashboards and, um, data products that, that our team has built and look for ways to, to deliver new values is really tricky. And, you know, I think we're seeing this also a lot kind of in the, the world of AI where a lot of companies are, you know, putting the cart before the horse, so to speak, right?

Everyone wants a very strong AI strategy, they're looking at the companies around them, they're looking at VC investments, and it's pretty clear that you can get value from ai, right? I'd say in the first year or so since, um, chat GPT was announced, um, building differentiation in the space of AI really was about how quickly you could go take these off the shelf models, put them in your product, and as long as you could do that quicker than your competitors, that's now a reason for, for customers to come to your product over others. Um, I'd say gone are those days though, right?

Everyone has, has figured out some way to take an off the shelf LLM and either use it in their product to build differentiation or used it to make their their business more efficient. Now, that differentiation has to come from how you take these off the shelf models, whether they're open source models or commercial models, and supplement them with data that you have unique access to. Um, and so that puts again, the focus right back on these data teams who are now having to maintain these, you know, existing data products and start to be pulled into the world of ai.

Um, and you know, I'd say in the, in the net, it's great for these data teams to kind of be at the center of everything. It means that, um, there's a lot more investment being made in, in data teams and data infrastructure across a, a wide variety of organizations. But unless they have the right kind of mindset, the right tooling, um, and the right processes, it becomes very, very messy very quickly.

Well, so is it, I'm gonna force this issue. I mean, to your point about the cart before the horse, eventually we, we will figure out to turn the cart around. So do you think that in a lot of ways this whole rise of AI is pulling this data ops conversation along and maybe it's just long overdue?

Yeah, I mean, I think absolutely it is, right? Like there's, there's now an increased level of, um, scrutiny and understanding of what data teams are doing, right? When you start to rely on data teams, both as a source of kind of using data to build a data-driven business, but now, um, to also supply these AI models and LLMs with, um, data becomes a very, very core part of your business.

Um, and you know, I think if you look back five or 10 years, um, data teams, you know, have, have always been around, I'd say the use cases at the time though, were more around, um, supplying internal dashboards and analytics and reporting. Um, and that's certainly helpful. Um, you know, it helps you make data-driven decisions.

It helps you understand kind of the, the business and the markets. Um, but if you fail to do your job as a, as a data team, you know, 10 years ago you have maybe some angry exec somewhere that, you know, is frustrated that their dashboard isn't updated. Um, if a data team can't properly deliver data reliably today, um, that business loses a source of, of competitive differentiation.

Um, you know, they, they lose the ability to make data-driven decisions. Um, if they're not supplying data to, you know, these AI models in a timely manner, um, these models will become out of date and, you know, they'll give, um, potentially inaccurate answers. And trust is such a core problem with, um, LLMs and, you know, kind of generative AI more generally because, um, of how non-deterministic they are, right?

Like, if I go start using an LLM or an AI tool and it gives me a bad answer once, uh, my level of trust in that system decreases very, very quickly. Um, so I think certainly the, the kind of rise of AI has put data teams back at the, the center. Um, and that means that, you know, data teams now have to go adopt, um, whether they call it a data ops mindset or not, um, you know, they need to treat themselves more like software teams who are responsible for not only delivering a set of, you know, kind of products and use cases today, but also can do so in a way that they can continue building new things instead of spending all day maintaining the existing things.

We talk a lot about how data is core to building the AI models, but, uh, will we apply AI to the data ops itself and will that become some way that we, or maybe you're able to manage all this data at scale? Yeah, I'd say for the, for the teams that can do it well, it's gonna be very, very instrumental, but in some senses it's a, it's a double-edged sword, right? I think if you are a data engineer or data scientist or data analyst who is very intimately familiar with kind of your, your data platform, um, and you can use AI to your advantage, it's gonna make you more productive, it's gonna make you more efficient, um, and it's gonna kind of decrease the, the burden of the existing use cases with things like, um, AI assisted troubleshooting or, or pipeline offering.

On the flip side, though, AI lowers the barrier for other folks to contribute to the data platform right now, an executive somewhere, or, you know, someone from the, the marketing team as an example, can go, right, um, can go to chat GBT and ask it to generate a SQL query, um, to go analyze some data from the data platform. Um, and that's, that's a good thing because it means that I think there'll be increased participation and engagement in these data platforms. But when you start to do that at scale, right?

If everyone from the marketing team and the sales team and um, other teams start, you know, writing a bunch of pipelines and running a bunch of queries against your data platform, if you're not able to kind of control that at scale or help those folks interpret results or maybe understand nuances and data, you're gonna end up in, um, a, a, a world of hurt, right? Because you're gonna have a bunch of pipelines that, um, less technical folks have, have contributed that maybe don't really understand how, how the data platform works. So I think if you use it in the right way, um, and you use it as kind of a way to go make the existing data teams more productive, I think it's gonna be a huge unlock and I think it can help lower the barrier for other folks to contribute to and interact with data platforms.

But you have to be careful on, on that side of things. We, you know, you hear the phrase all the time, right? Data's the new oil, and I think, uh, the problem is, is we don't really know how to refine it, and as a result, we just kinda board it and we have lots and lots of data everywhere, but we're not getting a whole lot of value out of it, and storage costs are adding up.

So is there a sense of frustration in the system right now that needs to be dealt with and maybe data ops is the answer? Yeah, it's, it's a great question. Um, I, I think definitely, you know, data ops and also kinda the advent of, of AI can help you get a lot more value out of your data.

Um, I'd say that, you know, the other trend that I've noticed going on is, um, data teams can now kind of collect and centralize data at massive scale today, right? You have a bunch of technology out there that can help you with that sort of thing. And, you know, once you get all this data in one place, the question becomes like, what do you do with it?

Right? It's, it's not as simple as just let me go build a dashboard on top of my entire data platform to, to help me understand. Um, and that's where honestly, I think businesses have a, a great opportunity to use that data to their advantage.

The two examples that I like to go back to are Netflix and Spotify. So, you know, they were both quoted in, uh, Andreessen Horowitz's original post around software is eating the world because in Netflix and Spotify were able to take these, you know, very traditional brick and mortar, um, industries and, and music and entertainment, um, and disrupt the entire thing, right? By building, um, software that really lowered the barrier to accessing music or accessing TV shows.

Um, and you know, that's been incredible to see it kind of put them at, at the forefront of that. But if you look at how they maintain their differentiation today, how they maintain that position, it's really through their use of data, right? There are other platforms now that give you the same experience, the same access to music and, and entertainment as Netflix and Spotify do.

But, you know, because Netflix and Spotify have so many users that they can build, um, these great kind of data models and personalization models and recommendation engines, um, that keep you super, super engaged. Um, and I think, you know, they to me, represent very canonical examples of like companies that have been able to make this transition from, you know, building software as a source of differentiation to, um, you of course you still have software, but you now also lean into type of data that you as an organization have unique access to, um, to build, you know, stronger retention, user acquisition engagement on the platform. Uh, so I, you know, I think you're absolutely right that just getting the data in one place is, is not enough.

You have to go figure out clever ways to use it as a way to, you know, drive your business forward. So who's in charge of all this? A while back we saw people talking about chief data officers, and now we hear about chief AI officers and there was digital CXOs, and now the CIO is kind of stepping back up and saying, Hey, you know, it's all about the data, so I'm in charge, but what are you seeing?

Yeah, you know, I think there's really two ways to look at it. I'd say there's a very tops down motion of executives now care more than ever before, how they're able to, you know, collect, understand, and use data at massive scale to go drive their business forward. I do think AI has helped a lot with that conversation because it's, um, kind of forcing people to understand that before you can go build an AI strategy, you need a, a data strategy.

And you know, for better or worse, um, AI is, is on the top of mind of, you know, every kind of boardroom and, and executive room out there. Um, and so that's increasing kind of the investments in the intention and, and data teams. And I say that happens across the executive team, right?

From the CEO who understands that you can go use data as a source of competitive differentiation, whether that's in traditional, traditional ways or, you know, using AI and ML to, you know, CTOs, CIOs, CDOs, um, basically see anything oh, at, at this point. Um, but there's also a super interesting kind of bottoms up motion that, you know, we're, we're seeing as well where, you know, data engineers, data scientists, um, and other kind of data teams, they're the ones that have a very intimate understanding of the data platform and the type of data that you have access to. Um, and they're able to go build a strong sense of intuition for how you can use that data.

Um, some of the most successful examples I've seen of companies adopting AI and ml, um, the, you know, in a very pragmatic and realistic manner has come from the data, right? I think a great example of this is, um, RAMP who, you know, uses air airflow. Um, you know, one of the products that we work on, um, you know, their data team has the full context of what their users do on the platform, what their support team is responsible for, the type of data they have access to, and they've been able to build an incredible suite of, you know, AI and ML products on top of that data that didn't come from some executive somewhere saying, Hey, we need to go do this.

It came from the intrinsic interest that, you know, these data teams have in, in where the market is going and where the technology is going. And so that's been really an, an incredible movement to see as well. All right, folks, I think we're all starting to finally realize that this whole AI thing is about the data.

I think the part that we forgot about was that it was always about the data. Hey, Julie, thanks for being on the show. Of course.

Thanks for having me, Mike. All right. And thank you for all watching the latest episode of the Techstrong ITSM series.

You can find this episode and others on our website. We invite you to check them all out. Until then, we'll see you next time.