DataOps: Much More Than DevOps for Data | DataOps Day

Hi everyone, and thank you for joining me, Kevin Klein for the session. DataOps much more than DevOps for data. com, as well as on Facebook, LinkedIn, and Twitter at ke Klein.

So, to tell you a little bit about myself, uh, I'm, uh, I've been prominent in the database community for many years. I wrote my first book about databases back in 1994. It was about Oracle, and I've written many books over the years.

My best known book is a bestseller, seql in a nutshell, from O'Reilly and Associates. And, uh, I've been a Microsoft M V p, uh, data platform, M V P for almost 20 years, an a w s Community builder, community hero, uh, for the past, uh, three years or so as well, uh, very active in this space, and, uh, very active in the community too. So you might see me speaking at a conference near you this year.

So what are we gonna talk about today? We're gonna talk, uh, you've already had the introduction, but we're gonna talk a little bit about DataOps and DevOps and their origins and how they fit together. And there's actually a surprising third leg to the stool that represents the practices and processes and workflows.

That is the methodology for data ops and DevOps. There's a third one that most people don't think about. So we'll go into that.

We'll compare and contrast, and we'll have a summary and a wrap up at the end. Now, let's start, uh, with the way I like to learn, which is to have a broader understanding of the context, the context of a given set of technologies. How did they come to be?

How did they arise? What were the market forces and the business issues that drove the creation of these either technologies or methodologies? And so that's where we're gonna start.

Why did all of this start happening? Well, pre DevOps, DataOps, DevSecOps, all of the ops different, uh, methodologies. There was different technologies and techniques to try and manage your software development lifecycle.

However, they were quite ineffective. And one of the things that the, uh, McKinsey Global data, um, transformation survey revealed, and by the way, McKinsey is probably the world's foremost management consulting firm. Uh, they studied a lot of, uh, implications from having poor controls over it and dev processes and poor governance.

And so what they found was that you take two similar sized companies, two similar, uh, groups of staff of the same size and the same level experience. And those that use the DevOps data ops data governance approach to their IT estate outperformed the others by about 74%. And that's a, a significant, a significant difference in performance.

Um, it hurts the most in, in several specific parts of an organization in risk management, in operations, and in their finance. That's where it really hurts the most not to have governance and controls in well organized processes in place. We also find that a lot of companies have managed to do okay without these things, uh, without these methodologies, except when they go through big transitions and changes.

So I started my career, for example, these at the tail end of the mainframe era in the, in the mid eighties and worked through mainframes. I worked through the client server kind of approach to, to enterprise it through web 1 0 2, oh, web three, oh, um, the public cloud and so forth. And so, um, we find that it's those transition points where having these governance, uh, capabilities really adds the biggest amount of value.

If, for example, you're doing a digital transformation today and you don't already use DevOps, uh, you don't already use data ops, now's the time to start using those on a regular basis and making them a part of your corporate culture. And more than anything, uh, as we go through these different steps to talk about, um, where we are today with this, with these different methodologies, um, that is a, a great place to get started if you haven't already done them in the past. When you're building out your first set of data like lakes or data warehouses, you're doing big data integration projects, uh, you're building your first set of highly customized bespoke applications in-house instead of buying those.

So those are all great times for you to go ahead and say, all right, let's get serious about having a culture that works towards efficiency and quality and reliability of our builds. Now, I mentioned that DevOps is one leg of a stool, and DataOps is another leg of that stool, but there's a third one that most people lose track of or aren't even aware exists. And that is data governance.

Now, there's a number of components to data governance, and many organizations only do one or two of them. Um, many organizations maybe are very, uh, very indirect about all of them. Uh, somebody does these things, but it's not a structured or centralized approach.

Um, and I'll talk some more about this, um, about data governance and so forth. But one of the things to keep in mind is that these are, uh, a lot of these GA data governance issues are at least in part solved with the other two methodologies. And so if you only implement the methodologies for DevOps and DataOps, you'll get a few of these aspects.

Uh, for example, DataOps is very, very driven when it comes to data quality and data privacy, security and compliance. It also caress a lot about master data management, reference data management and data analytics. Okay?

Uh, meanwhile, DevOps is very concerned about metadata management and sometimes about the information lifecycle management of, of your data. And so you may not have all of these, but you might get a couple, even if you don't really pay close attention to data governance itself. I'll talk, as I mentioned a moment ago, I'll talk more about it.

But the main thing to think about when you hear data governance is that it's all about aligning what we do in it with the needs of the business. Now, that might sound elementary to you, but, uh, you'll find if you change jobs much, uh, over the course of, uh, a long career, that many companies do it and they do their business, and the two of them do not talk very often or enough. And so that's the starting point with data governance.

Okay, so, uh, I even have some more information if you're curious about these other parts of data manage, uh, the data governance, um, uh, process. So I, uh, I can explain all of these other, uh, different sections of the slide here for you, um, if you're interested. Okay, so now let's dive in on, uh, I've introduced the mystery third leg of the, uh, of the stool.

Let's dive in on DevOps and DataOps. And one thing you'll see me emphasize over and over again is it's much more about cultural changes in terms of your processes, your practices, and your workflows as it is tools specifically. Okay?

So I have used, uh, a kind of a broader top-down, uh, kind of view of all this data. Like I said, I like to understand the context of what we are, um, trying to learn compared to just learning individual facts about the thing today, I need to know where it started and how it came to be. So when we talk about data governance, there were many data governance, uh, processes and practices over the many decades since we first started with it.

Back in the fifties and sixties. In the sixties, banks began to realize that if we don't manage our data as well as we should, then people lose money. And in fact, that was one of the big impetus to move away from hierarchical and coil data models, uh, offered through IBM's products and move towards relational databases.

Um, so that was happening kind of concurrently as these new data governance frameworks were being introduced. Things like itil, things like, um, I s o 38 500. The problem was they were very big and clunky, and they're very slow to implement, and they add a lot of overhead, and that's not really what people want.

At the end of the day. They want to ensure that their data offers several specific kinds of characteristics. The big thing was laws started to be passed around the world about what you can do with data.

And so data governance was first about meeting regulatory and legal compliance issues. Then as the technology began to show its weaknesses in terms of mainframes and running batch processes, you'd have situations in which, uh, the data quality was poor, it was duplicated, it was, um, inconsistent. And the banks that we have today have grown through acquisition after acquisition.

And so they're all disparate systems being cobbled together as they build out all these, uh, modern application systems in, uh, database systems. So they have to pay attention to data quality. Uh, for example, if email addresses 300, uh, uh, variable character 301 database and variable character 1 28 and another, well, you're gonna have data quality issues.

And then finally, change control and risk mitigation. And in the mid nineties, when was when I first had to start sitting in on, uh, the change control board and decide when we're gonna do upgrades and things like that. Uh, so the whole idea is when we introduce change, we introduce risk.

And so we have to control that in particular when we change something and then have to roll back to a previous version, that's extremely risky. So we had to get a control of that kind of, um, set of issues. But all of these together feed into data qual, uh, data governance and data governance in turn into IT governance and overall corporate governance.

Okay. And this date I have, uh, saying 20, uh, 2005. It's not the origin per se of data governance.

This is, uh, I'm using as just a, a milestone, the prominent first big conference for the methodology or the concept. So, uh, in 2005 was when we had our first data governance, uh, multi-vendor, um, conference for the user community. If we come forward to 2008, that's when we start to see DevOps really formalized in terms of, uh, the public mind and being ready to prepare and deliver sessions at a big conference.

And so we see some similar kinds of, uh, uh, bullet points here with DevOps. The idea with DevOps is that we need to make sure that our software is in touch and aligned with our business users. And so we take short sprints, we use the ag of various agile methodologies.

We're constantly in a feedback loop with our customers. And so this is extremely important for us. Um, I've got the three bullet points to the left, agile continuous integration and continuous delivery.

In those short sprints, it's lean. So we use statistical controls and we monitor the, uh, the different increments. That's not to say that Scrum particularly, or Kanban or any of those methods have to be used.

They don't. You can use those that you prefer or even invent your own. But the idea is don't overbuild it.

Talk to your customers. Start off with a minimal, minimal viable product, and then add features. And then two other things that are really important here is automated version control, and if possible, automated testing and QA and then containerization.

And both of these features of DevOps enable us to, uh, much more effectively produce software quickly, uh, at a high quality that the business users can get immediate value from. And so, because containerization, for example, is, uh, associated with products like, uh, Docker, for example, and automated version control is associated with Git, people think tend to think of those products hand in hand with DevOps, but in fact, they actually aren't necessarily, uh, essential. It could be, uh, it could be GitHub or it could be GitLab, for example.

And then there's other products as well, Atlassian and so forth, um, uh, with, so there's a lot of different tools that can help you here. Now finally, we come to data ops today, and its first conference was held in 2014. Um, it also uses some of the same terms that we saw before.

Agile and lean mean pretty much the same thing as what they did in the previous, um, sections. However, what we're building off of is all the goodness that has already come to us from DevOps. And the thing to remember with, um, with DataOps is, uh, the biggest mistake that I hear people make, uh, is that they assume from the titles that DataOps is like DevOps, except it it's for database artifacts, like stored procedures, database schemas, um, any kind of code or batch processing that you might do in the database.

It's not actually true. Uh, we'll talk about it in a bit more detail. Um, it is more about data integrations pulling data from different places into one spot, like a data lake or a data warehouse, and the data pipelines that facilitate the data analysis.

So they're, they're not data ops is not DevOps for databases, it is its own discipline. That being said, though, we use many of the concepts that were initially started with DevOps. So you will want C I C D, for example.

You will want automation. Again, we'll dive into that a bit more. Okay, so let's compare and contrast or actually harmonize our understanding of, of the three of these data governance, DevOps, data ops.

Okay? So at that top level, even if your organization doesn't have a formal data governance, uh, process, and they don't have a formal data governance, uh, mission statement, it's being done somehow, maybe very informally, maybe too informally, maybe too casually, but it's about data quality and mitigating risks to your data. So if we have data in our, let's say, our sales database, and we have, uh, six entries for the entity known as Hewlett Packard Enterprise, we have hp, we have HP Inc.

We have Hewlett Packard, we have Hewlett Packard Enterprise, we have all these customers in our database. Well, that's redundant data, and it, it will cause risks to us when we start to process our reports to see how our sales are going, because we might not see in fact that this is one customer making purchases. And so each of the salespeople don't really want to give up the, their little piece of that pie because they're judged on quota.

And so that ends up making the business less effective, um, as opposed to the business processes in which the salespeople are measured. So data governance is about mitigating those kind of risks. It's also about making sure we uphold the law.

So although DA data governance started in its earliest forms, even as far back as the sixties and seventies with banks, we didn't have a whole lot of laws that we could get in trouble for if our data wasn't well controlled. Nowadays, for example, if your company works internationally and works in, um, at the eu, if you violate G D P R data privacy rules, you could be fined 20% of your revenue for a year. And that is a massive amount of money.

So we have to keep on top of that, the, the, the most fundamental aspects of data governance also then feed into DevOps. Now, with DevOps, what we're all about here is to integrate and automate the S D L C with our operation staff so that we have short, high quality releases that happen a lot. And again, you can kind of see how this would be the sort of thing that would wait until the 21st century to really catch on, because we're releasing our apps now instead of onto CDs twice a year.

Uh, we're releasing them immediately to a website where that change can immediately start to benefit our customers. And so for that reason, it really be moved from being great ideas, but not critical to being really good ideas that are critical to implement because they make us better in the marketplace. They're more, uh, tools that we, or software that we write with DevOps is more effective and capable and more satisfying to the customers than without using DevOps.

And that also means that we sell more, right? So think of, uh, DevOps as all about collaboration. We're talking very, very frequently with the, the end user.

The persona who will use this, uh, product that we're writing will get feedback all the time. In some cases, it was always a joke in the nineties and early two thousands that, oh, you know, you've gotta have a well-written requirements documentation. But with DevOps, because you're talking so frequently with your customers, you might have less of that, um, not a big thick 30 page manual full of requirements, but you might have just a, a couple sketches and a couple requests for what the customer is asking for.

And then you get on the phone and get clarification as you finish up, uh, your first release in that one or two week period of time. Automation extremely important. Uh, things like GitLab, GitHub, where you can fork your code, where you can, um, uh, package it up.

You can have versions and so forth. Uh, you can do it constantly. And, uh, as I mentioned, we, we get that feedback from customers.

So we're, we're never spending months of time doing something that is not at all what they want. Also, I would say focus is a key word here as well. We need to always keep the end target in mind.

Uh, I can't tell you how many times I've been involved with projects that get derailed by something called scope creep. So, you know, you started with an initial design, it's gonna take, uh, nine months to, to finish that project. And then the project owner says, oh, we want you to add a whole new reporting module.

Oh, well, that's three more months of work. So is that necessary to release the product? In those days, they would say yes, but in fact, no, you can have that minimally viable product far earlier than that.

So that's one of the things that DevOps emphasizes. Keep your eyes on the prize on that final form. And then anything that is requested from that point on the start of the project now goes to the end of the queue so that you can continue to make progress against a living, breathing application that the customers really want and start using right away.

Right now, data governance fed into DevOps, DevOps feeds into data ops. Okay? Now, uh, again, there's a, a lot of words here on our data ops, uh, page, but it has the same sort of emphasis on things like continuous improvement, right?

With human feedback, uh, automation. So we want to use a scripted methods for creating data pipelines rather than manually, uh, creating data pipelines or using a gui. Um, you might skip that and go straight for the scripted solution so that you can just, uh, modify and tweak the, uh, the pipeline a little bit anytime you need to make a change to that data source.

Um, benchmark information. So this is something that, um, each of the different previous methodologies mentioned as, uh, lean, they'll say that word lean, and it, it originated with lean manufacturing invented by the Japanese at Toyota and, uh, six Sigma. Uh, and some of those other kinds of statistical analysis for how many defects we have.

Well, that's, that's a big part of, uh, data ops as well. And so we need to be monitoring our systems. When we get to the stage where we're using data ops, we wanna know empirical metrics that tell us how well, how fast, how, um, few or how many errors are occurring, what the latency is and so forth.

And so, benchmarking and maintaining a collection of performance information is extremely important because you need to know how much your users are having to contend with in terms of performance. And as you grow and add more data pipelines, then you need to monitor more, uh, data activity and integration as it moves from various locations into your data lake or your data warehouse. Now, one of the things I also wanna point out too, uh, on this point about established data governance principles, this is also, um, kind of, uh, a look over your shoulder.

If you don't have data governance discipline, you need to add it now that you start to use a lot of data ops. Um, so I mentioned some of the, the initial impetus to get started with, uh, data ops is the fact that we want to have clean data. We want to be able to rely on our data and not waste a lot of time double checking that data and so forth.

But one of the other big things that, uh, started in, uh, in the second decade of the 20th century, um, is we began to see a marketplace for data. And so what we have is a variety of organizations that create their own data, but then they also supplement that by purchasing additional data from other sources. For example, you might use Salesforce to, um, to track all of your c r m, uh, all of your customer transactions, and you might use HubSpot to do, or Marketo to do all of your marketing.

And then you might also purchase, uh, census data, uh, from the census department. And then you may also purchase data from Dun and Bradstreet and Bloomberg. And you bring all of these together.

And now, hmm, who owns this data? Who controls the data? What's the lineage of the data after it goes through two or three different information systems to start producing reports or, um, power BI or Tableau dashboards for management?

So those kind of things are very prominent questions that DataOps helps you question. Um, I'm sorry, helps you answer those questions. As I mentioned, it's more important than ever before because we have all of these software as a service, uh, products that we can buy into, and we want to take advantage of the information that is aggregated in those, uh, collected in those applications.

So that's another reason why we need DataOps. Again, it's separate from DevOps. Uh, it is separate from data governance, but if you do the three together, then you are enormously stronger than your competitors and you're gonna give much better value back to your customers.

Alright, so to wrap up, uh, one of the things I like to think about here is that there's sort of an integrated message when we start to look at these methodologies. Um, the first is, although tools are gonna be part of that process, it's really about culture change. It's about changing the way we communicate and work through our processes and what those processes are in their workflows, okay?

That's where the rubber hits the road. And, and cultural change is hard. That's why I always encourage people to piggyback a cultural change on some other major initiative like building out your first data warehouse or your data lake, uh, building out a digital transformation strategy.

That's a good time to start to include this kind of transition inside of your company because culture changes are hard. So if you add those two together, it, it doesn't make it harder in aggregate. It's just one hard thing that everybody has to go through.

Um, so not about specifically about the tools and technologies, um, but each one has recognizable aspects to them that carry through to the next set of methodologies. Data governance, make sure we have high quality, uh, data that is not, uh, duplicative, that it is, uh, reliable and that it helps us achieve our business outcomes. Um, you know, that it's secure and that it meets all the regulatory and legal compliance standards that we have to adhere to DevOps more in use by the development teams.

But again, it's about working together and communicating, talking to the people who own the data and the data sources, talking to the people who, uh, own the projects and applications we build on top of that data. And it's, uh, gives us a way to ensure velocity quality of our pro uh, products and, uh, alignment with what the customer's needs. And then finally, data ops is about the data, uh, data pipelines and, uh, data, uh, e t o processes and transforming that data for analytics so that we can make better choices, uh, with our day-to-day business.

Now having an understanding of these three pillars of a good data practices, and, uh, maybe you're unfamiliar with more specifics about it. So, uh, what's your next steps? Well, you're actually following through on the first most important next step by attending this conference, which is to learn more.

The other thing to do is to start to look at the different tools that you might want to use. Uh, I mentioned Atlassian already. There's um, many others, GitHub, GitLab, uh, and, and quite a few more that can help with DataOps.

There are tools specifically for, um, DevOps, I'm sorry. Uh, there's tools specifically for DataOps like Data Kitchen, um, and then data governance. There's tools like, uh, Matillion and Alteryx and several others.

So there's quite a few ways that tools will accelerate everything you're trying to accomplish here, uh, on, on the level of your processes and workflows. Now, having said all of that, we're at the end of our time. com and also reach out on me.

Uh, reach out to me on Twitter, Facebook, LinkedIn at ke Klein. So with that, I'm very thankful for your time. I'm grateful.

I know you have a lot to do, and I hope you'll join me at a another event sometime in the future. Thank you.