Kohsuke Kawaguchi – Data-driven DevOps: The Key to Improving Speed & Scale

Ok, hello, everyone. Today, I wanted to talk about the data driven DevOps, it's something I have in my own experience talking to various folks. So before I start, let me quickly introduce who I am.

So I'm probably best known as the guy who invented Jenkins. That's the open source project that more than 200,000 installations around the world. And chances are pretty good that some of you have seen this somewhere in your journey.

And I also was involved in Cloudbees from the very beginning. And among other things, we helped enterprises everywhere doing DevOps Digital Transformation through Jenkins or by other means. And now I think for about a year now, I moved on to Launchable where we started.

Well, I have the next to this company where we are trying to focus more on the QA, the smarter testing. And then I'm going to talk a little bit about that later, because naturally this came out of my progression, so. Part of the I guess the benefit of me being in this role of the creator, of Jenkins and being involved in Cloudbees and so on, is that I get to talk to a lot of software development teams doing stuff for them, often having challenges.

So from what I can see in the industry, we did make quite a bit of progress. We have evolved quite a lot. So, in fact, I remember back when I started Jenkins, we are talking about maybe running the test every once in a night, and that was it.

But today, if you talk to people, it's far more common to talk about something like a pipeline that spans from the building to the system testing all the way to production deployment. And I'm sure a lot of you have already done this are on the way to get there. Now, this screen is taken from one of the Jenkins UI, but the reality is a return that is not like this, but more like that.

So you it's like they're the turtles all the way down under you. So the things that they're building on top of also came from some other things that you might come out of a busy routine to be part of the software development cycle that's close to you. But once where the things that you depend on come from, let's take that based on captivities you might have or the things going to happen to the binary device.

You ship them, let's say the container, once you hand over to the operations team or in the authority of what's going to happen, that's usually not really something you have good delivery on. So by and large, I think what I observe is a lot of things in what we do is scripted, automated and so on. But they are put together through human processes, communication, and that's not quite, you know, between the automated and trackable.

So at times when I ask people to show me, you know, hi there, their delivery pipeline is put together, they will see their Jenkins. Oh, this is where this happened. And this is where the tests are ran.

But trying to piece together the entire that story is like being a bit like watching Beehive and trying to understand the beehive's behavior by following individual bees. There's just so much activity at the lower level and they're so disconnected from the higher level fixture. So.

So that started. That's something I started thinking about. Now after talking to so many different teams and they all have different challenges, but it does start to feel.

I started feeling like there are two kinds of, you know, the companies that one means you donkeys and the other is the unicorns. And they are both well known software companies doing software development on a lot of the teams in software development. But but one seems to be doing much better than the other.

They kind of expect that performance is entirely different. And instead of thinking, well, what differentiates them is actually how they use data that coming out from all of these automation that they have already put in place. So and that's sort of like the ones that I got.

So I wanted to explore that today to see what the difference the data can make in improving your software development process. So. And another way to think about this is I don't know if this is entirely true or not, but if you buy bullion, the service that we throw out contains a lot of precious metals like gold and so on.

So by bullion, that has a higher concentration of these precious metal than the actual mine. And this is a kind of feeling I get when I look at how people are doing automation today. So thanks in part to Jenkins and whatnot, people running a lot of automation that's producing a lot of data.

But by and large, these data are simply wasted and not looked at that if you can find ways to productively utilize those. I think it's probably hopefully you can imagine what kind of impact that this might not be difficult to imagine that you can make a good impact just by recycling some of these, actually. So that's the so to make this concrete.

Let me look at some of the concrete examples that I actually need to come across. So the one time I landed in this company in the Silicon Valley and this is a software startup. So by this point, they have hundreds of software engineers working on tens of projects.

And in this place, the effort was nicely centralized. So I was talking to this one team who was running the entire shared CI infrastructure, the Jenkins infrastructure for the entire company. And everyone is putting a lock on this single instance, single infrastructure.

So which is great because before it's all scalable and low overhead and whatnot, and then they are racking up like in the order of hundreds of thousands of dollars that we spend because all the build and tests are costing CPU and whatnot. So things are going, the cost wasn't all that concern for a startup that's aiming to grow quickly, but when the company is so successful, at some point, what's going to happen is that they start thinking about the IPO. So the new CFO would come in and make sure the company looked good on the book so that in the some state filing it's going to look great.

So one of the things that the CFO discovered is that, hey, there is this so much cost spent on AWS. Like, what is that doing? What's the return on investment on these things?

Like what exactly is that for? And then this team suddenly realized like shoot that we thought we have no idea who is using these things because the central, one of the central DevOps team now they are just running this infrastructure and no visibility into the Veracode. and people in the projects, they have no visibility into the cost that they are racking up because for them, as far as they're concerned, this infrastructure is free.

So. So what should have happened, and this was something that they are thinking about doing that while we we're discussing back then, is just provide some visibility into the cost at the project level. Instead of making them feel like this is a free service, give them some level of awareness that what it's costing and that kind of provides often enough incentive for people to do the quote unquote right thing.

So in this case, their central service is providing three different kind of the same type that just call it the small, medium and large. Obviously, the larger one is cost more money that the bigger and faster and everything so that the price signal, the incentive obviously is just always go for the biggest possible one that you can get because like, why not? So but it's something even as simple as if you can just help those people like, hey you have these three choices.

And this is the implication of making good choice, both in terms of the time and cost, then that's going to help them make the right choice. And if this is if this is a query based validation or something that makes it makes the developer wait, then it's probably worthwhile to do this quickly, even at the cost, because the people cost are far more expensive. But this is something that only needs to run in the night.

When nobody is watching, then it's probably OK to take longer. If it saves money, then that is the kind of tradeoff that the people close to the project can be, but not the central theme level. And additionally, if they can segregate this costing through different projects, then that's going to also help them see, you know, which product line is driving revenue versus cost and so on, so forth.

So that's a pretty trivial addition. So remember, this is one example where the data can make that drive the behavior change and make the process better. Here's an example from another team.

So these are the far more established companies in Europe and they have thousands of engineers actually around the world working on a large scale embedded devices. And it's also great that they have managed to have this one central DevOps team in China, I believe. And they they they run this very large scale testing infrastructure for the entire team of thousands of engineers.

So you can imagine the scale they need to be operating. So the challenge is that these guys are facing another need to be from previous example. So here the the idea was when something bad happens in this infrastructure, let's say people run the tests and they fail or the people run the build, they fail.

Who should get that notification first? Or you see what's going to happen. What's happening here is if those failures legitimately caused by the changing applications, like one of the application developer in Germany inserted a problematic code, then that's going to blow up.

Then it's that level who should be notified, but in other times, the sometimes the CI infrastructure develops some problem, let's say that this type of space or the database server used to run the test went down or some of these issues are complex in our system. And then when that happens, what you don't want they don't want to do is to notify these application level wherever because they are not at the fault. It's the DevOps team who needs to be notified and look at need to look at the problem.

And when large organizations where there's still a separate team to handle these things, especially in this space for DevOps team, it's super important to maintain the credibility that what the system in the service they're providing is trustworthy. The test result and the build result they are reporting is that is trustworthy. So otherwise, you know, they they've stopped coming across more like a crying wolf.

And then people start losing trust in automation and everything deteriorates and kind of gets ripped apart. So this is another example where so what they actually saw, what they did was it's incredibly simple so at first they deploy this regular expression matching, so they looked at the last 50 lines or so. Then they use that to decide who should get a notification first, because I guess the idea being there are well understood failure mode that's not caused by the applications.

And so that's going to help them decide. So I was just I'm struck like how simple minded the solution is and still pretty effective, obviously, that that they're reporting. And then just kind of speaks to the point of how much data are getting wasted, like it's something that can be essentially all of them are wasted today.

So even simple attempt can produce a big return. And then later, I was talking to another company who had the exact same program and they deployed something a little more sophisticated, which is that Bayesian filter, which is the same technology that people use for computers. So the idea here is, you know, like every type of notification goes out that contains this, a button that says this is not my problem.

So let's say the system just choose to send and this notification to the application developer and if they trust this mechanism, my problem with them being that this is the future that misclassified and then the algorithm kind of slowly learned about this in the same way that the future passed by your mocking emails and spam or not. So, again, this is a little bit more sophisticated, but not too much. Again, this is a technology that exists for multiple decades that.

That so so though those are those are the kind of technical things that got me thinking about this, but that's the way I got I started seeing this data problem from a different scale, the organizational level. So all too often, when I talk to the practitioners in companies that the problem is like their challenge, rather, is that they know what they need to be doing in order to up the game for software delivery and by in large. When I hear their stories that I kind of agree with them that what they're thinking about doing seems right.

But the challenge is like they are failing to rally their organization rather than getting the resources necessary to move those efforts forward. So as a leader driving these DevOps before I actually think this might be more important than the second quote unquote, the technical solutions, this program that I described in two of points the key thing that the leader, the only leader can do is to convince the rest of the organizations that this is more this program is worth solving and this is the kind of impact it's hoping it can create and so on. And I think our time in the industry that we tend to do this in by using the technical term for the languages that other people don't understand.

And I think the data has the power that crosses this technical jargon that professional boundaries and this is something these are the things that the business people actually are trained and offer far more keenly than we can. So part of the style of thinking is, well, you know, we sometimes we think we don't need to do this because we know that these are the problems. But in order to make other people see it having data in the story that backed up by data I think is crucial.

And once you have those stories, then it's actually also good for you that you think you might know where you need to apply for. But this kind of thing allows you to verify that that is indeed the case as opposed to us just thinking and believing in face. And then no, and let's see this mode, it also helps that you can start to show the impact as they happen as opposed to waiting for the anecdotal story at the end of the year, like in Asia, if our story is like, hey, we need to take on this digital transformation or like a DevOps transformation and it's going to take two years.

The culture change in the organization, then maybe you start seeing the impact then it's no wonder that your CFO is like a little bit wary about giving you the medium of the drive, whereas like you can tell if you can show that, hey, that's the thing that we spent money and time on. It's producing these kind of impacts through these KPI or the metrics then that I think that only adds to your credibility, making it easier for us to ask for more things. So that is OK.

So for me at least, the newly discovered power of data that is crucial outside the domain of technology. So the the kind of moral that this is, the mental picture that this is trying to take is really to see the software development process as a little bit like that factory is a repeatable motions. And, you know, the idea of being a continuous software, the factory, the idea gets turned into a functioning software.

And we are in the business of doing this better. We are doing this in the business of doing this over and over. So what we want to do is to monitor the process itself and then see if you can use that to drive the improvement in the process so that some of the what we looked at is very much in the spirit of the fact that.

We are monitoring and notification that's going from the system to the whole person and deciding where to send them, and then by picking these things, it gets better that I don't think there should be any like a groundbreaking discovery or anything. We've been talking about the software factory for like a decade or two. So that's not really new here.

But I think what actually new is that if you if you look at this kind of continuous improvement mode, continuous learning mode, like it suddenly start to make sense that some people are trying to rely on the power of the machine learning in order to help some of these things. So I wanted to kind of give you a few examples of that, because I find those pretty exciting. So here's another company.

This one is also in the Silicon Valley. And so they have they are huge, hugely successful company. And then one when one product that they had a service.

So they have this large modularize code base with it's well organized is just so massive. And the entire software, the bulk of the code based on repo. And then I was talking to their DevOps team of this company and they are responsible for turning this BigCo base into building and testing into the runnable executable software.

So, you know, these companies in business for, what, 20 years probably. So you can imagine the size of this and the amount of testing that accumulated. So, you know, by simply becoming impossible for them to just run all the tests, all the time against all the changes, the amount of change is just the high amount of code based too big and the number of tests is also too large.

So what they are trying to do is to find ways to cut the cost of time to do the software delivery that the what they have already done ages ago is this dependency based on all this is to reveal to the people that test execution. So this is a simplified picture here where, you know, the diamond at the at the bottom represents individual files. It gets aggregated into squares in the middle, and that is the modules and then the circles of the top, those of the test.

So, you know, they're built system understands those dependencies between different modules and different tests. So when somebody makes a change in, let's say, you know, the file on the very left, the square on the very left, then they can point out that the build system can be for that. Oh, there is only one module that needs to be built and they'll need to see that needs to be re-run and everything else.

It's like, yes, that's really far that there is no need to be doing any of this other stuff. So it does add you can imagine it helps. You know, it's code base.

And again, I actually made a point that that I mean, it seems pretty obvious like there's no there's no leap of faith or anything like that. But then if I go, I think this is already more than like, you know, I think the top 10 percent is the answer to this kind of effort, because most of the places they visit are simply de-compiling everything and running every test all the time for every single challenge. They just like anything beyond that is the complex or whatever for them.

So that just talks about it again. I highlighted the low hanging fruit are still left around in what we do. But anyway, so this is this is not what I.

So this is just a baseline. So what's going to be my mind or what's what this shows off the level about solving this problem is they wanted to take the next step and then call that a predictive test of excellence. So the idea is that let's train the machine learning model that predicts the useful subset of the test, not just the ones that the subject of the build to determine that that might be necessary.

And then because there is a scale of development, it's so large that they have hundreds of thousands of attendees flowing into the system. This code for every single moment. So they just decided to that one percent of that traffic.

And then for the attendees or those one percent of attendees, they did train the model by re-running every test, the lot of attendees. And then that is ultimately on your model, so to make a good prediction on which of the subset, and narrowing down the amount of testing, much farther down than the analysis based approach. So what they did was they were able to cut down the test number by five to a third.

And then the obviously the concern of running just a small amount of testing, that it might miss some digressions. 1% of the broken changes it's accurate system and then the tests. But they were able to cut the testing to third and then the execution cost they cut costs by about half.

So I imagine that their scale, they must be feeling, maybe seven. Is it like a six or seven digit number easily to the provider. So to be able to cut that by half by doing something like this is really impactful.

So then I think this particular one is useful way beyond the companies. I think the predicting probability of these test failures is very useful in many situations. You know, like I my own project, my Jenkins when I was engineering Jenkins I had to wait for one hour for the CI system to run the entire test on my change before the core review begins.

So I felt like, oh, if I had this technology, I could I could use this in my project. And that was great. And then for I know many places have these large integration tests or system test or whatever you call it, that that's so bulky they can only run once a night.

Those are like a month by our being. So if you don't but we know that most of the time most of these tests aren't failing. So if you can find predict which ones are more likely to fail, then that the same technology would be applicable for for for quite a bit of use cases.

So the idea is that by some means, we try to estimate the likelihood that the testage fail. And so that's where like a machine learning model, I mean that if you can make a good estimate, then what we like to do is to order these tests so that the tests that are like more failing. It's going to get executed quickly.

So that's the and in this way, as we start executing the test, we can expect the failures to happen quickly. So, you know, in my earlier thirty five hour waiting for if I have a pull request then and I made a one line change to a big system, then I expect that the tests that are sensitive to my change, it's going to be a pretty small subset of the entire test. And if you can get those around quickly, then I can expect to hear about the failure quickly for them without the contact switching.

I can go back to working on that tends to fix that regressions. So that's the idea of deducing the time to failure and that helps the developers. And then we can also do that.

Take it one step further. Well, if we know that some of these tests are unlikely to fail, like what if you just cannot run those tests at all and then just cut them out and we call that adaptive adaptive subset. So in this way, then we can kind of pull in the completion time as well and also deduce the workload down the line in a cloud or hybrid infrastructure.

And that's also a great, great, I think, the boom to the DevOps team. So, so so this is like I got so excited about this, this is what I'm working on now at Lauchable. I'm trying to democratize this technology.

And so far it's looking good if you or your team is having this kind of program. I'd love to have conversation with you. So interesting just following up on that anyhow.

So that's just one example of, you know, like a machine learning, delivering the impact, the software development process. I just toss to another example. They say this is another company.

I was the subject of a guy who runs the SRE team there. And they have this is then on the service that consists of hundreds of Ops and they're already pretty well into the continuous delivery. So he said on average, every one deployment per hour per day.

So that's kind of astonishing for me that they have. What that means is for the society, there's the hundreds of deployments happening every day, but I can also see it's kind of unnerving state that you have this firehose of things coming at you and you're not for keeping the production up and running. So what they were always interested in doing is to try to flag the risky deployment beforehand so that they can have adequate support in place just in case the bad things actually happen.

So what they did is they train the learning model. So they looked at the forty thousand deployment record that happened in the past 12 months, of which the by more failure. And then they just spent a number of interesting what they thought was interesting parameter features.

And then they tried to make the model produce like return of those deployment will likely cause a problem. So if I understand them correctly, what they are saying is that in the end, eventually there was a little bit of a struggle, but in the end they were able to make like, predict 99% of this failure before hiring only like a 5% false alarm ratio, meaning the model will be trained at all. This is a risky deployment, but it turns out to be OK.

There's only 5%. So that's almost like unbelievably good to the point of. It's scary, though.

So, you know, what does this allow them to do? So there are a couple of things they said, so they said, you know, if they know that the deployment is going to be risky, you know, for somebody to be on the call from the engineering team so that they can they are ready to jump on in case of trouble they could also restrict window, the time window of the deployment so that they won't be able to do that in the late Friday or even when the peak traffic arrives. So that seems like a sensible indeed.

It makes sense that these information can help them do things better. But they also what actually I got. What got me more interested is the lesson that they learned from this.

So they said something like, you know, hey, from this through this model and through this Veracode, we were able to learn that the the attendees that are rushed. So that's the second a short time span, still up to about 10 days that the rest can be risky or that the deployment of a long running service like service that's been long maintained tends to be more risky. And before they brought that machine learning, that they are originally depending on the developers to flag the deployment, the risky or not.

So it was know like honor system, we expected the developers to be honestly telling these things, because at the end of the day, their interest should be a line that good service did have a service, but it turned out that there are not a good indicator. So, yeah, so they were able to keep doing that. Like you might think hearing those things felt like duh like this is all obvious like that's that makes sense.

There's nothing new here. But actually I think what's different is. So there's a lot of difference in these claims being anecdotal, we're told by the engineers versus theories backed up by data.

So, for example, if you could start saying, like the hey a long maintain code is more risky. 5% to the deployment risk. Then suddenly now you can have a constructive conversation about, hey, we need to refactor this or we need to rewrite these services.

And previously, you can never sort of, you know, make a convincing argument when the service needs to be rewritten, anecdotal and mood and that kind of belief. Now you can buy a number based conversation that makes it far more easier to argue. So I thought that's actually a pretty amazing progress.

And these are that's not the kind of things the only possible through the data analysis. So this kind of gave me a kind of pause, really, because I used to think and I think this is the prevailing wisdom that if you have a small team of elite developers, you can you can be nimble, quick, and you can be amazing for this. So you could be like David as opposed to Goliath, who is like large and slow and clumsy.

So that's kind of how, you know, the developers often pride ourselves that our individual caliber makes a big impact. But if you notice in those last examples, then it's only possible in this large scale development shop, like the one prior, like tens of thousands of attendees coming through the system. But how many of you have that or how many of you have cycles in us about this kind of deployment risk prediction system?

So in a sense, maybe the actually the this picture might not be correct. Perhaps the software development is getting into this or app scale thing, kind of like manufacturing has been so many years ago that the smaller like a mom and pop shop just can't compete with large scale operations of these factories and fashion companies. So I, I started thinking maybe this is the kind of things that's happening in the software development industry.

I don't know if you like that or whatever else for what it's worth. And it's believable. I think it's believable the directions.

So with that said, it's not like every large company is a unicorn, right? I mean, the probably most of us are feeling like our company is large, but we are not doing anything sophisticated. Itf anything we do feel like the size is dragging us down.

So and then I do feel that's the same way. So then what are the difference? Like, again, like what makes all the difference that creates the second donkey like large companies that just a development team and a unicorn like team?

And that's been kind of like my quest. And on one hand, I do feel like the donkey kind of companies, different teams are doing things differently. So there's always a local tribe and culture.

The entire landscape looked more like a positive hodgepodge of villages. Each of them speak different languages. And so in those places, because of the diversity that come when one tribe, its culture is different from another culture, the central DevOps team, even when they are made, they end up effectively either just a glorified IT you run some key infrastructure services, but with that big understanding what's happening on it.

So that was one of the ideas that we talked about or the kind of professional services where this team has that automation export them, that kind of like a missionary, like they get assignment and then they go through some of these villages and they vandalize and they come out and they make the impact in that one village and they come back and they move on to the next village, which is great, but fundamentally not scalable. And at scale, the organizations are like they are wondering if this is how long it's going to take to convert the move, the entire nation of this space. So it feels like a very slow, painful journey.

And despite the fact and then that every team is doing things differently, these application things, these people in the village feel like the central theme is imposing this like unreasonable thing or bullshit on them, and they'd much rather retain the control of how they do things on their own. So no one tried to create the frictions that if I go through these unicorn kind of teams, I much more rather are often. I see that everyone in the whole nation, whole companies doing things one way or the uniform way of doing things, so that allows DevOps teams things to be much autonomous.

But that's the thing that only saved by a long way of doing things. So they don't have to send people to the village that they can do the control they can control. They can make changes to the DevOps delivery process on their own without needing anybody's permissions.

So they have far more control. But despite that, these villagers, these application teams feel like they they they're feeling great because they feel like the bullshit is taken care of by this DevOps teams. These engineers are not interested in, you know, thinking about feeling it like to be in the test process so that somebody that they are actually appreciative of DevOps team and then looking for opportunity to get more of the things onto this DevOps team, and that this creates this very positive feedback cycle where this company, one way to do things is the best way of doing things.

So maybe a great example of This is what we hear from Google engineers that say that they like their internal automation tooling around development and so much that they it's actually becoming a bar that prevents them from going from Google and the few that that goes out more often, that they try to reinvent the system that they have in Google. So that just speaks to the speaks volumes about how good they are that positive feedback has been in that company. So you know how some obviously then another question is like I observed this .

Like most of us are trying to think of what we find interesting, moving from one to the other. Like, how can we go from donkeys to the unicorns? And then that's the and.

I wish I had to answer that, like, most people are feeling like, gosh, it's just so big and they don't even know how happy we are to start across this gap. So that's the kind of desperation that I see and I don't really have, you know, like I already said, maybe that answer here and then this is the kind of question that nowadays, I mean, to talk to these different teams and I think there's clearly like. I like to think of that as a critical mass of values that the DevOps team can provide so that this positive feedback loop kicks in and, you know, the will get going that same kind of time that you show a critical mass happens.

So that's what I described as the well-funded DevOps. Obviously, another factor in is like getting building initial success and then tried to like and grow that success into progress with on a scale, what success you should try to build where that seems like a crucial part of getting yourself out off the ground. So that's another thing in this thing.

You know, like sometimes I've seen some people try to tackle the brown field product. The main thing, the mothership of the company, like the main product, and that's always has a good high impact if you complete that everybody else around. It's just easier at the front, but it's also hard.

So perhaps it's better to focus on the green field up on the peripheral, an effort, but then the you risk creating something that actually doesn't apply to the brown fields. So there's pros and cons on the both sides. I'm curious to hear people's expense on those things.

And, you know, the other thing is, like people I know who've seen this kind of effort develop a productive, therefore attached to a larger cloud migration, microservice transition is trying to show so and then basically piggyback on their justifications. So that could indeed be a viable path. But it also makes the whole thing go slower and that makes it difficult to see the value just on the segregated benefits from the larger effort.

Yes, I think these are the kind of things like a lot of us are trying in different parts of the world, and I think we all need to be spreading information, sharing our notes. Anyway, just to wrap up the whole thing. I wanted to make, the claim that I wanted to make here is like automation in 2020 is a table stake.

And I don't think anybody is seriously questioning that more and more people are doing this every day. But I also claim that the automation alone is not enough and that the next frontier of our software delivery process is trying to use the data that's coming out of the automation. And that's not that's that's not still widespread.

But because of that, I think there's a lot of low hanging fruit that even some simple efforts can have a big impact. And that one of that impacts, includes rallying the organization around the effort. And they also made the automation that a unicorn seems to be using the quote unquote could be beep from the software development, the affection within themselves.

So that's it makes us wonder how the rest of us can get there. We can be doing the same thing we can. We don't have cycles to do the same thing they are.

So then what? So on that note, this is what I wanted to talk about today. So thank you very much.

And I hope you enjoy the rest of the event.