Scaling the DevOps Workflow Mountain with Eric Billingsley

This is Textron tv. Hey guys, thanks for the throw. We're here with Eric Billingsley, who's vice president of SaaS for CloudBees, and we're talking about platform engineering.

Eric, welcome to the show. Thank you. Nice to be here.

I think we all talk about platform engineering, and some folks are viewing it as maybe the revenge or centralized it and other folks are saying, we're gonna finally fix our DevOps ills because, well, we're having a hard time scaling things and things are a little expensive, and folks are all over the spectrum on that opinion. But ultimately, do you think if I do decide to go down the path to platform engineering, then it kind of drives me to more of a SaaS mindset for managing my DevOps frameworks? And if so, why?

Yeah, I think that's a great question. Um, so when platform engineering first entered into the market, what everyone started doing was kind of building things themselves. There wasn't a good set of tools to build upon.

And so you ended up with things like backstage being the, almost the definition of what platform engineering is. Um, but it's more of a portal, right? What I think things are going to evolve into is a much more fully featured system that allows you to have a one-stop shop for engineering process, engineering documents, engineering tools, so that, uh, as you're bringing on new engineers, as you are scaling up your team, the system actually grows with you.

And the natural way for that to be delivered is as a SaaS product, or at least that's what we think. Uh, so you know, you're seeing that more and more enterprises even are moving to SaaS for their source control. They're moving to SaaS for pipeline execution.

They're moving to the cloud to actually run their workloads. And so as things start to move outside of the enterprise firewall, the natural progression is for more and more of these things to become a SaaS service. How do I get there from where I am?

Because to your point, most folks built something themselves and it took a lot of effort and there's a lot of customization. And so how do I get that from where I am into something that is a SaaS kind of experience? Uh, so yeah, what we see is that, uh, a lot of companies actually understand what it takes to build a platform.

They don't know what it takes to maintain a platform. And so, um, what we find is that a lot of our customers actually were willing to invest upfront to build it. They're not willing to maintain the investment going forward.

And so as we're working with customers, we're working with them on that transition strategy so that the capabilities they've come to depend upon in their internal world can get pushed into our platform so that they can actually experience that as an end user and not lose that functionality, but do lose the maintenance overhead of it. Is AI gonna fundamentally force us down this path anyway, because I need to aggregate all this data to train the AI models and the only place I can aggregate the data is to put it somewhere in a club environment? Yeah, pretty much.

Um, so we're seeing that a lot with customers as well. Everyone's excited about the, the possibilities of ai. They're excited about their engineers getting, um, you know, access to copilots and and pair programmers and those sorts of things.

Uh, but in order to make those really work in your organization, you need a level of customization that's based on your data. And so, um, by moving into the cloud, by moving into more SaaS products, that you almost as a byproduct over time are going to get the benefits of their AI solutions on your data. Whereas if you kept it all in-house, now you have to understand it, you have to build it, you have to be the ai, um, engineer on top of your day job.

And you know, a lot of companies just aren't set up for that, especially not for their internal processes. So if I move to a SaaS model without necessarily exposing my data to something, and probably I don't want it to be seen by, but in general, I should be able to benefit from all the data that's been used to train those AI models there, that that's beyond just my own. Well, I mean, if, if you're working with SaaS companies like us, um, where you can opt in to those sorts of, um, benefits, then yes.

Right. So, um, you know, if you're going to anyone and they're just leveraging your data to train their models and you don't have a say in it, that's a problem. As we come back to platform engineering and, and think about ai, what's that team gonna look like?

'cause I can imagine a world where it's gonna be, uh, you know, a bunch of human DevOps engineers augmented by 10 times as many AI engineers that are helpers, or, I mean, and I gotta manage this workflow and orchestration and hand off tasks as that where we're headed. Uh, I think there's a bit of that, but, um, more importantly, what what we're seeing with platform engineering teams is that they're evolving into, it depends in some, in some companies we see that the platform engineering team is the rebranded DevOps team, which was the rebranded SA team. Uh, and others we see it as teams that have taken on backstage, uh, and in the more mature organizations, what we're seeing is that there are teams who are really trying to tie together the processes of the company.

So as an example, I'm under ISO 27,001, I'm under SOC 2 compliance, I've maybe we're a medical device manufacturer. So the FDA actually cares about our software. Um, under those regulatory requirements.

Every bit of my code, every bit of functionality I introduce into my code actually has to go back and I have to have an audit trail for the requirements, for the code, for the tests, for the pull requests, for the deployments. I need all of that in one place, otherwise my audit is gonna take six months. Mm-Hmm.

And so these, these new high functioning platform engineering teams are actually thinking about that whole process and how to standardize that for my organization so that we can actually pass audits so that our, we're delivering high quality software for our customers and that I can get my engineers into this complex system quickly and easily so that they can concentrate on the code and the processes working around them. Do you think that DevSecOps is also part of this conversation? 'cause it seems like we're moving towards a world where there's gonna be more regulations applied to how we build software, and I gotta audit that and I gotta show that it's secure and won't that just push me towards platform engineering anyway?

That's right. That's exactly right. So, and, and that's why I said, you know, a lot of teams, they have just rebranded their DevOps team as the platform team, or they've lumped that capability on top of the DevOps team to go build.

Um, these are all one idea. It's how do I produce software quickly, securely, and at high quality for my customers, uh, and maintain all of the processes that I need to maintain for a regulatory point of view, right? All of that together, you know, whether you want to call it DevOps or platform engineering or just software development in 2024, it's what's required to get the job done.

And the tooling that we put around that is there to aid that process. How do we strike a balance between the developers, many of whom embrace DevOps in the first place, to get out from underneath a centralized IT motion, um, and the need to achieve our goals as we just outlined? How do we kind of get them to buy into it?

'cause a lot of them are, uh, shall we say, you know, they, they, they like being able to choose whatever tool they wanna choose when they feel like it. Yep. Uh, so that's, uh, one of the reasons that actually backstage has been successful as that, uh, kind of platform portal for a lot of teams is that you're giving people choice, but you're giving them a choice of a set of tools that have been, um, checked out ahead of time that have been approved by the organization.

And the ability to get new tools into that system is actually the interesting part. So, you know, you can experiment in areas that are less regulated, but in order to roll it out more broadly, you have to go through some sort of approval process. You have to make sure it's checking the boxes and actually accomplishing the goal for that particular piece of technology.

And, um, what I'd like to see is that this becomes much more standardized and formalized so that people understand the trade-offs they're making. Right? If I throw Jira out and I go to a spreadsheet, because it's just easier, um, do I lose my auditability?

Do I lose the, the ability to understand the differences between different types of issues? Do I lose all my reporting capabilities and those sorts of things? Does anyone care?

Um, so understanding how this thing, how this tool is being leveraged by the rest of the organization, that's usually the part that platform engineering teams care about. Uh, and the local engineer may not, but making sure that they understand how all these things plug together in a way that doesn't actually put more pressure on them or change their job away from developing software into managing process. That's really where good platform engineering products come into play.

Will we therefore need a, you know, as we think about the platform and what that means, um, are we gonna start to see a lot of consolidation of the tools that we've been using so far, these, these many years of stitching things together? Um, and more of the tools that we're using become features of the platform? Yeah, 100%.

Um, in fact, that's a, an approach that we've been taking with our platform wherever possible, is really defining clear integrations with different types of products, standardizing the data interchange within the platform so that we can start to draw those linkages out. And so that you don't have to build, you know, glue code between, you know, the tools of today and your tools from yesterday and make sure everything's working together. It's our job to piece those things together for you.

And that's exactly the sort of work that I was talking about when I was saying, you know, the companies usually sign up for the building, but not for the maintenance. The maintenance is actually evolving the platform. Because at the end of the day, this is a software product.

And if you're not an organization that's willing to build, test, manage, and maintain a software product for your internal developers, if you're not ready to sign up for that as a, you know, first class citizen of your engineering organization, it's probably something that's not worth your time. And you should be outsourcing it as much as possible. We hear a lot about developer productivity is a reason to embrace platform engineering.

And some of us scratch our head and say, well, that sounds great, but how do I measure that in any kind of meaningful way? Because, you know, we're well past, uh, counting lines of code. So what is the definition of developer productivity and how are we gonna improve it?

That's a very loaded question. So, um, and there's probably an answer for that that equals, you know, the number of companies out there times 10. But, um, the way I think about it, uh, personally, is really how quickly are we getting new capabilities to our customers at a level of quality that's acceptable?

And so the, the caveat on the end of that is always the quality piece because you know, you can, you can ship really bad code very, very quickly, and in some situations you actually want that. You're just trying something to see whether the customers like it. And if they do, you'll invest more.

That's one motion. Most companies actually aren't embracing that yet. They haven't adopted the feature flags.

They haven't adopted the progressive delivery. They haven't adopted the mechanism to roll it out to subsets of, uh, customers get that feedback early and use that to inform the next set of iterations. But that's where modern software development is heading.

And in that process, the goal is to actually get to higher and higher levels of quality as you go so that by the time you roll it out to all of your customers, you've got it fully tested, fully embraced, fully within the process. Now as far as the productivity goes, time to market for features that will fundamentally shift how your customers see your company. That's the real measure, and that's a hard one to get.

Um, and so if it, if the work you're doing isn't directly related to revenue, you know, uh, one of my favorite metrics is, you know, dollars of revenue per engineer. That's actually a productivity metric from where I sit. And so, and to me it's probably one of the most important productivity metrics.

So how much revenue is each one of our engineers generating through the changes they're making now, the value of the, uh, engineering dollar is wrapped up with all of the support around it as well, right? So are we building the right things? Are we building them at a level of quality that our customers are going to actually be able to use it?

Are we shipping it, uh, to the right set of customers? Those sorts of things all impact that, but to me it's really that, that dollar in revenue per engineer, that is the productivity metric that we need to work towards. And a lot of folks are saying that we're about to develop more software in the next two years than we have in the last two decades.

And, but to your point, are we in danger of most of that software being garbage? 'cause we didn't have the right guardrails in place to manage it? Yeah, that's a very high probability.

Uh, so I would expect our, our first few years with AI assisted software development are gonna be exactly that. Um, you know, I, I use copilots myself when I'm developing and, uh, which I do still get to do as a vp, which is fun. But, um, what I find is I spend as much time working with the copilot and adjusting what it has written as, uh, as the value I get out of it.

Uh, that will improve drastically probably over the next two years. But in the short term, what it's allowing me to do is actually improve my testing game pretty substantially. So it's really good at writing tests.

Uh, it's not so great at understanding the value I'm trying to deliver to my customer Is also the nature of the application's gonna be more complex. 'cause we are embedding AI models into just about everything. And this is a new artifact that's moving through the DevOps workflow, and don't we need some sort of more structured approach to managing that process because the cadence is different around that artifact than all the other artifacts?

Um, I am not sure I'd agree with, with that statement. So the cadence I think is extremely high for everything moving through the system these days. Um, you know, we're, we're shipping to production multiple times a day.

Uh, most of our customers are kind of in that same boat. The AI is evolving, uh, very quickly as well. But, uh, we still tend to have QA cycles, uh, because you have to, um, you know, almost every AI system out there ends up having a, a mountain of users behind it who are validating that it's actually producing the output that we want to, to push.

And so what I'm actually seeing is AI models are evolving slower than the rest of the code right now. And so what I'm curious about is, and it it's much more of an API driven system as well. So what I'm expecting is that they're going to evolve at completely different cadences based on how quickly we can get to higher and higher quality.

So the thing that's gonna really unlock, uh, AI development from where I sit, is when the quality management of the AI becomes part of the ai. All right. Last question.

Is platform engineering superseding DevOps or is it a subset of DevOps? Where does it fit in the whole conversational flow about, um, the history of DevOps? I think it's somewhere between, um, well, I mean, my, my personal view of this, the history of platform engineering from where I sit was way back, you know, 10, 12 years ago, people started building DevOps teams to help the engineers.

Then at some point, Gartner, and I think this was about five or six years ago, said, you know what? We need to invest in platforms for DevOps that will allow internal teams to own all the tooling and centralize so we can standardize, so you can get, you know, um, economies of scale that has then rebranded into this platform engineering concept. So to me, it's actually just a progression of the same thought.

And so I see platform engineering as just the next iteration of DevOps and, um, it's gonna continue to evolve into probably, you know, a, an AI driven version of this on top of SaaS. All right, folks here, heard it here. And one way to think about it is, um, platform engineering is a, is a DevOps methodology for managing things at higher levels of scale, and it's all part of the same family.

Hey Eric, thanks for being on the show. Thank you. It's a pleasure.

All right, back to you guys in the studio.