Enhancing Developer Productivity with System Initiative’s Adam Jacob

This is Textron tv. Hey guys, thanks. Let throw, we're here with Adam Jacob, CEO, for system initiative.

And we're talking about, well, what phase of DevOps are we actually in? Some people would say, well, there's definitely been a phase one. Some people are like, well, we're past two and heading to three, other people are saying, well, we're actually leaving one and go on to two.

And who knows, there might be a four or five. What Adam sorted out for us, you've been around a long time, where are We? I mean, it depends on how you measure.

Is that a terrible answer? Right. So I, I think that there's, uh, I think we're sort of in an extended phase.

One is my belief where, you know, and mostly that comes from thinking about it as if you look at the behaviors of the people doing the work. So not the culture piece. So I hate the split between sort of like culture and technical.

'cause I think it's mostly a lie, but if you think about how we ask people to do the work, you know, you could a good enough starting place of like the first full example of what a DevOps sort of tool chain and process would look like would be John Als Paul and Paul Hammonds flicker presentation where they were talking about how they deployed flicker 10 times a day or whatever. Um, that presentation, if you go watch it, basically has the identical workflow to what we now are doing and are still doing in the enterprise or in most of our companies, regardless of how many changes have come and gone in the tech stack. So, you know, if you look at the individual tools they use, obviously they're not using ganglia anymore, you know, but like if you think about how they do the work, like, you know, they're doing continuous integration, they were doing feature flags, they were doing, um, you know, they would do orchestrated rollouts to a fleet of servers that were, you know, cattle not pets, blah blah blah, blah, blah, blah.

They're doing all of that stuff. We're still basically doing all that stuff. We've just sort of been rolling around in what the right implementation is for different pieces of what that float would look like.

Um, so I think we're kind of in an extended phase one where we're, we're kind of reaching, I think the level of saturation where we're starting to see less transformative returns by just adding, by improving one layer of the stack or improving one component at a time. It seems like at the moment there's a lot of conversation about managing DevOps at scale and improving developer productivity and yeah, it feels like a lot of folks are talking about, well there's just too much toil in the system, too many things that we have duct taped together. So we have all these bottlenecks.

I mean, is that kind where we are or, Yeah, I think that's kind of where we are and, and to make it, I don't know, better or worse, I think that's because that's by design. Like I think the way all technological systems have been sort of built up over time by the, you know, people who had problems to solve. If you, you know, I'm 46, I got my first job working on at an ISP when I was like 15, 16.

Um, and uh, so, you know, I saw all of the rise of the internet basically. Um, and that whole community of people was basically just solving the next problem as it came, right? Well first it was like, let's get everybody on the internet and then let's put content on the internet and it was applications could be on the internet and then, you know, on and on and on and on and on.

And I think that when we look at DevOps and we look at automation, you know, we learned, uh, how to automate data centers. We learned how to do infrastructure automation. We learned how to, you know, do configuration management.

We learned how to do a lot of those things. All of those things. Then we extended into the cloud, we extended into, uh, the sort of modern era of how we work.

Um, what we didn't know was where the edges were, obviously because we hadn't gotten there yet. And I think what we now know is that thinking about the infrastructure layers, thinking about the, um, the way we design our environments and manage them as code is actually kind of a mis not a mistake, but it's certainly reached the end of its use of full life. Like we know what the returns look like if that's how we do it now.

Um, and I think we're ready. And you see this in a number of different sort of startups and their approaches to the universe. We have to start thinking differently about how the whole system is designed because we've been replacing the individual pieces of that system for a decade or more now, and the results are roughly identical.

So maybe it's not the tools individually, maybe it's actually the whole process. So we hear about the rise of platform engineering and yeah, folks are talking about is this is the new panacea for solving all these issues. And I scratch my head a little bit because on the one hand I get the point that we need to maybe manage things at scale, but on the other hand I'm like, well, is this not the revenge of centralized it?

And is this not why we embrace DevOps in the first place to get away? It's Totally the revenge of centralized it. You're not, but you're not supposed to say that.

You know, like no, it absolutely is the revenge of centralized it. Like, um, and, and it's built on the tools that you are already using, which is just even insaner where the argument is, hey, you're gonna get this like some order of magnitude improvement at scale using the exact same underpinnings you use today only with a different coat of paint that you can't manipulate the insides of because it's an uncurable abstraction or otherwise you lose all the benefits of the platform. That's our big plot.

That's the grand plan. Like I don't buy it, which I'm sure there's people out there who are like, we're having great success with our internal developer portals and like, I believe you, you know, like, 'cause we've, we've seen that movie before. We know that that's true for a while.

It's true at a certain size or a certain scale or a certain kind of complexity and you know, you can put a lot of thrust behind it. Um, but in the end, the, um, we know that the complexity kind of eat to, um, and I don't think that the platform engineering approaches I've seen so far do a whole lot for alleviating that complexity, really. And I can't help but wonder if it will just drive the developers back into the shadows because they'll sit there at home and they'll say, well, you know, you guys do whatever you want.

I work, but I got my own setup here. I got my own gear, I got my own development environment and I'll just write my code here and then I'll upload it to the quote unquote official platform and I'm ready. Yeah.

I mean, uh, I don't know if that's 'cause we're old, you know, but it does feel like it does feel that way to me. Like it, it feels a little like we know that the thing that brought about the best outcomes was how closely you collaborated with people of different skill sets and different backgrounds across the holistic delivery of an application. Like we knew that, that we know that that's the answer.

Um, what little studies have been done, you know, the Dora studies and others, like that's, it's the one thing that's always true and consistent across all of those things. Like if the number one, how often do people touch each other? How often do they talk to each other?

How often are they communicating? That's the thing, the number one indicator of their success or their failure. And I think when you look at things like platform engineering or to that movement tends to be that they shouldn't have to talk to each other.

That instead we should take whatever those moments of conversation or whatever those moments of interaction would be, and we need to push 'em behind APIs. And look, that's tempting to be like, everybody could just live in a factory behind a bunch of tickets and you stamp out your little widget and you go home it. But like, it's not a factory.

It's, it's got more to do with art and sports than it does with with factory construction. And, you know, that doesn't mean it's not tempting to think of the world like a factory. It's a, it's a very compelling analogy.

Um, but it's not true. It's just, you know, an easy analogy. So as part of this, we seem to be banging the drum for increasing developer productivity, but it's not clear to me like what exactly defines increased productivity.

'cause it's certainly not lines of code. So Yeah, and developers as far as I know, are not sitting at the edge of some work bench in a factory. They're kind of waiting for moments of inspiration to strike so that they can go and actually do something meaningful.

But how do we measure that? How do we help that? Yeah, I mean, part of the problem here is that you can't measure developer, so developer productivity.

So think of it as the difference between output and outcomes. So it's easy to measure someone's output, right? I can measure, you know, the code you wrote, I can measure whether you did what you said you would do, right?

If you, if you're doing scrum right now and there's a process that comes in and decides like, here's all the work we're gonna take for the week, and you can measure whether that work got done or didn't get done, right, that did you, or did you not achieve the outputs you signed up for? It's a lot harder to say, did that work achieve the outcome that what was desired? So I we, why were you doing the work in the first place?

Who was it for? What, what benefit were that was that person going to receive from having made that change to the code base? Did they receive it?

How do you know? And so thinking about designing our processes for outcomes instead of for output is, is kind of where that leads you. So when we design processes for, to measure and to, and to optimize output, we tend to design them to ignore whether or not the outcome is good or bad.

So, you know, a good example of that in the world, everybody has been on this team where you worked really hard for a year, you did everything you were supposed to do, you shipped it and no one cared. And sometimes, uh, sometimes that happened because the, there was no conversation. But mostly it happened because you didn't connect soon enough with the people that you proposed to help, right?

Um, even though you did all the output, right? Right. You put all the effort in, you got all the stuff done, but it didn't matter because the outcomes were bad.

And so when we think about the optimization, when we say we're dev, we're, we're optimizing for developer productivity, it's kind of a ridiculous thing to optimize for to some degree because what we're really looking for is developers being able to focus on the outcome being great. And how do we define that? How do we communicate that?

How do we share with those, how share the context with those engineers about what it is they need to do at the heart of DevOps, that's what it has always been about. How do we get people together and share that context of what the outcome is we're trying to achieve and all the different parts and pieces that each person plays in that game of delivering that outcome to those people, right? And you know, when we think about it, when we start to break it into silos and we start to say, well, developers do this part and we'll measure their throughput, and then these people will do this part and we'll measure their throughput and together it adds up to victory.

Like it tends to not, it's kind of a roll of the dice, you know? Well, you've mentioned this several times now where it is kinda all about, uh, the level of collaboration between the, the people equals value. Maybe we'll call that the Adam Jacob law from here on out, but we'll see.

But, um, I certainly didn't invent that law, but, you know, But, but within the concept of DevOps, we're also talking about AI almost every day now. So where will the machines fit alongside the people? Yeah, I am, you know, it's a, again, it's an interesting question.

So I, I think you have to look at where we are today in those pieces of technology and uh, and then say how do those things today impact what it is that we do? I think today even fantastic LLMs tend to be not the most useful for large scale tasks. So, or, or tasks that require real precision, right?

They're mind blowingly incredible at doing creative tasks, right? Uh, because computers couldn't do creative tasks before, right? I couldn't just tell the computer be creative and then creativity hap that like wasn't a thing that computers could do.

So now it can, and that's incredible. And as we get better and better at that thing, it can, to sum, approximation, look like it can solve much more complex problems and more finite domains. I think today we tend to lack the feedback mechanisms that would allow you to actually make those kinds of systems work very well.

So for example, I can ask an AI to write some terraform for me, and that's cool. If I don't know what Terraform is or I don't know what AWS does, then I can't actually do a whole lot with that or tell you whether it was good or bad. I need, I still need an expert to be in the loop.

And that expert I've tended to throw into the hardest part of the problem, which is troubleshooting small minutiae, right? So if you, if it invents something or it gets something a little bit wrong and it's just a little wrong on the edges, well suddenly we went from, yeah, it sucked. I had to type all that terraform to all this terraform, I didn't, right?

That I'm not quite sure I understand every line of I have to go find the needle in the haystack bug. And that's actually worse both from a developer experience point of view and from a time point of view. So we haven't really developed the systems that understand what the objective is we're setting out for those LLMs when it comes to things like infrastructure.

So saying, you know, I want to use AI to build me infrastructure. Sounds great. Infrastructure only really works when it's correct, right?

If I, if I deploy incorrect infrastructure, things don't work. If the plumbing in your house isn't plumbed true, water pressure sucks, it doesn't work very well, it's leaky like it f*****g sucks. And you rip it out and you figure out sorry for wiring and you figure out how to like make it better.

Um, and we're kind of in a similar situation right now with ai. So most of the time when I hear people talk about how AI is gonna transform, you know, DevOps or it's gonna transform more of those fields where, where precision tends to matter in the end, uh, and where there's a quite a big variation in exactly which pieces of precision matter. So it's not like there's one piece of precision that matters for all of us, it's actually that each of our individual applications have slightly different metrics of what the precise correct configuration for that thing happens to be.

Um, it tends to be that when you dig into the details of exactly how what you get is like hand waving, you're like, through the magic of the ais, we don't understand and more power, it'll work itself out. And as a technologist that is unsatisfying to me. Um, like I need a little more than just and and thrust.

Um, so yeah, right now I don't think it's doing a whole lot for us other than, um, kind of telling us being able to feed us information. We already know a great example, um, I use it all the time to answer questions like, how do I write this recursive query in Postgres? 'cause I know that that's a thing I can do and I even could under, I could remember how to do it.

That's a lot faster to just feed the table structure to chat GBT and ask it to write a recursive query for me, which it will do and it will be a little wrong, but it's close enough to right that it's, uh, much faster than Googling. So like, it's not that it's not useful, but it's not gonna, like, it's not gonna change how we do it or, or alter fundamentally the the shape I think not in the current state. And then given week, how much time did that actually save you?

An hour or two? Um, yeah, it depends, right? Some weeks, some weeks I'm, I'm not a, not much and other weeks a lot, right?

Um, I think, you know, but, but the, the argument that's like the AI is going to obviate the need for all of this DevOps who ha anyway is insane. Like obviously it won't, um, not without significant leaps forward in technology that I have not seen, which doesn't mean that there's not somebody right now who's doing that, but I haven't seen it yet. Is there a tendency to trust it too much?

Because it will convincingly tell us whatever it is it's presenting and we tend to look at it and go, well, that's awesome and away we go. Do we really have the understanding that it is probabilistic rather than deterministic and no makes a difference, But, but obviously you're gonna, so like, like these conversations right now tend to be on the margins. We talk about them like they're not, but they are, right?

So like who right now today is using AI systems to run all of their operations and replace the human beings in the loop? No people, zero humans, no one's doing that yet. We're talking about it as if it's going to happen, but, and it might, but it's not happening today.

Now somebody is gonna watch this and be like, we are like, and okay, sure, but you're not, not really, right? There's, there's still a huge number of caveats there and like, and there are expert humans in the loop almost always. And that's the sweet spot for it today.

It just is. Um, that doesn't mean that with more technology or different feedback loops that you, we we're not gonna get to a spot where those same systems could actually produce the right answer because the right answer is a thing they can observe in the world. Whereas right now, like LLMs have no conception of the right answer.

They don't, right? It's just, they're just telling you the next token based on context. And so like, it's, it's pretty difficult to be like, Hey, here's the system I need, it should fit in this way.

There's some interesting papers for doing like database tuning for example, where, uh, where they do this and it's, it's pretty good. And the results are like 80%. I think, uh, don't quote me, I'm not, 'cause I can't quote the actual paper, but my, my memory says it was basically like 80% is good as a like professional level, DBA at tuning a database after a couple hours of observation, which okay, right.

But that works because we can build a finite thing that's like, here are all the knobs, here's the way the knobs interact, here's the way that the system could then change those knobs. We can observe those systems and we could decide what to do. And like, you know, we'll have to build systems that work roughly that way for everything.

Um, and then we'll have to decide whether we like the outcomes and we'll have to see if they work or they don't work, and we'll have to understand how to fix them. None of which we understand today. So like, you know, sure, AI eventually changes how we do all of the things, but not the AI we have today and not in domains that look like this.

So coming full circle. Yeah. Now that we've talked about all these things, are we still at the end of phase one or is there another phase or where do you think we this, you know, well, I think we're, when when does phase one end?

Well, I think we're at the beginning of, I think we're at the beginning of what I've been calling the second wave of DevOps. Um, so we could call that phase two if you want, but I think there are people who have come to not just me who have come to the conclusion that the way we approach this problem is the now the problem. That it's that we have to think differently about the fundamental shape of the system itself.

So like system initiative, the way we think about that is by turning the whole system into a big hypergraph of functions and then using all of that, making those functions reactive to the inputs of other pieces of the information. So we basically, rather than think about it as code, we think of it as as reactive data that you can then build simulations on top of to give you like new user interfaces and faster feedback loops and a bunch of interesting stuff. It's taken many years to build and it's, you know, getting close.

Um, you know, summer I think, um, is when you'll be able to like use it in anger. Um, and I think the, you know, you can look at stuff like ELO and the people that make wings, um, where, you know, their approach is what we need is a programming language designed to use cloud primitives. And that, that eventually becomes so powerful that it replaces the way we think about building traditional applications and the line between infrastructure and, and language blurs.

Um, and they too build a simulator. Super interesting. Um, the guys who make Q think about it as like value lattices.

So there's like this set of people who are thinking differently about the fundamental shapes of the primitives that allow us to then build this kind of automation or build these sorts of systems. Those people will eventually figure it out. Um, I don't know if it's one of us, if it's all of us, if it's, if it's me or Eli, who knows whose bet is gonna turn out.

'cause it's a difficult problem. But, um, but I think we're entering a phase where, you know, it's early and a little punk rock and still a little counter to people's intuition. Um, but, but I think has lots of possible upside.

And so I think, I think we're the beginning of what will become a phase two, but you won't know it probably for a while because the people who know, you know, it's like, when did people say that punk rock was taking over the world is probably a decade after punk rock started, you know? Um, so like, it's probably gonna be another five years before everybody's like, it's a second phase of DevOps, you know? All right.

And here I was at a social distortion concert, not just last week, right? I love social distortion, right? There you go.

Like very mainstream now. Social distortion. All right.

Hey, there's an old joke that says, you know, the future of it is, uh, one person and a dog, and the dog is there to keep him from touching anything. And that was funny 10 years ago, and it sounds like it's gonna be funny for years To come Funny, still funny today. Yeah, I think so.

Yeah. I just, I think, I think it's a, i I actually think that right now is probably the most in one. Like it's the most interesting time in DevOps since we started in the first place.

Like the fact that we now have enough evidence and we have enough experience and we have enough, uh, data to tell us that the system as designed doesn't work now means that we can use all of that experience and all that information to design new systems and see if they work better. And that is, I think, just like incredibly exciting. Um, the only real problem is that we're not all going, most of us aren't going far enough, like we're stopping short of actually really fundamentally rethinking the way the system could work.

And instead we're just continuing to nibble around the edges. All right folks, we need to learn from our mistakes. Hey Adam, thanks for being on the show.

Oh, it was my pleasure. All right. And back to you guys in the studio.