Going Beyond DORA Metrics for Deployment Success with Planview’s Mik Kersten

This is Textron tv. Hey guys, thanks for the throw. We're here with M**k.

Kirsten is the CTO for Planview, and we're talking about Dora and the metrics, which I feel like has been with us for a long time now, and we all maybe take it for granted. But M**k, I guess my question to you is, as things have evolved, are the DORA metrics as relevant as they once were, or are we maybe overly obsessed with the DORA metrics as they are? Yeah, I think that it, it, it's definitely, people would tell you it's situational, right?

It, it depends where organizations are in their journey. I think there's just been this challenge and misinterpretation of those metrics in the sense that they can be extremely useful when you've got a constraint in your delivery pipeline, right? When you've not created the kind of automation, the kind of DevOps practices, uh, and, and tooling that you need in order to quickly, safely, and easily de deploy, code and deploy value.

Uh, and they're great for that. So personally, I think every organization should, should measure that, um, and should measure the, to our metrics. And I think what happened is because they were easier to measure, organizations relied on those metrics to actually be the proxy for their, their entire, uh, digital and, uh, product delivery performance.

And they're just not that, right? You're measuring this, this tiny slice of the value stream that's can be a very meaningful slice if you're not apply the right DevOps practices. Uh, and if you're not measuring the bottlenecks and challenges you might have upstream or downstream of that, well, you, you actually don't understand what you get going.

You need it faster or not. So I'll give you a very copied example, uh, on my teams. Uh, measuring the poise per day is completely meaningless, which is one of the main DORA metrics.

But if you're able to deploy on demand, what does it matter how many times you deploy a day? All that matters from a customer's point of view is what did you deploy? How much, you know, what cool new features were implemented or what, what really critical bug fixes were fixed?

Were resolving how quickly. So they're necessary but insufficient is, uh, and, and I actually think they have been, uh, used for the wrong purposes. Like I've, I've encountered many our enterprises who've bought the Dora metrics on their executive dashboard for technology.

That just makes no sense. Do we mean to therefore update the DORA metrics or create a separate set of metrics that go in alongside them? Or how do we create a set of metrics that are maybe a little more relevant to the business?

Right? So I think first of all, I think we should take that the, the DORA metrics and knock on telemetry as we don't need to update them. They're great for what they're meant for, right?

Where if you've got, again, you, you, you're behind on your investment in deployment and pipeline automation, um, or you're not able to successfully deploy changes. The door and metrics need to be the things lighting up in red that you need to address, and you should address those things likely before you address a number of other things. Uh, but we need to layer above them.

And it's a little bit similar to agile metrics, right? Like burn, so let, let's just say burn down metrics, right? How, how quickly a team's able to burn down story points or stories from their backlog, um, from what they've planned.

We shouldn't take that away from teams. Like I really wanna make sure my own teams are able to use story points for prioritization, get better and better at it. But that's certainly not a metric that I do present to our executive team on how our product portfolio is doing, right?

It's, it's just too detailed. So the, the key question becomes is and mighty data metrics are, are necessary there, uh, and they're very thought for what they're good at. Uh, agile metrics are, are really valuable, but how can we layer a set of metrics above that and there's actually more relevant to what the customers are receiving in terms and, and more, more relevant to business outcomes that we're trying to drive.

So, uh, and not basically repurpose Dora metrics or other agile metrics for that. So Is this another example where maybe, you know, we in it are obsessed with a set of metrics that, um, we didn't really connect to the business. And so when we go and have a conversation with the business about how often code is deployed, they kind of might, you know, they look at you and maybe nod their head, but it doesn't really mean anything to them.

Yes. And I think that's, it's interesting because there, there, I think two things that have happened. One is there's, there's been a lot of, I think effort, but into educating business operations finance people.

So why these metrics are important, why you need to care about, um, me time to repair and there's, there's, there's something to that, right? I think it's actually healthy and positive to elevate some of the technical metrics to the business for a company that's, that's e either is or is turning into a technology company. Again, the challenge is whether to fine grain, right?

I'll just, I remember when I first onboarded this decade ago, our CFO who'd actually worked at a lean of them, produ produce physical goods, like do winglets for aircraft. And he said, how do you measure your, your output? And I said, there's, you know, well the teams measure story points.

And he goes, that's, that just sounds like you're like in some kind of fairytale with the aircraft story points. But anyway, and I realized, okay, we, we, we need just the same way that he was used to measuring how many winglets got through their factory, uh, we need something similar that's more end-to-end and more of the business customer perspective that layers above all the various, uh, metrics that we've got. And I think a really big part of the, the reason that we do here is most things is organizations tend have been measuring what's easy to measure.

So the stuff they can get outta Jira stuff, they can outta jets, all these various tools rather than actually doing the thing that takes a bit more work, which is measuring things from a customer's point of view, from a business point of view, How much that do you think is, we're just comfortable measuring the things that we kind of know that we're good at and we're not really interested in measuring things that maybe are more outside of our control. Yeah. And I, I think, yeah, Mike, I think that's a very big part of it.

So, and it's actually the flip side I think of what I said, which is where measuring means they're easy to measure. Some of those things are the things that, that make us look good. Like, oh, our team closed this many stories, or our, we've got our deployment of x, uh, it's, you've, yeah, it a positive sign of that team or the automation effort that that took place.

Um, but it doesn't actually address the, the harder problems, which is, well, how, how fast is work flowing through the system? What's, what's, what's the end-to-end flow time? So the, I I do think it's been like that.

I think this is a really big part of this, is that divide we have between business and technology where the technology teams have created metrics that make them look good, but that don't actually indicate, well, where's the bottom? How can we address the things outside of our, to your point, outside our control and deliver more value to customers by doing that? So, and I think it's back to this balance, right?

Where so often what we see is there's, there's just this humongous gap between kind of the ideation and all the work being produced by the business, by the product managers and so on. And in this constraint and the delivery teams. So the, and the technology teams, and if y'all have a metric that combines those things, you'll continue these, this functions that throw things over the fence.

Or really what you end up with is a se separate to the metrics, right? The, the business and product people are measuring net promoter scores and revenue and retention, the technology people are measuring uptime. And, um, you have story, story point completion and, and the door metrics and, and there isn't actually a customer view in there.

It's just like, no, we're doing great. No, we're doing great. And, and you're not getting the, an actual view of where the actual actual, where the problems are.

We've seen this effort to kind of close the divide a little bit between DevOps and the business with the rise of things like value stream management platforms of which you guys are a driver. Um, but it seems like there's a, a gap between our DevOps processes, door metrics in the business. So are there things that we should be doing on the VSM side of this equation that would surface the right metrics?

Yeah, yeah, absolutely. So in my, in my view, bio management both needs to answer this and, and really is the answer to this. We need to measure our value streams at the end full stop.

So the question is, in addition to the door metrics, the, that we've got, how can we measure our value streams at the end? Um, you know, for me that for now a lot of organizations that has been the flow metrics, let's layer above that measure flow and see what was our flow was, was our flow time, just time to market from the customer's point of view, right? So, um, so that's it.

And, and I think that's, though, I think the key thing is, goes back to your, your previous question, Mike, which is are, are we just measuring the metrics that make us look good? We, we should measure the metrics that show us where to improve, because so often the improvements have to do with those hand ups, hand up between business and technology, doing technology and actually getting into operations and then getting the customer feedback that we have enough telemetry to see was this thing we built actually used and so on. So we need those end-to-end metrics.

And the key thing is that it, it's, it's not enough to say one side or the other measures them. You actually have to measure them together and then you actually have to a process where you are measuring them together, you have to have some kind of, where the business review incorporates all of your value stream metrics. So yeah, the whole point of value stream management is, uh, for the organizations that invest in deploying value stream management, uh, they actually didn't have these metrics.

And then of course the next step is to incorporate those into their operating model. And that's really been the, the, the really the big significant product operating models is they incorporate value stream metrics into how we operate your technology business. So, You know, and Todd, right, we have metrics for managing value stream and almost every aspect of the business except for software.

And yet we see more businesses than ever saying they're dependent upon software and they happen to be a software company that does X, Y, and Z. So what has been the challenge in getting everybody to wrap their heads around value stream management, uh, as applied to software engineer? I think it just goes.

So I think it is surprise to me, there's site surprising how challenging it has been, right? Is we're now spending some very significant portion, sometimes over half of our cost based on technology. Why don't we have an effective way of measuring technology outputs and outcomes, right?

It's, it's, it is a surprising thing when you actually look under the covers of large enterprises, uh, the ratio basically that the amount they spend and the fact they, they're only good at measuring the cost, the cost sign of the equation. And that's really the challenge, right? Because you ask any CFO, they know exactly what they're spending on technology.

Exactly. You know, how many staff they have, what they're spending on outsourcing their contractors and so on, but there's no value metric. So you'll know what they're getting for that.

Are they, when when, let's say there's a big, uh, offshoring or nearshoring exercise that's done, um, or cost and exercise that done what happened? No one really has the answer. So it's just amazing that so much, so many of these very strategic and consequential, this consequential decisions around technology are just made through gut feel.

And that's really the whole point of putting it in place value stream management, is that it provides the data to understand the impacts of those decisions. When we round up this news that area, did we actually get the flow that we wanted? And it really is just all as simple as, as, you know, envisioning yourself as a, let's say a manufacturer.

If you're building cars in the region, how many cars come up? This one, how many customer of that plan, what support that you have to get that plan? What were their constraints?

Did they get all the parts they need on time? Just the same way, let's say if you open a new, or you're moving a bunch of your developers now to be AI assisted in India somewhere, are you giving them the right support? Are they getting the right training?

Do they have bottlenecks on your headquarters? That's, um, but that's in North America or something of that sort. So really the the whole point of value management is to, to, uh, make these decisions be data driven because all of a sudden you get then get a much, much better higher fidelity feedback loop on, on these very consequential or organizational decisions that you make.

So, mm-Hmm. Um, you guys just recently acquired the tour, correct? Correct.

And so where does that fit in the spectrum of things? 'cause you were both kinda in the VSM space a little bit, so I guess what's next? That's right.

So it's kind of, it's interesting is I've, I've known in long a lot of respect for D board on who's the, the CEO of Torah. And he's, I think, just done an amazing job building a company that I'm very good at, the release management part of things and as well as the, the DORA metrics and other editing metrics. And for the longest time I was personally saying, okay, well we need to get companies again to stop over fixating on door out.

They're necessary but insufficient and focus on flow on end-to-end Valley Street flow. And then, you know, through conversations with LIBOR and looking at our own strategy and that way I kind of gave up. I thought, okay, well organizations need, you know, they, they need both, right?

Why we need to be the OneStop shop for both. If they, rather than telling them, stop focusing so much on Dora measure flow, we can now say thanks to LA Dora finally coming together, we will be your one stop shop for both Dora and Flow. And then on one point of glass you'll actually see, okay, our bottleneck is not deploying them.

Let's look over here or on this part of the business, our bottleneck isn't deploying and so let, let's go address it and let's bring that back into our OKRs and our roadmaps. So we actually are addressing it, uh, and it becomes a part of our plans and, and we've got that closed loop. So yeah, I think the key thing for us is at weak point of our value stream management platform at Planview HA was the, uh, more insights into the engineering and Dora metrics.

And so we really, we realized that was, that was a gap. Of course, a lot of our, um, uh, our history on building up this platform does come from developer experience. One of our goals is just to make developer experience better by removing these izer and to be upstream of developers or across different development teams in value streams.

So yeah, it's just been great working with Ian team and, and bringing these things together so we can have a platform that can pinpoint the problem and again, close the loop on helping you address it. So whether it's in deployment or, or upstream ideation, um, Do you think gen AI is gonna bring all this to a head? 'cause it seems like we're gonna be building and writing more code faster than ever and all this stuff is coming through these pipelines, and the question is to what end?

Right? And I think one of the challenges, gen AI has the potential to have a ton, lots more code over the pipelines and the que, you know, so I think there are two questions. How much more values come through those pipelines?

Uh, and what's the bottleneck to driving even more value, right? So let's just assume for the sake of argument that today and the enterprise can get 10 to 20% productivity gates in terms of lines of code generated through, uh, j ai, like a give out copilot or, or Amazon code whisperer. Um, is that code actually getting to customers is one question?

Um, or is there something else that's blocking it? Because the challenge that we see, and we just ran the 2024, uh, stated project product industry report, is that for most organizations, the bottleneck is not the development teams, right? The development teams are actually, especially the organizations of scale and hire those developers, the bottlenecks tend to be upstream and, and dependencies between teams.

So if you two, let's say we're now doing, we're now even further ahead on gen ai, and now you can get a hundred percent productivity gain if you're constraint on your development team through these co these copilots that generate coding applications, if that's not what you constraint, you're not gonna get any benefit from that. And the challenge is that you ask, uh, organization, technology executive state, and they, they, they just won't have the answer. They won't know if they just deploy gen ai, how much more value they get into customers.

How much was their velocity in terms of what got delivered, um, uh, improved. So the key thing is, you know, it's two things. One is that we, we do already able to measure value streams and measure the positive impact of gene ai, those value streams if you're not getting positive impact to figure out where you gonna fix the bottleneck to get that impact.

And then of course, the key thing we've been focusing on in our value stream management platform, um, as well as the product and portfolio management portions of it, is to apply general AI to help you find that bottleneck and resolve it as well so that you actually, we actually wanna get to the point where the dev teams are the constraint. 'cause we know that gen AI has such a big potential for, um, accelerating development there. All right folks, you know, there's an old saying that says things measure or things done, but if you're not measuring the right things, you may be doing nothing.

Hey, thanks to being on the show. Exactly. And great, great, me to close it.

Thanks so much, Mike. All right. And back to you guys in the studio.