Software Dev Metrics – Jeremy Freeman, Allstacks

This is Textron TV. Hi, I have the great pleasure being joined by Jeremy Freeman. Jeremy is co-founder and CTO with all Stacks.

Welcome Jeremy. Hey, it's great to meet you Mitch. Thanks for having me good to meet you too, talking to developer tools, the engineering manager information development software at the door favorite topic of mine before we get into that.

I'd love to have you tell folks a little bit about you and tell them about all stacks. Yeah, sure. I would love to so I've been in engineering moved into engineering leadership for you know, almost 15 years.

Now. I've worked mostly at startups interestingly enough going from nothing to actually having to build a real team and solve some of these problems and for brief scent actually did Consulting with my now co-founder with and we got to go out and actually see how many different teams work. It was a great learning experience getting in the room with leaders integrating and teams sometimes for only weeks or a month at a time and figure out how to get in understand what's going on be productive.

There's just something that we struggled with and that, you know, ultimately led to what we build it all Stacks, you know, one of the challenges we ran into is Even the leadership the people that did it day to day struggle to understand exactly where things were and exactly you know, how teams were onboarding getting set up or productive and we realized that with the Advent of all these tools and everything going through SAS products and platforms these days all that data was readily accessible and it just made sense to make it easy to access for leadership so that they could be more effective leaders. But it's not like everybody is automatically headed in One Direction on software teams more or less multiple teams, right things are changing constantly agile about successora. So it's sort of shifting sand, right?

Everybody's got running across it now trying not to sink. So actually given all that as you build out all stacks and created a product for what did you see as some of the biggest challenges is the engineering managers are getting onto an IRC Channel or slack or whatever they do or call each other and zoom and say well, I wish forget this out. What are those are what are some of those challenges?

Yes, it's fun. So, you know, we we entered the market at a very interesting time people have been trying to measure Engineers for you know longer than I've been an engineer longer than I've been alive for some of these measurements and their struggles with everything a lot of times, you know measurements tend to get weaponized by leadership. So people are Shy when someone comes in it's like oh we need to count the number of lines of code that we're committing.

So we know how productive our devs are not like, whoa. Hold on a second. That's that's not lots of lines of code if that's right.

Exactly. Yeah, you can you can drive not just really bad outcomes, but really toxic cultures. And when you know we started thinking about the problem it was a balance of okay, there's some good signal in the state of it.

You have to make sure you use it the right way. So there's a large part of sorting out just the volume of data that you generate from all this activity to what is actually meaningful for your team. Right?

And that's I think where we found The magic we were talking to leadership, like can we pull out metrics that are meaningful at the right time and actually guide you to making the best decision for your team? Mmm, you know some of the metrics we've evolved to be on lines of code and there been many function points and you know, lots of made a made-up ideas everybody trying to figure this out but now it seems like we're focused on how many deploys do we do per day hour week, whatever the measurement might be half an hour checking in code. You know how many times whether we're passing your family tests, you know, sort of standard measures like that, but they're still valuable measures or they're better ways of solving this problem.

So they're they're all great. I think that's the the interesting thing about what we're doing is I think about measurement in engineering as more of solving problems for today knowing that you know, we're Engineers. We love solving problems.

Like let's figure out how to measure this problem. Let's spend a month or two solving it get it to a good spot. And then we need to move on to the next problem.

So like you can pick different measurements or kpis solve them move on to different ones and that's sort of the way the world works. It's it's really hard to find one measure that's golden that works for everybody. But there's a couple that are really helpful, right?

So, you know the door metrics are really great for devops teams for engineering organizations knowing how frequently you actually should code implies a lot of organizational agility. And those types of measures are really good. barometers of how how sophisticated your team and your engineering processes are It seems also like there's an organizational agility.

Right? Some folks are not at a place where you know shipping code once a week would be a big challenge right just and it isn't a turn the dial from one to 11 and poofier you're shipping code multiple times an hour. It is a process.

I mean, yes, you have to have the tools you have to have the process you have to start. You know, we're doing work in smaller bits smaller chunks right to be able to deliver that maybe even you know, think about the architecture of your application and it's coding. So, how do you how do you if you're gonna build a tool like you have with all Stacks?

How do you what day did you take and ingest and look at or what do you put on top of that data to try to help make some sense? yeah, it's there's a lot, you know, we we've started with your code repository your coding activity and your ticketing system. That's usually where the bulk of your engineering team is engaged and you know, we've started to Branch out to other places.

So you get into your build systems and we get into communication Tools in some cases and you know further down the line, we'll start expanding into other operational metrics potentially like your infrastructure monitoring or your roadmaping tools even And the nucleus that we have inside of the engineering team is a really great place to start because one you need to know if you're actually delivering code and there's some really basic measures you can start with the number of pull requests that your team can get across. The line is a really good one a lot of teams think of that as their basic increment Then you can move on to are you getting tickets across the line then you start looking at how that maps to you know, organizational value delivered. So, you know, are we getting epics or Milestone shipped?

Are we slipping? And then on the other end, you start getting into deployment efficiency, right? So, how often do you deploy?

Are you constantly breaking the app and if you break it how quickly can you fix it? and those those help with what I think of as the other delivery kpis that ensuring owns for the business. Can you get your product out there?

Can you deliver service to your customers? And can you do it in a consistent reliable way? But it's interesting too.

There's a lot of A lot of emphasis on customer experience and how we can kind of tie that back into metrics that are happening through the workflow pipeline. Right? Whether it's through an observability kind of platform or even further back in a designing for Customer experience.

It seems like thinking about that early in the development process and what kind of metrics if might be performance oriented. It might be. You know failure types of situations or break testing or something like that that helps with improving a customer experience.

So it seems like that's something to consider putting together your metrics. Exactly and mapping those is not a non-trivial exercise for leadership. Right?

It's very easy for you know a customer to come in or your product can maybe well aware of say performance issues on a report for example and say, you know, we need this report to be better performance and there's an easy kpi there like maybe it's page alone time. But only under the hood. It's really hard to take like make this page better.

To the engineering team. It'll need to be broken down. You'll need to like identify the different Services.

You'll need to understand are we continually getting better in various sub components of this page? And I think that implies, you know, every other metric if you start with a customer experience, you can develop really good kpis, you start deconstructing those and eventually you can tie those to the activities that your team are doing like if we're constantly shipping performance improvements. On a regular Cadence, we assume or we can track that that's going to improve this actual customer output.

You know just Google we're introduced their web page metrics, right? We're all used to seeing the green and red bars, right even that you realize just a page load metric and what all goes into age may not be your code could be the services you're accessing or how your accessing it and all those kinds so like you said, You know any metric has a lot of depth into it. What are we talking about?

And what context and where does it occur in our process? Right? It also seems like you it's easy to fall into what I call it.

You usually call Vanity metrics, right? That's code. I'm going to ship.

You know, how much code did we ship every time so we put into production which are still maybe valuable metrics in some way but they aren't the thing right there. Ultimately. It doesn't I could trip a lot of really bad code if you want a lot of code to go into production.

That's not the point, right? Yeah. I have this is a really great a little snippet that my my co-founder who's CEO made and it was basically, you know a CTO coming to or coming to the CEO saying hey we shipped 96 times last week.

and being really happy in the CEOs like so what was that mean? I don't I don't know what value that means for the business. And yeah, that's exactly the type of vanity metric that I've even Fallen prey to like building a product that's supposedly helps.

You avoid this. It's so easy to get lost in the details sometimes and making sure that you stay on track and can map all this stuff back to business value. is hard so I imagine one of the one of the hardest things just getting started, right?

How do you go from where you are with maybe limited or or maybe too much data or no visibility or both? Giving you have so much information as folks start to work with a company like yours. What's the place to start, you know and devops at CI/CD and when it comes to kind of thinking about engineering the software process and how we manage that.

Are there some good starting points? Yeah, so I I love starting in places where things tend to get stuck. I think if you every one if you sit down and look at your development pipeline, you're like hmm things tend to like hang out here and don't move forward.

So if you look at if you look at really common places where things tend to get held up. You'll find you start focusing on PRS. So it's very easy for your code to get stuck in code review land like oh someone made a comment.

Oh, it didn't get seen now that PR is just sitting out there for two weeks. Like making sure you catch that and get those things across the lines are really easy place to start and you know, if you're a new team if you're like just reshuffled and restructured your teams, or if you just have a thought about this holistically as an organization, you may start measuring that and find oh God. Why why are we taking so long to actually get code merged?

Mmm, that's a really good place and then, you know similar in a similar vein. You can move on to maybe your QA stages. You know, we actually get code to code complete but then our QA team is so backlogged.

It actually can't review it for weeks. And those are the types of places that are great to start and a lot of times I would imagine you have a good hunch as a leader but starting with PR's starting with your QA cycle times always a good place to look if you're worried that you're your team's efficiency is struggling. I gotta believe there's a really good chance when you go talk to someone they're gonna have pretty good idea.

At least we're some pain points are right whether it's QA or yeah, it's interesting. You you talk about finding the bottleneck too. So that the theory of constraints right go find the bottleneck and then go from Bottleneck to bottleneck.

The other also is just hand-offs. You're talking about PRS and waiting for somebody every time there's a handout that's a delay automatically. It has to delay if you're handing off to somebody because we aren't we are in codebots, right, you know immediately pick up on exactly so there's a lot of places where you can start to examine and maybe even get the data, you know from from a tour from a platform back here to help you say.

Well, here's where the wait times are right letting a lot of time. And regardless of how you I'm sorry? No, no way.

Yeah, let's just regardless of how you actually track those things. You know, like you we have an alerting mechanism jira built-in has an alerting mechanism. You can have nudges and all your Source tools, but those are the types of things get really help you keep track of those once you sit down like identify.

Oh, this is a problem. It's very easy to say. Hey just ping me when a PR sitting out for three days.

And there's really easy fixes that somebody's problems that can result in Saving weeks maybe even months off of delivery times for some of your major features. Very good. We're working folks.

Find out more about all Stacks I can guess. I'll let you know. I yeah, thanks for asking.

Um, you know hit us up at our website. com come check us out. You can sign up today right now get your account set up immediately and then we can help you with any other questions you have as you start building out here measurement process and your assess service.

So not software to install on-prem or is that an option too? We don't have it on-prem offering but we we do we're fully Cloud multi-tenant, but we do have ways to connect to your on-prem services or wherever your services live. Yeah, I would imagine some people might be still running here in the cloud or yeah, right.

Yeah. Yeah. It's pretty common.

That you're integrating. That's well good, and there's this like a free account to check things out. Yeah, you can get set up first.

Okay, very good. com And appreciate Jeremy you come by and talk with us today and let's share with you know, there's a lot of emphasis on developer productivity. Not that they aren't predictive already, but we're always looking for how do we hope folks focus and good time to work do that work.

So it's a great topic and enjoy talking about it. Yeah. Thank you very much Mitch was great talking to you you bet.