Software Engineering Metrics – Dylan Etkin, Sleuth

This is texturing TV. The great pleasure being joined by Dylan Atkin Dillon is CEO and co-founder of sleuth welcome Dylan. Hi, it's great to be here.

Appreciate it. I'm excited to talk with you. This is a great topic and you know, leading a software team molding several soccer teams and you know development and just is great to talk with you because I'm really curious and want to find out more about what you're doing at sloot but also kind of what's happening in the market with efficiency and delivering code and all that kind of thing.

So, but first would you introduce yourself just a little bit about you and also tell us about sleep. Yeah. Absolutely.

So as you said, my name is Dylan Atkin, I am co-founder and CEO of sleuth. I spent keep some heaps of time at atlassian. So I was fortunate enough to find myself in Australia back in 2005 ended up being like the 20th employee at atlassian.

And one of the I think it was like third developer on jira and you know had an amazing ride with that organization just going from startup size to you know, make a Corp. And you know got to work on a lot of really exciting things always sort of been in the developer tools space. So about five years on jira working on jql and a bunch of other hard problems within the application kind of moved myself and my family to San Francisco with atlassian about the time that they purchased it bucket.

And did about again another five years since leading engineering on that. Product which was really exciting. I don't know if you remember back in the time but git was quite a thing GitHub was quite large pull requests had just sort of come around and so it was an exciting time to sort of be pushing.

You know, how we work as a software developers forward that's about the time where I started to have ideas about sleuth or what would become sleuth. You know, we were talking a little on the pre show and it was one of these things where I found myself I was you know, running the engineering team for building a Premier Source Code hosting tool we were doing pull requests. It was the jira company and I would get questions like hey, what did you guys ship in that day or you know, what was the impact of that on your customers is your way of working more efficient or less efficient than these other ways that we're working inside the organization and my answer to all of those things was I don't know and so that that didn't seem appropriate it seemed like maybe there was a gap there fast forward.

Years another startup experience but eventually the idea for sleuth really the market kind of evolved in sort of got there. You know, so many organizations are using cloud-based source code hosting things like pager Duty for incident tracking, you know metrics tools with data dog cicd systems that the groundwork was there the sort of like critical mass was there to be able to solve a hard problem which you know, we've been I think struggling as an industry to solve for quite some time and that's putting hard numbers to engineering efficiency. Yeah, it's a describe it as the tool.

I wish I would have had when I was doing those jobs, right? And so that's a great. That's the entrepreneurial Spirit right?

Go build what you need because you know, at least you have a customer of one you find out if the markets there which yeah, like you're in a great space too it in my case. It was one of these things where I'd had this thought and I would have shower thoughts, you know of like wouldn't it be amazing if this or that and you know, I'd like talk to peers in the tech industry and when I would get really genuinely excited was when I would talk about the potential here, and and I think that's like a very good sign that maybe maybe you should just go off and do it. Well fantastic.

So so let's talk about the whole issue or NF is an issue, but the focus emphasis on developer productivity, you know times are really buttoned down gotten tight folks are looking, you know, they want more for less or at least more out of their teams and make sure make sure they're getting the most I mean and plus developers hate wasting time on Sprite, right? I mean they want to be doing great fun things but producing fun things at the same time. So back to me a little bit about your sense of the market where you're from customers, we say developer efficiency shipping code getting effective at doing that what are some of the challenges people are wrestling with Yeah, so, you know, it's not a new industry, right?

There's been a couple of false starts or even starts that were like moderately successful, you know, and I mean, I like to joke, but I'm sure even in the day of Punch Cards, you know, there was some sort of thing about if you had five Punch Cards into your cooler than if you had 15, right and somebody was like we could use that as a metric to judge people on you know, and you know lines of code number of pull requests like these are things that we've attempted to do in the past. There was a phase there where it was kind of a fuzzy stuff where it was like how busy are you based on your calendar? You know, like are you approaching burnout or those sorts of things and As devops and devops practices have grown over the last you know, whatever.

Let's call it 10 15 years. I think that a lot of the practitioners knew some things early days that then the the folks over at Dora started to really tease out and put into like a strong format and to correlate some great research with outcomes. And those are sort of some key principles these four key metrics of like, you know deployment frequency change lead time failure rate, and then your meantime to recovery and You know with this information.

We have some metrics that are hard metrics so you can compare apples against apples and apples against oranges. You can do things that may or may not impact these things and you can know that if you're heading in the right direction that your organization is heading in the right direction as well. And so, you know having this basis I think has allowed a whole group of us to build this sort of next gen of this industry where we do feel like we can provide automated tools and take away some of that developer toil but then also show you via these hard metrics that that's actually working.

I had had a conversation with the former CTO of atlassian. I remember him just being like I am so tired of being in board meetings where you know, I'm talking about spending millions and millions of dollars on initiatives and the board is asking me how do you know that's working and basically boils down to me saying my gut is pretty solid after many years of honing it and I've hired well And all of their guts are pretty good, too. So trust us.

Mmm, no. I remember those days like I can tell where we are based on the kind of test failures. We're getting right just to worry the cycle that kind of gut feel totally, you know, it's interesting too about your timing.

And by the way, you know, we were talking I've used the products that you were developing on in those days. So I appreciate all the great work that you're done. It seems like that one of the things about the timing too is we now have Either hosted or SAS or at least software versions of things that where we can pull data from to get engineering and data from like every time we do a push or we do a you know, we pull request or remove push something into production.

We run tasks of cycles and got test data all of the tools that we use now are producing not just kind of off gas of data, but actually really valuable stuff that can feed into something like That's exactly right. And I mean that that was like the critical mass. I think that was missing, you know, one of the philosophical approaches that we take is that we're gonna meet you with where you live today, right?

So you're likely already shipping software and you've invested in a set of tools and maybe you're not entirely satisfied with all of them and you want to shift, you know from I don't know Jenkins to kubernetes or something like that. Right? But something is working and you're you're paying into these tools on a daily basis, you know your developers are you're investing day after day after day and it's that investment that we have to capture and then glean and sort of turn into these hard metrics and I think places where there has been failure in the past is when you ask developers to conform to some sort of convention, right, you're like you always have to label this thing where you have to make sure you change the jira state to hear and then you miss it and you get good enough but not quite accurate and then nobody has Evidence in the thing or developers feel under attack because they're like you're trying to measure me in a way.

That's abnormal. Like let me just do my thing. I'm a creative person.

I you know, they don't get upset. If you are gleaning information from what they're doing, especially if it's going to make their lives easier and better and also not like call them out for crap. That's not real because like, you know, they'll get very upset about that.

But if you're just saying hey, I'm trying to lift up your team and here's some really simple tools to do so and there's there's a lot of openness to that. Extremely good point because everything we pile on to a developer or development team like we need you to fill this out. We need you to go take this extra step to them.

That's toil right that's yes getting in the way of getting their work done and they don't necessarily see the value even if it is valuable or someone to take that step to the developers not right. It's slowing them down. It's getting them out of the mental state the groove they're in or whatever where they're working on and and I get it it takes a while to get your head in the headspace of the problem you're working on and pulling people out of that that's kind of productive that that right.

There alone is a big hit against developer or frankly anyone else's sufficiency. That's exactly right and you know, it is really about That toil it was interesting. We had a sort of launch event the other day where we just had a panel of folks that were talking about that, you know, like folks that have been in the industry for 20 years and you know, honestly folks that were working at the Forefront of trying to reduce that so like somebody from launch Darkly, you know, like disentangle the actual shipping of code from the turning on a features right and make it very quick to go back somebody from octeto, you know, creating like kubernetes based developer environments that just spin up on every pull request right again, like reducing that toil of like getting the developer to just be able to show their stuff easy make it easy to work, you know the US it's loose where we're saying.

Hey, let's measure this process and like how do you even know where the toile is that's like killing you, you know, I mean that that one's my favorite one just from like engineering manager days back when you know, you'd have somebody that is supposed to be doing performance work and you always knew you had a great engineer when they were like, I'm gonna measure because I can have all sorts of feelings about where the inefficiency Key, but if I measure right then I'm going to know and I'm going to attack the right thing, you know, like the amount of times I've seen a team shave five minutes off of their build process, but then come to find out many months later that like they're taking two days to pick up a pull request review, right? And you're like, well five minutes every test run that's kind of cool. But also if you could shave a day off of that that review process thing like you've made like, you know, infinitely more savings So like we all love to go into apple or code reviews, right?

We'd rather be working on. The stuff we like to work on but we've got to do this things too. So you can identify.

Hey, this is a gap for us or this is where we're wasting or spending time that we have to spend waiting for things to get done. It's always the yeah and then learn somewhere along the way of every hand off is a delay because by default there's a pause whether it's a couple minutes or weeks, right and there's that kind of and the startup cost depends on how big it is and the sort of the context that headspace transfer not I'm curious, you know, it it seems like as developers. We like to kind of build things what we need in the moment because we can create software but it does seem like at what what a tool like sleuth does like that's the right term as a tool is it'd be really easy to kind of start down this path.

Oh I can get the date. I can create a dashboard. I can do some things you get into it and the complexity of it really managing that over multiple Cycles multiple paths.

And you're measuring the product productivity benchmarking affect benchmarking against door. Sorry can't talk here. You know, it is a big problem space that you could easily, you know, you could turn yourself to be in the tools business where when you're in the manufacturing business and you don't have time to build these kinds of tools.

You're absolutely right. And I mean like most I think Nathan Industries. One of the biggest competitors we run on to is I can just build it myself, right?

And I like those kind of sales calls honestly because I'm like If you go off and do it for three months, I know I'm gonna have a conversation with you again because it ends up being harder, you know, and you you can yes, you can get your deployment frequency on a data dog graph like this right now if it's not doing what you want it to do now. What right like how do you tie that to the actual work? That's being done.

How do you tie that to your issue tracker? How do you know you know, like is it the build that's slowing you down? Is it the the rebuild that's slowing you down, you know like say you get it across the line.

How do you creating too many incidents? Right like the connections that you need the amount of information that you need to understand how that thing is going and how it's been received. It's very large and then, you know integrating with all of those tools suddenly makes that much harder than a day to dog.

dashboard Well and be one thing if we were shipping code, you know once a quarter once a month once a week, but I think but with these, you know much more frequent intervals of shipping code. Now those you know, seconds minutes hours days delays times how many times you're going through that cycle across how many soccer teams releases Etc that adds up very quickly when you're talking about something just that repetitive absolutely interjecting into a cycle may seem like, you know a short delay. It's actually not.

Yeah, and it's you also really, you know, the teams that are doing this really well are empowering individuals hmm, you know, and they're making deploys a non-event and they're building a safety net such that, you know, they're not doing the the Facebook line of like, you know move fast and break s*** like they're moving fast as well as like, you know, not breaking things making sure that they're looking after their customers and and that's Time together again like a lot of systems, you know like you I've seen teams that do it by having like a super SRE right and that person watches all the graphs and sort of basically sits there and like nannies things and says like Hey, Joe, you just shipped a thing and like I've seen this line do a thing that it shouldn't be doing but that's not very sustainable and it certainly doesn't scale but like building an environment where you are tying all these systems together and you can get like a slack notification straight away, you know a minute or two after you've sort of broken the contract for a North star allows you to kind of build that culture of you know ownership. so kind of speaking of sort of efficiencies and things and in the software world. There's always the time to Value right?

How quickly can I see something from a tool something from a product that I'm going to use talk about the adoption experience as you work with customers or maybe in the even in the sales cycle. You know, how long does somebody how long does it take before somebody says that's exactly what I'm looking for. That's data.

I wish I would have had a month ago. Well as as you kind of said earlier because of the tool sets that people are adopting, you know, if a team is oriented around like a lot of the Integrations that we support that can be a very quick event now, I'll be honest and say that for for SMB I've seen that aha moment show up in like five minutes, you know where because they they really understand all the things there's an individual who has like, you know, the ability to install a GitHub app or generate a key to talk to jira or you know, like do an off dance with pager Duty and they all understand like hey, this is our project. Here's the five repos that we have.

We do have this page or Duty service and they can kind of like connect it all up very quickly and then bang they have like a 30-day Baseline of what they've been doing and they can see their data in there and start to learn things that they maybe had a gut feel for and then they can validate that or there's often some surprises in there too. Now if you're a larger Prize it takes a little longer because you have to I don't know phone a friend for the ability to you know, get a GitHub app installed. Like maybe there's like some security team somewhere that has to allow you to set up the slack integration.

You know the person who's maybe leading the initiative to provide this capability internally isn't the one who really truly understands like what the build job is that's like taking something from to you know, staging to production and so it can be a little bit more of a lengthy process like kind of anything with with Enterprise. Um, especially something that goes across tools or across teams in an Enterprise environment you're talking about Yeah, some approvals some steps and Bureau tickets. Yeah, but they're definitely I mean that's one of the other things that's kind of brilliant about where we are today is there's Primitives that are pretty Universal, you know, like most teams are working with environments of some sort.

You know, it might be super super simple where it's just staging in production. There might be 15 different environments. It might be, you know, like a gradual roll out across GEOS or that sort of thing.

The concept is sound right like there's some sort of tool there's a ci/cd system that is taking something like from code producing an artifact or just taking that code and getting it to an environment and putting it in front of customers, right? Everybody's gonna use an issue tracker of some sort. So, you know, a lot of these things are Are not alien Concepts and and most folks are going to be using some.

Excellent. Well, it's been a lot of fun talking with you and I have had a chance to take a look at the I guess the demo that it's a live demo. You could really see data working in the system and how it flows and that's how we're building sleuth so we sleuth and sleuth and we figure hey if if it's useful for us and we can we can show other people how we use it.

I was wondering is that data in that demo coming? It? Looks like it's from your team.

Yeah. That's that's our team. Like our internal rule is trying to keep the cursing down to a minimum, you know the internet.

Yeah. It's a cultural thing too, right? You know, so we have our failure rate published out there and our meantime to recovery and we have ratcheted that down so that we're very we will call things that are not necessarily customer facing a failure.

And so when we get into a circumstance where that's quite High, you know, we're a little embarrassed and like that will motivate us to Turn around and actually fix it which you know, a lot of the stuff is about cultural and Automation. And so having that in place allows us to sort of engineer in the way that we would like well and then you you know, you truly are using the product you're creating and so you're at least have that experience, you know from your side of it that you can bring to not only the product but helping customers as they adopt the same kind of Technology. Absolutely.

We'll go. It's been great. So working folks find out more see the demo that I was talking about that I checked out.

io. There's a link up at the top which is live demo and like I say, there's there's data every day. We're doing between five and 12 deploys a day.

So there's lots to lots to see in there and you know, you can always chat to us straight off of that website as well and we have a 30 day trial and if you're one of those SMB companies, you know, like you can doesn't hurt to just try it hook it up and see if you can't discover something about your Engineering Process that maybe you didn't know and super charge it a bit as well. Very cool sleep that I know don't act in been a pleasure talking with you and wish you the best. Please come back.

We'd love to hear more about how things are going at smooth. Thank you. Yeah, I appreciate it.

Mm-hmm.