Clay Smith – Lightstep Demo: Distributed Tracing: The Ticket To Shipping Quickly And Confidently

I thanks very much for joining the Distributed tracing and deploys talk. This is the session we're talking about how to use distributed traces to shift quickly and confidently. This is a really short presentation.

It's going to be mostly a demo, but we have a couple of slides that are just going to set things up and introduce the concept. A few people at the very end, in addition to being recorded, they'll also be some time to answer questions. So just by way of introduction, my name is Clay Smith, I'm a partner engineer, have been working with continuous integration and deployment for a long time.

It's a topic I am passionate about and excited to kind of talk to you today about how to secure Tracy's fit into deploying software by way of introduction for light stuff. The company I work for one slide and two images that we like to show people. What are the difference between A, B, the public or popular perception of modern software and how it actually works for practitioners and people that are kind of deep into actually building complicated systems?

We know from experience that unfortunately the reality is much closer to the image on the right side there. And let's step in particular, we're a software company that really hopes to help sarees and platform engineers work in optimizing the software that's in that environment. And ideally, in doing so, we can help people build software more quickly and more reliably.

And so with that, obviously, observability comes up and there's been a lot of marketing around Observability and kind of what that means. And there's a lot of different ways to look at it, including the Wikipedia definition, which comes from control theory. But a really simple and straightforward way to look at it, too, or think of it is what caused that change.

And this becomes especially interesting with service deployments making changes in code and deploying those changes, because that's an intentional change that hopefully goes well. But we know from experience that's not always the case. But with complicated systems and distributed systems in particular getting to the root cause of deploying whiteouts, what break or what's what broke or what's causing that regression can be really tricky.

And so the demo and the talk today is going to be going deeper into that use case in particular. And we know from surveys and talking to a lot of people that some you know, the the story behind what caused that change after you send out could change. The production isn't always perfect.

Polls and surveys have shown that this process is more often not can be feared by people, and it's often slow and sometimes fragile and beyond all of that, particularly with distributed systems and teams and different services with different owners. The coordination in a large organization to kind of figure out who broke what can be really complicated by itself. So we think there's obviously a better way to do that.

And that's kind of the purpose of this quick session today is getting into that. So we've set up a demo environment that will be going into shortly of Project and GitLab. It's a it's a doctor based myCar service environment in a what?

We set up a GitLab CI/CD pipeline to deploy that to this cluster. In this case, it's single cloud using GitLab, manage communities, environment. And then we've connected that to let's step to actually understand what's happening during his deployment and how to understand those changes in the idea.

Here is all of these together. When you're using all of them, you can get back to the code and fix problems faster and more quickly. And so we're showing that shortly.

So I'm going to switch over to demo to kind of show this in action. I mean, it's on my screens here right now, and we'll switch windows. All right, so we're back in the demo environment and we pulled up to the hipster shop repository here in GitLab, so this is the microcircuits repository I was talking about earlier.

It's based on a pretty popular demo environments for Google. But the idea here, we can see in the service diagram we've got about eight or nine different services. That's powering a customer experience.

In this case, it's an ecommerce app that sells products for hipsters. There's nothing unusual about this. But there's two things we've done in this environment that I need to call out before we kind of jump into the observability piece.

The first thing is that all of these different services have been instrumented and they emit telemetry in particular traces in order to use those traces to kind of get to root cause analysis and understand what changed during the process. The other thing, in addition to having telemetry and generating traces, is that for each of these services, we've actually made a change in the pipeline. Here is a configuration file for scaffolded, which is a tool that manages these deployed managers.

Kubernetes deploys, but we've done here is actually injects the version from the jet lab environment variable into the instrumentation code. So when we're running these services in production and those services are emitting telemetry that gets collected and analyzed by LightStep, we know exactly what version was running when we observed those changes. So this is somewhat of a technical note, but just need to call it out that this really enables the full end to end workflow.

We're able to go from code to the telemetry data we see in LightStep because we've we've added this tag. So just a quick note and different ways to do this. There's some more documentation on the step side, but this is just an important note of what ties it all together.

So through the Gitlab continuous deployment pipeline, we've deployed to our communities, cluster running Google Cloud. Here it is. You can buy various hipster products.

And in this case, let's say that we've started to get some customer complaints the morning after. We know we've made some changes. So as I said earlier, the key question is, well, what caused that change?

What's actually causing the slowness that's causing customer complaints? And so at this point, there's historically been various ways to do that. And I'm going to show what it looks like in step, in particular the the service directory page of light stuff.

And so this will look familiar if you used metrics dashboards before. But there's a couple of important differences I wanted to highlight to. First of all, you see all of your services.

So far so good. These are all the micro services in the environment. And in the middle of the page here, we see key operations on that service.

So latency errors, operations per second because the complaint was around slowness. The idea here is, well, looking at latency, what got slow and when because we've tagged or instrumented the code in that environment to include the version number we have, these version tags appear. And what this means is it's the first time that light saw that tag in the production environment.

And so what it does, though, is it makes it really easy to see the before and after to change for that service. So in this case, this already looks super suspicious in the inventory service, in the update inventory operation. So here's a call that's happening in the inventory service after this deploy version, one point one four eight one eight seven latency jumped from a few hundred milliseconds to over a second.

So if we click on that, we compare it to one hour prior. We can have a before and after picture of what happened during that change. What we just did there in a single click is set up the baseline window right here in blue with the regression window right here in yellow.

So we want to understand why it's so much slower and yellow versus blue. We're going to go through this page and kind of drill down. So immediately we see that histogram of latency on the right hand side in yellow.

It's a lot slower and blue, it's a lot faster. So what actually changed? Because we're collecting tags with every request through these services, we are able to bubble up almost immediately.

So what's in the baseline and what's what's in the regression? And a couple things immediately stick out. One, the version is different.

So in the regression version, one point one four point eighty seven, it got slower. But also, this is kind of interesting. There's a tag that says a large batch equals true.

So this is kind of indicating that some code changes behind this and it's not CPU memory or like resource exhaustion. So by clicking on that, we're going to kind of narrow our field to. We then can see the service diagram or see all services in something immediately kind of stands out here in the inventory service when it writes the cash we see in yellow here, there's a lot of latency contributed to that.

In addition, we say all the upstream services, the Web app for the Android app, the IOC app. So we know it's also having customer facing impact. If we go in and clicking on the inventory service where they're going to get to go ahead and just narrow down into the right cash operation on it.

We're going to actually see correlated logs. So again, baseline and blue regression in yellow right here. We see that in the regression.

It's writing the cache between thirteen hundred and thirteen thousand items. And that's not happening. The baseline window.

So we've gone all the way from seeing the latency spike here, understanding the service, the version, and then lastly going all the way down to the operation. So in a few clicks we've gone from things seem slow. Customers are complaining to the individual correlated long, long lines that indicate that this is a code change related to the cash rate operation.

And then in doing so, this is more than enough operation to open a Jira ticket or open a bug report, roll back the change and go back to your GitLab project, see what happened in that version and make the appropriate change. So that's the entire workflow. And at this point, we'll go back to the demo deck here and answer some few questions and happy to answer those and get into more detail.

Thanks very much.