Greg Leffler – Observability: A Crash Course

Hey everybody. Welcome to my session and talking about observability a crash course. My name is Greg Leffler.

You're here for cloud native day, 2022. We're really glad that you're here during the session. We have plenty of opportunity for Q&A that I'll be answering.

So, please fill those in throughout. I'd love to hear what you have to say and to help answer any questions you have about observability. Because I work for a vendor we have to put in boring legal stuff.

This statement is here upon your screen. Please pay attention to the Ford looking statement slide during this presentation. I may make forward-looking statements about Splunk or about the industry.

I don't intend to do so, but they may happen if you choose to make any buying decisions about our product or anybody else's product. You'll want to make sure that you base those decisions based on published information and not anything that I might say during this presentation. com.

So with that out of the way, who am I I my face is here on the slide. And on your screen. I am the director of observability practice at Splunk what that means is basically I talk to developers software Engineers sres people like that and explain what observability is why you need it how you can get there all that sort of stuff and this stuff.

I'm going to talk about in this presentation. I was an SRE before I came to work here. I worked at LinkedIn for quite some time on their media service their key value stores their content ingestion service lots of different parts of their site and then I also was a manager there and you know have lent teams.

I started my career in the knock and network operations center raise your hand if you remember those but that's how we started off at eBay advertising and then what's a systems admin there as well. So I've been doing this for a while. I think it's really important to sort of Explain what observability is and the sort of evolution that we're making from monitoring to observability.

So that's something that I'll discuss in this presentation. So there's four main parts that I'm going to talk about today. One is before we talk about like what observability is.

I want to set the stage for why we need observability why it's important sort of what uses it has and why the industry is sort of moving away from this monitoring free Phase to an observability one, then we'll talk about what observability is what the requirements are. Like, what do you need to have observability from both the tooling perspective and a process perspective and even a mindset perspective. Like how do you think about this stuff?

And then finally we'll talk about how you can get started. Right? What do you do today to like make a step towards getting observability.

I'll let you know at the end. So first off, of course, we're going to talk about why why do we need observability? And why do we need it now?

if you look at this slide, one of the things that became really clear during 2020 due to some circumstance that a lot of people and a lot of transactions and a lot of businesses moved online. The things that you used to do in person that you used to not care about the app for your grocery store. For example, a lot of that changed right and now I use my grocery stores app every single week and I expected to work every single time whereas before 2020 I couldn't tell you if they even have one but now it's really important and we see this throughout the whole industry and throughout General Trends so you can see 70% increase in internet use during 2020 compared to 2019 right a 76% increase in e-commerce transaction volume and then 65% of customer interactions of all customer interactions are now digital in 2020 and we're probably gonna assume that that's higher through 2021 and 2022 but during 2020 all of this happened compressed through what would have been probably 10 years worth of changes happened.

Nuts it went from 0 to 100 for a lot of companies. And this meant that they had to have complicated sort of rushed to Market things to meet the digital demands. and you know, why did they have to do that like If we set back a little bit and think about how do companies deploy software.

How do companies operate we believe that every company is done some level on a cloud Journey right? There is eventually every workload will be in the cloud and whether that's today whether that's in a year or two years, we believe that most workloads of the company not every workload most workloads of the company will be in the cloud. And so we see sort of four phases for where companies are in this journey, right retain an optimize is sort of hey, I'm still keeping most of my stuff on premise.

I might have VMS but like I'm not using containers. I'm not using the cloud. I'm not using serverless.

I'm not using an orchestrator like any of that stuff. That's not even on my radar right the apps that most people develop that are using this model. There's really tightly coupled, you know, we talk about a monolith.

There's a microservice like these are definitely monoliths. There may be, you know, two or three tiers. Generally, it's a pretty simple sort of thing and these places generally still have that sort of firewall between Dev and Ops, right?

So like the dev team doesn't talk to the Ops Team other than if there's some code over the fence, right and the Ops Team doesn't talk to the dev team except to say hey you broke production. So that's sort of where a lot of people were before 2020 and where some people still are now because there are requirements to you know, some things need to be on premise some companies. Don't feel the need to invest because if it's working why change it, you know, there's lots of reasons for that but then As companies build new apps and think about new approaches and they want to increase their deployment velocity.

They want to be a little bit more agile about where and how they deploy things. They want to be able to respond to stuff faster and launch features more quickly than they could before they start to move to these other three approaches. So lift and shift is a relatively straightforward one.

It's basically taking that application from retaining an optimize and putting it in the cloud. Okay. So you take your VMS you throw them up on infrastructure as a service and now you did the cloud hooray, you know, that's sort of true that is a way to take advantage of cloud resources, but it's not really getting the full benefits of the cloud.

So over time as Devin Ops teams integrate and they move closer together. They sort of start to say like hey, why do we need to manage the database? Right?

Why do we do that? If we're already in the cloud we could use the cloud providers database system we could use I know SQL service we could use, you know, all the things that they provide. To build things faster and to focus more on what the business needs, right?

So the application becomes more modular, but the pieces still sort of depend on each other and they still depend on the cloud providers infrastructure as well. You know, you'll see here people starting to adopt container technology, but it's not as fully embraced as the last group which is to re-architect applications or to be cloud-native Cloud native company. These companies have fully integrated devops practices, right?

Everybody there is responsible for deploying to production and running production. They generally build their apps on loosely coupled microservices serverless functions, you know service meshes all the cool new technology. They're definitely using like Helm or something like that.

If they're doing containers their own private clouds public clouds, you know, but they don't have on-premise infrastructure. They don't really care about VMS, right? They're busy deploying the new parts of their app as they become available the downside of that is that that's really complicated right like as you go from left to right on this chart you start to have more pieces to keep track of and more pieces to deal with We see that.

This is a quote from an analyst that says by 2025 85% of organizations will run containers and production up from less than 30% in 2020. So people are adopting containers. They're adopting this practice.

But we see that as companies sort of figure out how to do this. They're doing it more. They're doing it in a more challenging way, right?

They're doing it in a way that is harder to do but has a lot of benefits if you are a developer and you learn a new language and you want to deploy as small service and that language that's cool on a microservice Model, right? Whereas if you're doing a retaining an optimize a monolith, you might need to do a lot of work to refactor a lot of that application into this new language. So let's you be a little bit more agile.

And so as you're developing new applications a lot of those end up being the cloud native kind of apps they're on the right hand side. This is because the world that people live in has never never been more complicated, right? This is Let's Pretend This is an airline that has a very small portion of their infrastructure shown on this slide, but even here right?

There are three touch points of customer can use right. There's a kiosk at the airport. There's the website there's a mobile app and there are all of these things in between that the customer could do.

Like if they check bags they need to talk to the baggage service if they want to book a flight with their miles. They have to check when with the miles service they have to check out the flight status service. It's checking with you check out microservice to actually sell the ticket like all of these apps talk to you all these other microservices which then could be built in a bunch of different ways.

Right? There's a private data center. There's a third party reservation system.

There's a bunch of cloud Technologies thrown in the middle there. There's serverless functions that could be used. Right all of this stuff needs to be monitored and kept track of right and if you Think about a traditional monitoring tool like say nagios or zabix or something like that trying to monitor.

All of this would be really really complicated and what we see is that like the traditional monitoring Universe amplifies complexity, right? There are so many things that traditional monitoring makes much worse. The more complicated your environment is right you have to use different tools that pull land from different silos sources of data, like you can't necessarily see things from one service in a different service if they emit different types of data or if they log in different ways if one has metrics and one doesn't you know, that sort of thing can all be different and it's harder to keep track of that.

Fragmented visibility is a huge deal. Like if you only have one monitoring tool that's looking at this one service. You can't tell if the upstreams are downstreams of that service are having problems.

You can't figure out where the problem is as quickly as possible. And these things also lead to lack of domain expertise High mttrs poor scalability. Basically traditional monitoring can't handle a complex microservice environment.

So we say observability is the future, right? You need to move from your monitoring practice and sort of evolve it to observability. In the monitoring world, you have a point tool that you have a tool to look at logs.

You have a tool to look at metrics. You have a tool to look at traces if you're using APM right in the observability world, you're moving to an integrated single screen view that tells you what's going on and they can integrate data from all of your different ways that you run the application. So from your on-prem machines your VMS your cloud services your serverless third-party JavaScript dependencies, like all of it can be put in one place.

With monitoring two you often just sort of see data up on the screen. You see dashboards. You see numbers.

You see charts and with observability. We try to give you answers right? You're looking at aiops Driven tooling that can say hey based on these metrics.

This thing is the most likely cause of your outage or the most likely cause of this metric being violated, you know monitoring tools aren't built to do that and they don't necessarily have all the data that's needed to do that. Like there's no way for that to even be a thing that could be done. So we really feel like an observability approach gives you integrated answers in a centralized and scalable way.

Right? Like you can scale your observability platform as your data needs grow, but you can also see all of the data that you need to figure out what's going on in your world and with your architecture in one place and a traditional monitoring tool, you know, you're looking in several different places, you're kind of running these ad-hoc queries when things break and you don't really get To get a cohesive coherent view of Your World which is really the main thing that observability gives you so we sort of setting the stage here for why you need observability. Let's talk a little bit about what it is and you can feel free to throw into the chat like your definition of observability and then I can tell you how it's wrong.

But you know, there's not really an industry accepted definition for observability, but there's a few things people say when we talk about what observability is and the most common thing I think that we hear is it's a metric traces and logs Well, that's not what observability is. Those are data sources that help you get to an observable state right or they're sort of necessary but not sufficient to have observability. The formal definition from you know, like a science book or Wikipedia is that observability is the ability to infer the state of the system by examining its output.

Sure. I mean that's that's true. But if you're trying to implement this as a devops practitioner, you don't know what that means.

Nobody knows what that actually means from a Computing technology standpoint. And it's it's a pretty dry definition and doesn't really help you understand why you need observability, you know, people say, oh, well Greg you work at a vendor. So observability is just a cool new thing to spend money on.

Sure. All right, like we'd love for you to spend money on it. But really what observability is is it's it's really a mindset.

But for this slide we're gonna say observability is a way to investigate unknown unknowns in your world by instrumenting everything and if we step back from that definition for a second one of the things that you really need to consider is The hardest problems to solve are problems that you don't know you have all right, because if you don't know something's broken, you're not going to be able to fix it and with a monitoring tool like it can't tell you anything. You're not monitoring. Right?

If you're building a factory and you don't have a flow gauge on one of your pipes. You can't find out when the flow stops like unless you look further down the stream and you see that you know, you're not building the project or running the widget or whatever, right? But in our applications, there are new ones every day right in a complicated microservice app, if somebody deploys a new application out to production.

Well that's gonna create hundreds and maybe thousands of interactions with other parts of your application with the way that customers interact with it or with the way that it interacts with databases or you know, no sequel data stores or whatever you're using all of these things are much more common than they were in the past simpler architectures. And so you know, how do you investigate an unknown unknown? How do you know that?

It happened? Well, you instrument everything like an instrument basically just means collect data from so you really figure out all right, what are all the components of my system? Right?

If I'm at the factory? I'm gonna put a flow gauge on every single pipe. I'm gonna put a detector on every switch to say if it's up or down I'm gonna wire in every alert light so that I can see what's going on all in one place.

Now, obviously if you were to do that in any installation of any complexity at all, that would be a string, you know, a five stories high with a million lights on it. You would have no way of knowing what's going on. So you really have to be able to trust the tool to help you make sense of all of this data and to draw your attention to what's important.

So, you know, the upshot here is that observability really is a mindset of I need to instrument everything so that I can figure out stuff that's going to happen or that has happened that I didn't know could Them because again as these apps get built and they become so complicated. It really is a challenge to figure out where things are going wrong. We see a lot of our customers will tell us that their mttrs, you know were extraordinarily high before they adopted observability because they couldn't find the problem.

And if you look at a service map for a modern service, there's hundreds of microservices connected to each other in all kinds of ways and you just can't retain that state in your head like trust me. I tried there's no way to figure out. Okay, all of these Services, what do they do?

And when and how is this customer transaction flow going between all of these like it's just not something people can retain in their head. So what do you need then to get observability? Well, we believe that you need four things.

We believe in observability product needs to be full Fidelity. That means it needs to instrument and collect all of the data in your environment without sampling any of it. You know, anybody who's done a production operation before like I'm sure you could say in the chat like hey tell me about an outage that you couldn't solve because you didn't have any data about it right happens all the time and especially if you sample, so if you only take certain data points, you might not take the data point that had a problem.

See you might not know you had a problem and we think it's really important for observability that you have that full fidelity. We also think it needs to come in and be operable in real time. Like today's systems.

You have serverless functions, for example that can last less than a second and if you're waiting five minutes to hear about a problem with it, like that's just not going to cut it right you're not going to be able to solve the problems and figure out what's going on. And you know, one other aspect of having a real-time system is that you can do data manipulation on the screen as it comes in so you can find those anomalies faster you can figure out when this metric is out of line quicker. You can alert on a trace that's going to go bad faster.

You can solve the problems faster than you need to. Obviously a open Telemetry an observability system needs to be scalable. You really need to make sure that it can grow with you and it can handle the volume of traffic.

You're going to want to send it. You know, lots of people will say, oh we can support millions of transactions and a day right? It's like okay.

I mean that's that's not really that impressive you really want to be able to deal with something that can grow with you and that can handle an arbitrary number of transactions and to assign tags to those transactions and arbitrary number of dimensions for those tags. You want to be able to slice and dice your data as much as you can so you need a scalable solution. And then finally we think open Telemetry like specifically is important for observability and open Telemetry is a way to get data into your observability system and it is open as the name would suggest and it lets you own your observability data.

So if you decide to change vendors or to build your own product you can That without having to redo all the work you needed to do to get an instrumented in the first place. So open Telemetry we think is absolutely critical to a good observability practice. So, you know when we're evaluating these requirements what what is the the how can you figure out if your solution meets all these requirements?

Well, I'll be obviously Splunk sells a product that does this called Splunk observability Cloud that meets all of those requirements that I just talked about but this is applicable to any observability system. Right? Like what are the components that you need from another ability system, right?

And we believe it's these six that are up on the screen. And so I'll start here at the top and talk about synthetic monitoring first synthetic monitoring is a way to run tests and browsers all around the world and figure out what is happening to those browsers and through those network connections. And it lets you detect problems with your site before your customers do so, right.

You can simulate the checkout flow for your shopping cart and you can simulate it around the world and all kinds of different browsers and you can find issues before somebody else does it's really powerful. Of course infrastructure monitoring is always going to be essential even in a cloud first environment. There is still infrastructure involved.

If nothing else, there's the cost to operate and run. Your Cloud environment is one of those things but most people will have some degree of hybrid or multi-cloud and figuring out do the VMS. Look okay are my containers running the way that they should be or the resource limits that correctly like all that kind of stuff is an infrastructure monitoring question log Observer is our name for a log analysis tool logs are still always going to be critical.

You need a way to know what has happened and why something broke logs are how you find that so you need a log analysis tool as part of an observability system rum stands for real user monitoring and it's a way to see what's going on in your users browsers, you know, a modern application has so many third-party dependencies. There's JavaScript. There's Frameworks, there's libraries there's fonts.

There's fraud protection. Fingerprinting, there's a million different things that run in the user's browser and you have no clue if those have failed without run to tell you. Hey, I had a problem with this script or hey, this font didn't render and it's really powerful way to see how your customers experience what's going on.

On call the world famous on call you can share your experiences in the chat about how what it's like to be on call but you need a tool that alerts the right person when something is wrong and that also gives them context around the problem and that alerts the right person the first time so that you're not paging six different people to solve the problem Splunk on call has all those features but like any on-call tool is an essential part of the observability world and then finally sort of the flagship of an observability practice as the ability to use APM or application performance monitoring. This really is a way for you to find out where problems happen and I really complicated sort of microservice multi-cloud world with APM. You can slice and dice by particular types of customer even particular customers potentially to see how they experience your site.

You can see which servers their requests hit you can see how long each thing took you can see if they had any errors you can do that by Like your customers who are high Spenders, you can do that by people who you sent a marketing campaign to like you can really slice and dice what happened with a transaction or a group of transactions with an APM tool. So these six things sort of make up a mature observability product. and you know those things all get combined in your observability products to help you make decisions, right?

So in Splunk observability Cloud, we say it's analytics powered we use a unified entity model to connect everything together so that you can click through each thing to figure out what is happening. Right? So on the service map, you can see the Angry Red Dot for when something is going wrong.

You can then click into a trace that shows you hey, this is what failed and it shows you the waterfall of all the services it called and where exactly the problem was you could then click on that and see the log line that says this is where it failed this sort of analytics power capability has led people to 80% reductions and mttr. Lenovo is one of the examples that had that huge reduction in time to resolve and this is done. This is capable because we have all the data and we have advanced machine learning and AI that can tell you this is where the problem is.

This is the root cause of the problem like it's this service. That's what's broken. As I said earlier open Telemetry is really vital Splunk observability is open Telemetry native.

We don't have like a Splunk format of open Telemetry format is our format and that lets you use a single open source, Open Standards based agent for metrics traces and logs and more data sources to come which gives you better control. Like you can only send what you want to send. You can redact Data before you send it you can send off as many copies as you want.

So you could use a vendor product you could use Prometheus. You could use something you've developed in house. You could use two vendor products like all the same time all with one simple agent that gives you a lot of flexibility and they're really key thing about this is it lets you instrument one time and never have to worry about doing it again, like a lot of people will say, oh, we have automatic instrumentation and we say that too but there's no automatic instrumentation tool that can know what's the critical business.

Flow for your business right automatic instrumentation can't tell this is the start of a customer's checkout transaction. So I really want to pay attention to that. You can do that by instrumenting your applications through open Telemetry and it's something that you don't want to have to repeat if you choose new technology so to us and to me and to all of you hopefully open Telemetry is a requirement and a lot of vendors will say.

Oh, yeah, we read up on Telemetry, but if you follow their instructions to instrument they have you instrument with their proprietary agent, right? They have the instrument with something that yeah, you can check in it's gonna be hard to check out when you want to leave. So let's one thing that like if you follow our documentation if you reach out to our support people at Splunk, like we're gonna set you up with them Telemetry because that's better for everybody in the long run you owning.

Your data is the way that it should be and that's not negotiable to us. Right? We're really firm Believers that the value of observability.

It's not how you collect the data, right? The observability is And that value of the observability product is the insights you get from that data, right? So the AI and the ml that you know, the vendors running or that your product is running that's what's impressive and that's what we need to try to like protect not the data itself.

We are a very very prolific contributor to open Telemetry. We have people who work here who's full-time job is to contribute to open Telemetry and to make it better but it is a community project like we're not the only contributor. It's not a Splunk standard that we just claim is open.

It really is a community developed product and it's something that is really the future of observability. So Let's talk about how to get started. Like I mentioned, you know, I think that's Punk has the most comprehensive set of observability capabilities.

That's the reason that I work here. But we really think that you need the six things around the circle to have full observability into your architecture and into your product. So, like I said, you really need to have synthetics real user monitoring incident response or on call log analysis infrastructure monitoring an APM.

Like these things are just critical to having observability and to being able to figure out like, how do I get from A to B is start adopting these products these products should be full stack, you know analytics powered the analytics is really the secret sauce for an observability tool. So that's really something you should evaluate closely Enterprise great so they can grow with you and then open Telemetry native. Like it really is critical that you own everything that is relevant observability data like it's your data so we want you to hold on to it and then finally It's really important that you're able to analyze your data no matter where it comes from.

A lot of cloud providers will advertise and observability product. But if you can only look at the stuff that's hosted in their cloud. Like that's not great because you can't correlate data with stuff that happens outside and there are very few people who have their entire product and application living in one cloud service, right?

It's either in multiple clouds some of it's hybrid. There's on-prem, you know, there's all sorts of different deployment standards, but we think you need to be able to get observability across any of them. com slash.

Oh 11y Dash trial or you can scan the QR code on the screen. We do have a 14-day free trial and you can start instrumenting stuff with open Telemetry right away. And then if you choose to use a different product or different vendor like you can take that with you.

So that about wraps up our time. If you have any other questions or anything else you'd like me to answer. Please feel free to throw it into the chat, and I will try to get to you as quickly as I can.

But otherwise, thanks for your attention, and I really hope you enjoy the rest of cloud native day. Thank you.