Challenges in DevOps: Moving Beyond Monitoring – Abhishek Singh, Elastic

This is Textron tv. Hey guys, thanks to the throw. We're here with Abby Sch Singh, newly appointed general manager of Observability for Elastic.

And we're gonna be talking about, well, what is going on with observability and all things related to telemetry. Abby Sch, welcome to the show. Thank you.

Mike, What's your sense of how far along in the observability journey are? We, I think we've been talking about it for a while, but a lot of folks had to instrument agents in the applications and then they would get the observability platform, and it wasn't clear to me they knew how to use it. So where are we as we kind of move down the path here?

That's a, that's a really great question, Mike. Um, you know, observability sort of, um, ha has been something, as I think about, um, you know, I think it was, um, started becoming top of mind for people, I would say, you know, late, uh, 20 16, 20 17, um, ish. Um, you know, with the advent of, um, APM becoming more relevant, um, open tracing back then becoming an open standard.

Um, we had some vendors, um, like LightStep Honeycomb, and a few others coming on the scene. Um, I was in the midst of all of that, um, launching AWS X-Ray back in 2016. Um, and I truly think, um, you know, there was this belief that, um, observability tied in with, um, historically what's been a notion of, uh, application performance monitoring.

Um, so as you mentioned, introducing, uh, that concept of instrumentation, uh, being able to instrument your code to produce data, whether it be as traces or structured events, and then being able to question, um, the, the system that you have and understanding from it, um, what was actually happening within the system. Um, so as we think about that, I think we've, we've come far along that journey of, um, observability being in everybody's consciousness. Um, but I do think we still have a lot of work to do in term of making observability easier for customers.

Um, and for everybody out there to actually implement. I think it still is fairly difficult today, um, with things like, um, the instrumentation being intrusive in a few languages. Um, so, you know, we, we've, as, as I was interviewing with Elastic, one of the things I told him, I said, you know, I've been involved in this world since 2016, and I think the arts move forward a lot, but a lot has stayed the same.

And I think, um, there's an opportunity to truly reinvent, um, what observability means, not from a end customer perspective, but how they implement it. A lot of folks had it in their heads, maybe that they were gonna replace monitoring systems with observability and monitoring systems tend to attract predefined metrics. But it seems to me that we're kind of layering observability and on top of those tools 'cause we're not quite ready to give them up and there's still a use for those.

I, I totally agree with you, Mike. Um, it's, um, you know, using my experience, um, at AWS as an example here, um, when I was first at AWSI was working on a platform as a service offering called Elastic Bean stock. Uh, and back in those days there was this big debate between infrastructure as service and platform as service, um, and which one would win and which one would be better.

And one of the things I learned, um, from that debate back in the day, which is very similar to this notion of monitoring and observability, um, and them being, you know, constructs at different layers effectively, is that both are actually useful. Monitoring effectively is the proactive approach of, um, being able to tell you that something's wrong in your system if you've got the appropriate things instrumented, um, observability to a certain degree is the reactive approach of something's gone wrong, I don't know what it is, and I need to figure it out. Um, so you've got more of, you know, when you're doing observability, you are playing the detective of really trying to understand what's happening in the system without having predetermined outputs.

Once you figure out something's wrong, it's a good bet to go and instrument those pieces to make sure that if that same sort of error is happening, again, you are indicated or you're alerted on those errors. Um, so I do think that in, in terms of observability and monitoring, they go hand in hand, um, using the traditional sort of notion of monitoring. And I think the utility of both will continue to be valid for as long as I can see.

I think one of the other reasons, um, both of these go together is a lot of our customers that we're talking to at Elastic, um, they have a software estate that's been around 20, sometimes 30 years. Um, and you have a wide variety of tools that have been deployed. Um, so it's extremely hard to go in and tell, you know, a Fortune 2000 company or a Fortune 500 company that they need to re instrument everything and they need to implement observability.

It really ends up becoming a significant effort that they have to take on. And unless we can show what the business value is of that effort, it's hard for us to go make that case. And I think that's one of the fundamental issues with observability today, is that we haven't done as an industry a good job of tying back, um, what you get as a business value from making systems more observable.

How do we achieve that? Because to your point, one of the issues that I find that people encounter is, well, they don't really know what questions to ask, so they can't frame the question to go look for the answer, because if they knew the answer, probably they wouldn't be questioning in the first place. So how do we make this whole thing more accessible To people?

I think, you know, as I'm thinking about the accessibility bits of it, I I, I do think, um, one of the patterns I've seen in the industry is that a lot of vendors have leaked, um, effectively their implementation complexity onto customers. Um, so, you know, includes AWS, it includes, uh, my previous employer, it includes a number of the vendors I find in the space, including Elastic today to a certain degree, is that we've taken this view on telemetry types and optimizing for things like metrics, traces, logs, instead of just being able to collect all of the data that your applications are producing, um, and then being able to allow you to question that data in a way that's structured and makes sense. And one of the reasons why I chose to come to Elastic is, um, elastic fundamentally, um, because of the ELK stack and the ability for the Elk stack to consume all kinds of data and not really think about it in, in sort of these telemetry ways, but really just thinking about, you know, sending us the notion of logs, um, as a data stream effectively, or metrics as a data stream or phrases as the data stream.

And then being able to collate all of that together and be able to ask questions of it, um, which is where the search, uh, aspect of Elasticsearch comes in, which is where some of the work we're doing around our relevance engine, Ezra comes in, some of the work we're doing around Vector search, um, gen AI and LOM comes in. Uh, it's really about taking all of this data and then being able to find relevance points, uh, and being able to operate on that. One of the things, you know, I, I tell my team almost as a joke, but I think it's extremely important point, is today, if you go and look at any observability tool, um, you'll see, you know, they have hundreds of dashboards.

You have, you know, tens of thousands of metrics. What people really want is answers to four questions. And I think as folks in observability, we really have to focus on how do we get people answers to those four questions.

And what I call those four questions are, you know, one is at any given time or over a period of time, what is the state of my system, you know, however I define that system. Two, are there any services in my system that aren't operating as I would expect them to? So think, you know, SLOs and SLIs, uh, three who's impacted by any of those issues that I'm having in my system, whether internally or externally, and four, how do I fix those issues?

And so as we look forward, um, at Elastic from a observability specific focus, our focus is really how do we help our customers answer those for questions and not give them, you know, hundreds and thousands of metrics and alerts and dashboards, because that's really us taking our complexity and giving it to our customers and then asking them to do, do the analysis, asking them to understand the data instead of really helping solve that problem of, you know, what's wrong in my system and how do I fix it? One of the issues you hear from folks that are down the path early is they're struggling with the amount of data that they're now being asked to store. And, um, it gets expensive.

Is your approach, uh, at least as I understood it, seems to be normalizing the data. So does that kind of reduce that headache or how do I manage all this data? Uh, I, I think you know, it, it's from a data volume perspective, um, I expect the data volume's gonna continue to grow as people, uh, either do more in-depth instrumentation or understand and put things in place like EBPF, uh, which are gonna automate some of these things.

Now, the goal of automation and the ability to ask these questions as you mentioned, relies on data being there, I think the important thing, um, that we have to understand is how do we make sure that it's one cost effective for people to ingest all of that data? And two, how do we ensure that people understand what data ties back to value for the business? So, you know, a lot of times I've seen people will go in and instrument, um, will inject auto instrumentation and all of a sudden they have, you know, a hundred metrics coming out of a single application.

Maybe two of those metrics are being used by somebody or being used by an automated system. It's on us as vendors to be able to give our customers the tools, um, that then allow you to understand that you don't really need those, um, 98 other metrics. Um, you really need just two of those metrics.

Um, I don't think we've done a good job as an industry to really, again, as I said, tie back business value to the data we're collecting. And I think if you look at the notion of, you know, once I know that I only need two metrics, I can remove 98 of those metrics, I think you can go back and start optimizing the system. Uh, and hopefully we can do some of those things automatically as we get into automated agents for instrumentation and things like that.

It really is about connecting that business value to the data you are collecting, because at the end of the day, customers understand that if something ties back to revenue that the company's producing or if something ties back to business value, they can make the case for investments in that. So the issue that I find in observability is not that customers are spending a large amount of money, it's just that the champions we have in the, in these businesses don't have the ability to go back and explain to the business side to the COOs what value that dollar spend is producing for the company. Um, you know, and so using an example of a e-commerce company, it's easy to say that if you prevent two outages a year, you've saved $10 million because an outage costs me $5 million in revenue.

It's hard to do that if you are a bank, for example. Um, unless you have a regulatory, um, sort of requirement around SLAs, um, it's hard to do that if you don't have that clear line between what your system's producing and your revenue as a business. And I think that's the connectivity that we need to produce in the systems.

Where do you think AI will fit in this equation? Because in theory, these algorithms might surface insights first without me having to know the questions, but how do I get there? I think there's been a little bit of a bad name associated with the term AI ops when it comes to observability and operations.

Um, and I think with the advent of technology such as LLM, um, you know, which have caught the imagination of everybody with, um, tools such as Chad, GBT, we're not able to look at data in context, um, and be able to understand effectively patterns. And what the LLMs are really good at is, is understanding these patterns. So, you know, if you look at the work that Elastic has been doing around search relevance, as I said, as well as um, vector search, we're able to, what I fundamentally believe is, is move the needle, um, in, in a step change way, uh, forward when it comes to the, the application of AI into the world of observability.

Um, so recently Elastic launched the Elastic AI system for observability. And what we're finding is that the LMS are really great at understanding patterns and learning patterns over time. And so things like, you know, you have a set of run books that you've operated on before, what the LMS really good at is looking at the patterns and saying, Hey, these patterns implied this specific runbook, at some point they can even go a step further and start applying those runbooks to your systems.

And so that's where I think the advent of LLMs, um, has really changed the game and what we are gonna see when it comes to the application of ai, um, to the world of observability. So ultimately, what's your best advice to folks as they kind of go down the observability path? A lot of them are somewhat intimidated, but how do you get started and how do you keep going?

So the best advice I would have for folks, um, you know, that are getting started with observability or thinking about observability, um, one is to look into the world of open source. Um, so we have open telemetry that's turning into a standard. Um, I expect that that's going to become something that equalizes the playing field.

And what I mean by that is, um, today instrumentation effectively is the thing that creates lock in. It creates issues at a layer where it makes it expensive for you to consider modernization. Um, open Telemetry is effectively allowing folks, um, to start emitting data in a standard format.

And what that then allows you to do is, is start to get into understanding if that data's valuable for you or not, and then making sure that you are sending it either to the best place where you analyze the data and get understanding from it and insights or to a variety of different vendors depending on what you're trying to do. I think the other thing that's important as people think about observability is to think about that business context that I mentioned. Um, the business context is extremely important because more and more what else, what I'm seeing, especially with things like alms, is that you need to understand the operational context of your systems and those operational context then need to flow into business decisions.

Um, and so tying in the notion of observability with the business data is going to be extremely important. So as people think about 2, 5, 10 years out, you know, I would ask them to think about observability, not purely from a software operations perspective, but think about it from a business operations perspective and how are you gonna tie back what the software does to what the business is doing and be able to collate that data together in a place, um, that allows you to answer not only operational questions, but also business questions. Alright folks, well, you heard it here.

You may not know exactly what is meant by observability, but I'll tell you what, when there's an issue, you'll know you need it very much. Hey, Abha, thanks for being on the show. Thank you, Mike.

Back to you guys in the.