Reducing Cloud-Native Observability Costs: A Path To Long-Term Control | Cloud Native Now 2023

Thank you everybody for attending today. Uh, as is gonna be, um, pretty, uh, in gonna jump into a demo if John, you wanna jump to the next slide? Yeah.

So Before we dive straight into demonstration and product, just do a quick overview of what chronosphere is, what we do. Uh, so we are a observability platform. We support ingesting metric and tracing data in open source formats.

So if you're using, uh, things like Prometheus Open Telemetry today, uh, we're fully compatible with, uh, with those sources. Uh, and the main thing that separates chronosphere from, uh, other solutions that you might be familiar with is the control plan that you see in the middle here. Uh, this is what we're gonna focus our, our conversation around today.

Um, this is gonna let, uh, you as a customer analyze the data that you're sending to us, um, shape it to, to match your needs, and, uh, get a lot more overall understanding of what value you're getting from your data, and a lot more ability to manage it as your organization grows and your, uh, observability needs scale. Um, won't talk too much about the, the data store underneath it. Um, just know that it's, uh, based on the open source M three db, uh, technology that was developed at Uber.

Um, it's been proven to scale to, uh, as highest 2 billion data points per second. Uh, and we are also, uh, a hundred percent open source compatible, uh, on the, the consumption side. So, uh, a hundred percent prompt QL compliance, uh, able to use things like alert manager definitions for alerts and ANA dashboards for, uh, for visualizing data as well.

Uh, and Scott, I don't know if you wanted to add anything on top of, uh, the, the brief overview there? No. You mean we're gonna go over the control plane, which, you know, uh, is really important in the new world of cloud native.

And so we're gonna walk you through, um, a lot of the features, but also talk to you about why they're important in the, um, you know, with regards to, uh, cloud native. So if you're new to this, uh, you're just getting started, um, I'm gonna give you a preview of some of the things that the, the challenges you might run into, and this is why some of the features that you'll see today we built to address those. Yeah, absolutely.

Uh, and so if I jump forward here, there's couple of our use cases that we wanna cover today. There's a lot of content here. Um, but just to, uh, narrow our focus to a few different specific areas, we're gonna talk about how you can use Kronos sphere's, uh, quotas capabilities to, uh, contain, uh, data explosions when you have, you know, uh, unexpected cardinality, um, that would result in overages or downtime and other systems.

Uh, we're gonna talk about how you can analyze your metric traffic to understand, uh, how expensive it is, where it's used, who it's being used by, uh, and really understand the, the cost versus the utility, uh, that your data is providing to you. Um, we'll talk about how you can apply, uh, aggregations to shape that data, uh, so that you can better align it to the value that you're actually getting from it practice. Uh, then also how K chronosphere helps to, uh, improve performance by automatically optimizing slow queries that are, uh, kind of showing up in your, in your workload over time.

So, why don't we go ahead and jump right into it and we can start talking about, uh, how we address all these, these things here. Lemme jump over to the product and just confirm that's showed up for you, right, Scott? Yes.

Awesome. Yeah, so let's get right into it. Um, first thing I wanna talk about is when you're dealing with high cardinality data, particularly in cloud native environments where you have a lot of ephemeral infrastructure, uh, it's really easy for, uh, you know, seemingly innocuous changes to introduce a huge increase in the volume of data that's being produced.

Uh, cardinality explosion. Uh, and one of the things that we do in CHRONOSPHERE is give you the ability to very easily manage, uh, when this happens, reduce the scope of who is impacted, uh, understand what happens very quickly, and trace it back to the source so that it's impact on everyone is limited. Uh, and the primary way that we, we kind of do this is through the ability to set quotas in your environment.

So what I'm looking at here, we've got a dashboard showing quotas for, for my tenant. Uh, I've got a overall capacity that I've set, uh, in chronosphere at 50,000 data points per second. Uh, and I can break that capacity down, uh, into groups based on any shared aspect of my workload.

So I've got a handful of applications that are, you know, uh, sending data into my, my tenant. So there's a, you know, portal service, there's a, uh, ordering service, there's an authentication service. Uh, each of these have their own, uh, quota of data points per second they consume.

Uh, I've also got a, a large chunk of, uh, default quota available for, for new services or use cases that come up. Uh, and if I scroll down in the dashboard, I can see the utilization for each of these pools of capacity. Uh, if you look at the off service, for example, it's got 2000 data points per second guaranteed to it.

Uh, that capacity is, it's to use, uh, and right now we see it's, you know, about, uh, half that being, being actually consumed. Uh, contrast order service, same quota, a little bit higher utilization, not too much. Both of them are well underneath the, uh, the commitment, uh, that's been made for, for those services.

And I, I say this in terms of a commitment because the way that this works is if I am the order service, uh, and I have 2000 data points per second that I'm allowed to send, and there is a cardinal explosion somewhere else in the system, uh, say the off service adds a new metric or a new label, and the volume of data that is being produced in my tenant increases, you know, significantly 10 times or a hundred times, and we exceed the capacity for the entire tenant that I've set. Uh, it's not going to impact my order service here at all. Um, my capacity for the order service is guaranteed.

The system is going to protect it, it's gonna protect my experience as well. What happens is the off service, or the problems introduced is where we're actually gonna see, um, uh, the, the quota impact take take place. So I'm allowed to exceed this 2000, uh, rights per second quota for the off service as long as there's room available in the tenant overall.

But when we hit that, that threshold, um, the top line that we saw at the, at the very top of the dashboard, uh, we'll actually start to enforce this quota and it'll just start to shed additional data points, uh, that are being written in by the off surface. So in the context of a cardinality explosion, something like that, uh, what this means is that only the off service is gonna get impacted. Everyone else's capacity is protected.

Uh, and because we have information showing how much data each service each pool of, uh, quota is emitting, uh, it's very easy for me to track and simple alerts around this as well. So if my off service is getting close to its quota limit, if it exceeds it, if we start dropping data, these are all very easy conditions that we can look for and alert on so that the owner of that, uh, service gets notified very quickly and they can go in and take a look at what's happening in more detail. I think a good point here to make too is that we're kind of shifting like the, you know, we're delegating responsibility, right?

Something that hasn't really been, uh, done before, it was generally developers would, you know, uh, deploy, create metrics. All metrics are important, right? So nothing ever got dropped and everyone just kept consuming and the price could cost and data volumes kept going up and up.

They never had any context for understand how their changes were actually making, uh, or are impacting the overall system. Uh, and so I think what this helps to do is start to put some focus by those service team developers on what the metrics that they're, uh, creating are actually how much volume they're, what are they costing, right? So they can start to see that.

And then I think is also, um, why don't we, do you wanna talk a little bit about, uh, what's coming in terms of like, prioritization, like how that can help them to so that it's not just a, uh, you know, oh, the most important metrics are being dropped? Yeah, absolutely. Um, yeah, so the way that this works, right, is it's designed to reduce the scope of an issue.

If somebody introduces a problem and, uh, all of a sudden the volume of data is, is completely out of whack, we're gonna, uh, minimize, you know, who can be impacted by that, uh, to just the area where it was introduced. But that doesn't mean that we want, you know, the off service to have a, uh, all hands on deck fire. Uh, you know, we wanna make sure that they can still get critical visibility into that service.

So you have critical metrics that are part of this workload that are gonna be used for things like, you know, alerting you about, uh, how the system is, is behaving, and, uh, making sure the service is available and, and not, uh, in an error state. Those metrics we wanna preserve, um, other stuff might be less valuable, and that's where we want to, uh, to apply quota enforcement. And so that's something that's actually, uh, coming soon in, in the platform, is you'll be able to enforce priorities and say, these are the metrics that are high priority to me.

I care about them, I use them, my alerts. The other things are, uh, less important. Uh, it's okay if that data actually gets dropped as part of quota enforcement if there's an issue.

And I do wanna, uh, to stress that this is really about handling that unexpected case where somebody makes a change and it has a way bigger cost and impact on the system than you would expect. Uh, it's very easy to go in and adjust the capacity available to you in Chronosphere. Um, you know, the, the kind of expected case that people have when you sign up for like a pay as you go service is that, uh, whenever there's an increase in your utilization, it's by design, right?

Uh, and that's no different with Kronos Fairwear. It's very easy to go and, and, uh, increase your utilization by design. But when it's not by design, you don't want to have a vendor turn around and take all that data and then bill you a huge amount, uh, for information that was ultimately not, not really valuable, not what you're trying to do.

Uh, and that's what we're trying to, uh, to correct here is that, uh, you know, not expected case where things do not go as planned. Yeah. And to reiterate, if there is, uh, extra capacity in the system, by default, you will be allowed to exceed that quota.

You'll get notified, um, and then you have time to remediate it. But it's in those cases where you're maybe right at the edge where it's a hard core to set. So, but that's up to the, um, administrators.

Yeah, exactly. Cool. But there's also great thing, there's also things you can do, right?

So if this is happening, then we're gonna move on to that next, which is interesting cuz it's not just, oh, you know, it's gonna get dropped and it's nothing I can do about it. You can actually do things and just start to analyze and look at it and make changes. Um, and so why don't you go into that next?

Yeah, absolutely. So I mentioned before that you can alert on, uh, when you're getting close to your quota or when you're exceeding it, uh, just to show an example what that would look like. Um, here's a, an alert that we've got defined for, you know, if somebody's, uh, exceeding their quota, uh, in the system.

And when that happens, uh, you can set this up so that, you know, notifications are dispatched to the, uh, members of the team that are responsible for that service, that portion of the workload. And they can go into chronosphere very quickly and understand, you know, what has shifted in their traffic that's causing them to, uh, send more data than they anticipate. Uh, one of the main things that, uh, we have available on the platform to help with that is our metrics analyzer.

Uh, there's actually a link in my work that takes me right there. Uh, what this will let me do is I can take a look at the live traffic that's actually coming into chronosphere in real time and understand the metrics that are involved, how much they cost, what dimensions they have associated with them, and what the cardinality of those dimensions looks like. So you can see here, for example, I'm looking at the ordering services, uh, traffic.

So here's all the metrics that we see that are associated with that application. On the left-hand side, here's all the dimensions that we see, uh, associated with the metrics that are, are coming in and their cardinality. Um, there's some obvious thing, obvious things in here like name cluster, availability zone, uh, and I can sort these to see what the highest cardinality, uh, dimensions are, like instance here, 34 unique values.

John, um, we have a question. Um, and please do ask questions. We should just put this out at the beginning.

Uh, someone asked, um, what does cardinality explosion mean? Maybe we should define that just to be clear. Yeah, absolutely.

So a cardinality explosion, uh, what we mean when we talk about that is, is, uh, if you have, uh, a metric, um, that you're, you're, you're tracking, um, say let's, let's do something, um, specific. Say you're looking at, um, the request per second going into a particular service, you're gonna have different dimensions, uh, that you add to that metric to understand, uh, request for second broken down by different things. Uh, like the endpoints that that service has is probably gonna do more than one thing.

Uh, so you'll have broken down by endpoint. You might do things like HTTP method is another dimension that you've had that has multiple values. Uh, the status response, uh, for those requests that are going into the, uh, the application, all of these dimensions that we add have some number of values that, uh, you can track them by.

And the total across all of them ends up being the, the cardinality of the metric. Um, and it's not additive, it's multiplicative. So if I have one service with five endpoints and there's, uh, two different httv methods for each endpoint and five different status codes, then you've got five times, five times two, uh, different, uh, time series that you're tracking.

Uh, in that example, you start adding additional instances of the service, uh, then you're gonna have that for every single, uh, container, every single instance of the service that you're running to. If I go and add another dimension to it, like say, Hey, I wanna know what users associated with the request, that might be something that is way too many unique values. There's too many users that could be requesting in the service where the number of Times series that you're tracking has now increased significantly.

Uh, let's say there's, you know, 10,000 users that could be, uh, requesting of the service. You've got five times, five times, two times 10,000, uh, that would be a cardinality explosion. We've, we've significantly increased the amount of data that is being tracked, uh, uh, by, by that particular metric, uh, in the system.

Or you can add a new metric. Uh, it could be, you know, just purely, uh, starting from, from nothing. Uh, and it's just much more expensive than you expected because there's more unique values, uh, than you anticipated.

And these are pretty common in a cloud native environment, right? I mean, yes, as you break teams into smaller teams and they're focused on smaller services doing independent employments, frequently, uh, cardinality explosions become kind of the, the norm because people are introducing new values and metrics and not understanding how they impact it in production, right? Yeah, it's super easy to do this.

You don't understand the cardinality of a metric, uh, as it exists. You can add something that's seemingly innocuous and it turns out it has a really big impact because that was an expensive metric already. Um, you can add something and not realize what it's gonna look like in production.

Um, you know, it's very easy to go and do something and test it out in a local dev environment or a staging environment where the cardinality has been much smaller. And then production, you know, proves that your, your staging environment is not, uh, production like, and so you got, you know, a hundred times more, uh, time series than you were expecting. This happens all the time.

Uh, and to your point, Scott, the more teams, the more services, the more changes you have going going on, um, the more chances there are that somebody's gonna make a mistake and do something that's really expensive. Yeah, and that's one of the main use cases for this, uh, this analyze incoming and the metrics analyzer. So, uh, that's what we're talking about here.

Yeah, exactly. If you have a cardinal out explosion like this, um, we've talked about how you can limit the, the scope of it, limit the impact, um, only the team that introduces the problem is gonna get impacted. Everyone else is gonna have the same experience they've already been having.

You can notify that team when it happens. They can take a look right when, uh, the problem is introduced and not have it be kind of working behind the scenes. And when you do have something like this, uh, show up, it's gonna be really obvious here.

Uh, I don't actually have a, a, a cardinality explosion on my screen, but you can see that it's very easy for me to see, uh, dimensions with a lot of unique values. It's easy for me to see if there's a metric that is particularly expensive in terms of the number of data points per second that is generating, uh, those things are gonna pop right to the top of the list. And so it's gonna be kind of like a flashing neon sign.

Uh, when you're coming in here and looking to see what caused an issue. Um, it, it'll be very obvious. Uh, and I wanna quickly highlight as well that you can break down the, the workload that you're looking at in the metric analyzer by, uh, whatever you want.

So I'm looking at, uh, all the metrics from the ordering service right now, but if I want to see data broken down by, you know, each cluster in my environment, I can change the, uh, groupings here. Do I turn this back on? It'll actually show me for each cluster, how many data points per second are we getting from the ordering service, for example.

Uh, so it's very easy for me to, to mix this up, um, and get different subsets of the workload and understand, uh, across any, any metrics, any dimensions, any portions of my, uh, my metrics. What is the cost? Where is the, the cost coming from, uh, what is involved in that, uh, and what dimensions are are driving it, uh, over on the, the left hand side here.

So being able to, to understand this, uh, type of information, it makes it a lot easier to manage these types of problems that can occur in, uh, more complex environments where you've got a lot of changes going around and a lot of services, and it's hard to track things back down to the source. Uh, Kronos Air makes it really easy. Uh, once we, we, we do that, um, you wanna focus on making sure that what you've already got in the system is, uh, efficient and is beneficial and is good and is, uh, is worth the cost that you're paying for it.

So being able to, to quickly identify when there's a problem, and, uh, undo that problem is one thing, but being able to look back at your historical choices and say, have I done the right thing for the business, uh, as a whole matter entirely. So, and talk, Sorry. And talking about the addressing the problem quickly.

One of the unique things about K Chronosphere is that you actually don't have to have developers involved when you want to, to make the, if you want to do something, to, to stop a cardinality explosion, right? Yeah. Uh, basically you can do that.

Um, write a, write a creator shaping rule. Do some, do something. You can drop the metric, uh, no developer.

And we don't only charge you for what you're storing, not what you're ingesting. I think that's another key differentiator. So this is, you're seeing this coming in real time, and you can actually drop that metric before it hits your database.

You're still actually, uh, it's still being, uh, sent, it's just not being stored and you're not paying for it. Yeah, absolutely. Um, I'll, I'll show an example of what this looks like in practice, uh, in a, a couple minutes as well.

But, um, when you have this sort of a problem and reverting the change that introduced the, uh, the issue is not, uh, super practical, um, we don't necessarily have to go and, you know, fight down that road. Uh, you can, if it's a new metric, write a drop rule to filter it out. We just won't store it.

Um, we'll go back and fix it, uh, and then undo that, that drop rule. If it's a new dimension on an existing metric and you don't want to lose the whole metric, uh, we can write an aggregation rule and that'll allow us to actually, uh, remove the instance from the dataset before we store in the data database. Uh, so you can go back to the status quo, uh, using shaping rules in K chronosphere without having to go back and undo the change in code.

Um, or you can do, you know, one, then go back and do the other, or, uh, the other one is time permits. Um, resolve the issue as quickly as possible, rather than having to, uh, do changes that are gonna be, uh, you know, potentially impactful the production, uh, in order to, to do that. Yeah.

Uh, so let's talk about how we can understand, you know, whether or not we're getting the best value out of our, our data. Uh, in addition to being able to understand incoming traffic here in the metrics analyzer, we do also have a tab to understand usage of data that is, uh, being sent to the system. Uh, and so by default, this is gonna look at your least valuable metrics.

Uh, you can see an example here. I've got this service DB request time metric. It has no configuration references.

We're looking across all the dashboards, all the alerts, all of the recording rules, and, uh, other shaping rules in chronosphere. We haven't found anything. Uh, we were looking for any direct executions that me, uh, reference this metric.

There's nobody looking for it, uh, either, you know, using our A T I, uh, using the Metric Explorer. Uh, it's obviously not any dashboard. Someone's querying it from there.

Uh, it hasn't been been read in the last 30 days. Uh, you can see in the top right hand corner here, uh, we take references and, uh, gray executions and turn it into a utility form. Uh, in this case, since there's no references, no executions, no utility, um, but it still has cost.

Uh, so this is, you know, 430, uh, give or take data points per second. Um, and no one's looking at it. If you're being asked to improve cost efficiency, reduce the, uh, cost of observability related to the cost of your infrastructure, for example, uh, this is a pretty obvious place to start, right?

I can write a rule to drop this metric. If anyone comes back and says, Hey, I needed that, uh, you can say, well, why weren't you using it? Because no one's used it in the last 30 days.

Uh, so it's very easy to go and find, uh, these unused metrics that can be working in the system. And we see this all over the place with our customers. Um, it's very easy to generate metric data that flies beneath the radar, because when you have so much data, uh, it's really hard to, to sift apart and find just what's valuable versus what is actually not super valuable or not even being used at all.

Uh, it's just a factor. Again, the, the complexity of the environments that we run and the sheer volume of information that we tend to collect from systems. Looks like you have another question that just popped up.

I see, Uh, so you're asking, is there a way to detect and control sudden increase in metric points for given service that could start throttling other important metrics? Yeah, absolutely. Uh, we have fine grained visibility into, uh, the data that's coming into chronosphere.

So in addition to the, uh, usage analyzer, uh, you can break information down by, uh, arbitrary labels. We showed this a little bit, uh, when we talked about quotas. So in my tenant, for example, I have the data points per second being generated by every surface in, uh, microsphere environment.

And so if I wanna set an alert on that, it's very straightforward for me to do so. Uh, and because we're, we're fully compatible with, uh, things like prom, ql, you can write very expressive ex uh, alert queries. So it's, uh, quite straightforward actually for me to do something like compare the data points per second for all of my given applica, all my applications, uh, to what it looked like a day, or a week or an hour, whatever timeframe you want historically, uh, in the past.

And if I see more than a say 10% jump, I want to get notified about that and take a look at what's going on. Um, that's, that's a completely reasonable thing to do. It's something that our customers do too in practice.

So finding metrics that are, are just not being used at all, uh, is a great way to, uh, you know, trim back, unnecessary spend, uh, and, and be a little bit more efficient. This is definitely something that, uh, I'm sure is interesting to a lot of people right now, given the, uh, widened focus on, on cost efficiency that we're seeing across the industry. Um, there's also, you know, the, I would say, more interesting use case of show me the metrics that are, um, being heavily utilized or are, uh, you know, expensive.

And I wanna see if they're actually justifying their cost. It's not that nobody's using them, it's that I want to understand if, uh, they're worth the cost that, uh, that we're seeing for them. And so we can do all of all of these things.

Um, we can rank our metrics based on, uh, costs. We can rank them on utility. Um, we can compare cost and utility and come up with a, a definition of what is the most valuable.

Uh, the, the metrics, uh, analyzer is designed to, to help with all those, these cases, uh, and also show you can search, uh, across these. So, um, if you recall when I looked at the, uh, the analyzer for incoming traffic, I saw a lot of, you know, service underscore metrics, uh, however I might ordering service. So I can search for all of those and see what they look like, uh, and what their utility looks like.

And for the ones where they're being used, we take a look at, uh, an active example here. Um, here's an example of a metric that actually has a, a fair amount of utility, right? Uh, it's being used in a lot of different dashboards.

It's got a lot of different, uh, query execution specific. Um, it is, you know, more data points per second, but we can see, uh, uh, where the cardinality is coming from, uh, for this metric and for each label, uh, on the metric as well. How many unique values does it have?

Are those values referenced, uh, anywhere in the configuration? Are people querying for data by this dimension? Uh, or is it a dimension that doesn't actually add utility?

So, uh, this makes it really interesting because we've got data that is clearly valuable to us. Uh, we're, we're querying this metric in a lot of places, um, but we're looking at it by cluster or breaking it down by application name or by availability zone. We're not looking at it by instance.

And if I look at the dimensions on this metric instance is the, uh, highest cardinality one, this is contributing a lot of cost, but it's act not actually driving any value for me. Uh, and we can go and confirm that by looking at the details here as well. So if I jump into usage details, uh, you can see all the information that we track about your, uh, the, the metric in your workload.

Here's all the dashboards, uh, that this metric is being used in. Uh, and I can expand it and see all the queries that it is, uh, referenced in. So it's being used in multiple, uh, panels on the dashboard.

Uh, you can see all the different, um, references to it if people are querying the metric directly. I can see what those queries look like as well. I can see who's querying it.

Uh, you can see all the unique users, uh, and how many times they've created this, uh, this metric and where they've query it from. So it is really detailed information to let you understand where's it being used, how is it being used, who is using it, and how is it actually changing over time too? Uh, so when you have a metric that is, uh, being used, but it's also very expensive, we can start to have a deeper discussion about is this justifying the expense or are there ways that I can make it more efficient and less costly to the business?

Uh, and this is again, an area where, uh, the ability to do shaping on your workload can come in handy. So for my h p request time, Ms. Total metric, I clearly am not using the instance label.

Uh, it's not being used anywhere in the system that we can, can see in the last 30 days. Uh, this seems like a great candidate for if we could just store this metric without the instance label, it'd be significantly cheaper. Uh, and shaping rules, the aggregation rules and chronosphere let you do just that.

Uh, and in fact, I actually have, uh, an example of it, I think, in this list. So here is the same metric broken down by application name, for example. Uh, the, the roll up version isn't actually being used, uh, cause it's a, a new aggregation, but you can see the data points per second that's being generated as result of aggregation is tiny.

Uh, basically one, one data point every so often cuz there's so few unique time series. Uh, the cost is, is minuscule when we look at just, uh, a particular dimension compared to the original. Uh, and you can make multiple different aggregations to serve different use cases, uh, for things like performance as well.

Anything you wanna chime in, uh, on the metrics analyzer? Scott? I I just wanted to point out that, um, for the rollup rule, right?

Talk a little bit about one of the things somebody will say, well, recording rules can do a lot of the aggregation stuff too, right? But record rollup rules are something that, uh, we've created that there's a little bit more unique. So it talk, it allows you to drop data if you choose to do that, whereas a recording rule won't, right?

Yeah, good call out. Um, with recording rules, you're writing data into, uh, the storage platform, then you're querying that data back out in order to do, uh, aggregations are shaping on it, and you create a whole new metric with, uh, chronosphere with aggregation rules. We're actually doing this in streaming fashion as the data comes into the platform.

So I have the option to stop storing my original version and only store aggregates, uh, that are, you know, omitting certain dimensions that I'm not using in the system. And that way you're able to get a much, uh, lower cost, uh, in terms of data points per second. Um, much better performance.

There's less data that you have to, to query in order to load your, your dashboard, uh, without having to store all of that raw data originally. Uh, and as you mentioned earlier, Scott, with chronosphere, the data that you persist in the system is what matters from a a cost perspective. So when you're making these shaping decisions, when you're doing aggregation, if you drop the raw data, then you can save a significant amount in cost, and we can help show you exactly where it's, uh, beneficial to do that by helping you identify these unused dimensions in your workload that no one's actually querying on against.

Um, anywhere you come in the usage analyzer and you see there's a, a dimension that just has no utility, no references, uh, it's a, a candidate where you can go and add an aggregation rule to, uh, to remove that dimension and save money. Uh, and these rules that we've mentioned a couple times here, they can be very targeted. Uh, you can roll up a specific metric and remove, uh, a given dimension.

You can also create some broad policies, uh, in the system as well. So administrators can do things like say, Hey, uh, I'm pretty sure we're not using the instance label across any of our services, so I'm gonna go ahead and create a rule that says, for all of my application metrics, I wanna go ahead and remove the instance dimension and drop the raw data. That's gonna give us huge savings because we run a lot of containers.

Uh, and if anyone actually needs that per instance, uh, information, they can let us know and we can go in that exception of the rule. I think that's also to read, we talked about this earlier, but you're doing this, you're, you're dropping it, uh, or you're, you're aggregating it away, but it's still being sent. So if you need that back, it's simply making a modification to that rule that allows it to be, uh, sent back into persistent in the database again so that it's not being taken away and it should still be coming, but if you don't, you're just not paying for it anymore or storing it.

Yeah, absolutely. It's, it's very easy for you to, uh, to go and turn it back on if you, you decide that you actually need that dimension you've removed. Um, because this is being done server side, it's not a case where you have to go and make a bunch of instrumentation changes in order to, uh, to to ship the workload.

Um, we're doing it all for you in a much more, uh, flexible fashion. Uh, and with real data to help you make these decisions, you can be confident about what the choices that you make, uh, when you're able to pull a report and say, look, no one's used this in the last 30 days. It's not referenced anywhere.

Uh, and if we, uh, get rid of it, it's gonna reduce the cost of these metrics by, you know, 50%. That's a pretty compelling case to, to, you know, I think any organization. Uh, and it's exactly the kind of data that you'll have at your fingertips with Chronosphere.

Yeah, I know we're, we're probably getting close to the end of our time here. There's one last thing that I wanted to highlight, which is I alluded to the fact that, um, these aggregation rules can improve performance. If you, uh, create a aggregate with fewer time series fetching, the data that you, uh, are using to your dashboards is gonna be faster.

Uh, we actually have a background process that will do this automatically for our customers to speed up slow queries, uh, in the system. So we call this the, uh, query accelerator, uh, and there's actually a dashboard that we have that shows, uh, what the results look like in an environment. So this shows me, um, there were about 900,000 queries executed in my environment over the last week.

Uh, we accelerated about 28,000 of them. Uh, that's what this pie chart here shows. Uh, and the total savings in terms of latency is about a full day of user time.

Um, that's pretty significant, uh, time that could have been spent waiting for, for queries to load. Uh, you can see some other statistics here, like, you know, what is the average time of the original query versus after we've done the, the acceleration? This is a background process.

It happens, uh, transparently based on the, the data that we're already capturing about the system and the workload. Um, when slow queries, uh, show up in the workload, the query, et cetera, go ahead and transparently make them faster for particular runs. And that way, uh, developers just get good performance.

They don't have to worry about making things faster themselves, uh, it happens for them, uh, and they get to re you know, pretty great savings in terms of, uh, time spent waiting for data. I think we covered everything that we set out to, uh, to discuss here. Is there anything that I'm missing, Scott, anything else that we, we should show before we wrap up?

I don't think so. As, as you said, we're a little over time, but if, uh, anyone has questions, we're happy to answer them now or to dive deeper into an area if, uh, anybody would like just, uh, post it in the q and a. Um, we'll wait maybe a minute and, uh, if not, then we can wrap it up.

Yeah, absolutely. I'll also note that I know this was a, a fairly short session, a bit of a whirlwind tour of a, a handful of, uh, features in the platform. There's a, a lot, uh, that we can, can talk about in Chronosphere if there's something that you want to follow up and get more information about, or if there's something that you, uh, you saw that you want to get, uh, a bit of a deeper look at, um, please do feel free to reach out to us.

We're always happy to talk about, uh, what we've discussed here today in more detail. We can, uh, end the demo. All right, and I think that's it.

Thank you everyone for your time today. Hopefully this was, uh, useful and educational and uh, have a great day everyone. Thank you.