AI, Observability and the Coming End of Token Maxing — Inside the Next Era of Telemetry

Hey guys, thanks for the throw. We're here with Andre Scott, who's developer advocate for Coralogix, and we're having a little chat about the impact that AI is having on telemetry and observability and all the things that go with that. Andre, welcome to the show.

Thanks for having me, Mike. How's it going? It's going well.

Okay. Hey, before AI came along, we were starting to struggle with the amount of telemetry data that we were collecting. We started using things like OpenTelemetry, and we could collect more than ever because we could afford to.

But now, or at least then, we were already suffering from what might arguably be considered too much of a good thing. Now comes along all these AI agents and everything else that goes with it, and they're going to be kicking off even more telemetry data. So how are we going to cope with all of this?

What's your thought about how do we observe all of this data and turn it into something meaningful? Yes. And really interesting topic right now.

I remember my first talk two years ago, Mike, and it was before the AI really came into enterprise companies, and we still had a problem then with the observability span. This comes down to the architecture. Traditional vendors, I would say, store your data in index.

Most of the big players out there that have been around a long time. And that was whenever your infrastructure went linearly with your observability span, they kind of matched each other. But now we're seeing this real uptick in AI and more and more data going into that same architecture, which is a big problem.

We're going into an index storage, high cost. LLMs are producing maybe 50% more telemetry data than the traditional services. So I think we're in this place where your observability architecture is becoming even more important.

It's really critical now to have some sort of object storage to store your telemetry data. That's one part. But of course, there's many more layers to that which we can get into.

But I think the core thing is still the same from my talk, as I said a couple of years ago. It's all about your observability provider. Choosing an observability provider that has the right architecture for this next generation of telemetry data, which is, as we can see, massive scale.

And can you still take all that in and get actual meaningful data from that? It would seem to me the challenge, though, isn't so much storing the data because, well, there's a lot of options, and you mentioned object. Yeah.

And I'm sure we figured out how to tier it so that- Yeah ... maybe we can get to their most important data sooner. But rehydrating the right data at the right time requires a certain amount of skill, expertise, and technology.

So, do we have to get smarter about data management? Yeah, 100%. Data management is critical.

There's also another point there. Okay, so storing in object storage, that's fine. That's available, I know.

But then what do you do with that? A lot of providers can offer this, yes. But okay, you've stored it in S3, it's in Parquet format, but what can you build dashboards from that?

Can you build alerts from this data? Can you actually observe it? That's something these providers maybe cannot provide.

We have a query engine that runs on top of S3 called Data Prime, and that allows us to actually inject the data and do the observability on top, the dashboarding, alerting, log parsing rules, et cetera. So that's another thing there is on top of that is, when you still have that cost on top potentially, where you have to use proprietary software, proprietary query engine to actually use the data. So something to be careful of.

S3 is not just the solution, it's about the layer on top of that as well. You don't want to get charged twice for the compute on top, which is the query engine to actually use it. So are we going to also be in this new era with AI agents?

And part of the challenge with them, as far as I can tell- Yeah ... is that they're unpredictable, and we don't know exactly when they're going to do something and who's going to launch a prompt that will task them to do something, and we don't know how much data that they're going to go access, when and where for what purposes. So is there going to be a newfound respect for observability as a result?

Yes. It's a really interesting topic right now. AI observability is, I speak with engineers all the time at all the trade shows, and AI observability is the first thing they're asking for now.

Companies are really pushing to get this into production. As I mentioned at the top of the call, AI can compare to regular infrastructure, front-end applications, et cetera. AI workloads can generate 50 times more telemetry data.

They can touch many different tool calls, many different services, which will incur a lot more spans, okay? So that's one problem. Another one is, as you said, what are these AI applications doing, their reasoning?

To get around this, you need to first, you can't observe what you can't see, so you need to wrap it around with, I would always recommend open source. OTEL, the GenAI, semantic conventions are all out there right now from OpenTel. And then once you can see inside the black box, you can see exactly what's going on with your AI application.

You can start monitoring the sessions and the outputs. You need to put in some sort of guardrails. So with Coralogix, we have the ability to do this with our AI center.

We were one of the first in the game to have this available, where we can set custom policies around your AI application so you can evaluate it for correctness. We have got maybe 17 out-of-the-box evaluators, so you can actually completeness prompt engineering attacks. Can weCan we see if that's happening?

Can we evaluate it, and can we actually put in a guardrail and stop that interaction from happening? These AI applications have the keys to the castle. They have access to your SQL databases.

You cannot be giving that away. There's so many horror stories out there of people using prompt engineering to get through chatbots to get access to proprietary company information or even access to some of their API keys for Cloud code. I've seen this happen.

So it's very critical now, companies need to be aware. It's a different kind of world where I lived in, when I was in SRE, it was always deployed first and we'll monitor it after the fact. It can't be that case with LLMs and AI.

As you said, they're unpredictable. You need to be thinking about your observability by design, and getting ahead of it before you release it out to the public, because if you don't have evaluation and guardrails in place, you're at risk. Who's taking the lead on this?

You're kind of the developer advocate, but a lot of developers I know are just happy that they built the agent and they got it deployed, and they're not really concerned about what happens after that fact. So who's kind of stepping up here and leading the charge on the governance and the observability of AI agents and associated technologies? Really good question.

There's twofold to it. In the company-wide, and with companies and developers I'm working with, it's going to the AI committee and it's taking a lot of time, it goes through a lot of layers, politics to get decided what to do, but that's too slow. But at the same kind of thing's happening with government as well.

There's a lot of legislation coming out, and that's kind of leading companies to kind of be more aware of this, what they need to do, like the AI Policy Act in the EU is coming out pretty soon. So that's leading the way in terms of that, but then I still think it's far too slow. I think it needs to be on the developers who are developing these applications to implement observability by design.

I always say that. That means then that you're set up from the word go with an open source technology, OTEL. That means you just code it once, and then you've got your telemetry and your evaluation in there forever, really.

We've been talking about observability for a while, but I think a lot of people still conflate monitoring and observability, and they don't really make a distinction in their minds. But, as we go forward here, will we have to revisit our whole approach to monitoring and observability, and will people be looking at either different platforms, or will they be able to aggregate some of the monitoring tools and get rid of them in favor of an observability platform that, in my mind at least, goes beyond just tracking a bunch of set of predefined metrics and is a little bit more about helping me figure out what the root cause of an issue is, hopefully sooner than later? Yeah, that's a really good one.

I think to answer that in one really quick swing would be minimize your tool sprawl, bring everything together into one observability platform that can cover your entire stack. Why? Because if it's all in the one place, we're moving into this place now where I'm actually doing a webinar later on headless observability, which is observability without even a UI.

It is just using agentic platforms to talk to your observability provider and ask questions in natural language and get the answers in real time, and also to work autonomously. When an alert fires, it's able to go in and read your observability data. With us, we have unlimited retention, which is also quite critical.

You can see historical patterns, et cetera. And then it can make decisions on your behalf with a potentially human in the loop just observing the observer, which is really, really interesting. That's the way we're going.

So to get ahead of the field and to be ready for this next generation, I think, where we're potentially not even using a UI anymore, is to minimize your tool sprawl, bring everything together, try and own your data wherever possible, try to stay away from proprietary formats and proprietary vendors, and you'll be AI-ready. And of course, as I said, observability by design, always be thinking that way. We're coming into an age now where things need to be working more autonomously.

Agents are doing things on our behalf. The only way to get there is by bringing it all into the one place in one structured open source language framework. And that, as I always say, is always OTEL for me.

In theory, at least, we're supposed to be building more software than ever, faster than ever. Yeah. But ultimately, at that pace, will that just force this conversation that we've been having around platform engineering, and will that become kind of the de facto way we build software with a unified team, because the days of having 20 different DevOps teams running 20 different platforms and tools might not be feasible anymore.

Yeah, exactly. It's not feasible anymore. I think there's a really interesting trend right now where a lot of companies are deploying with the likes of Cloud Code, et cetera, and these agentic platforms doing our coding for us.

Software's springing up left, right and center. But there will come a phase where it's already happening right now, where token spend is the cost per employee, token spend-wise is crazy, but it'll have to come back to a point where it's about optimization. And this trend right now where we're seeing all this AI applications being spun up at left, right and center will come down.

I think it's all going to be about token efficiency, and that's when things will start to tighten down. Right now, we're seeing just a crazy uplift in everything. AI token maxing is everywhere, and that's, again, it's good for observability platforms.

We're seeing a lot more data come through. Businesses are spending an awful lot, but at some point, this trend has to come down, and we have to start thinking about optimizing our systems againBetter DevOps practices and engineers that really know what they're understanding the code and what they're doing, and that's going to be a critical thing. Well, to your point about token maxing, are we already moving past that point because we're starting to hear people freak out about the cost of all this stuff, and people are running out of tokens and allocations and all kinds of fun stuff.

So do we have to get better at maximizing the context window and the prompt, per se, so that maybe we're not stuffing as much data in there all the time and just get smarter about how we use those things? Exactly. I spoke about this in my podcast last week, and this is now the trend.

Well, not the trend right now, but I foresee this coming. Right? So the proper practices being brought back in, optimizing our token spend.

I've seen some companies that are spending billions on tokens per employee. That just can't happen anymore. There will be a point where OpenAI, Claude, Anthropic, et cetera, will be pulling this back in, and companies will have to be wiser about their spend.

So yes, that'll mean, as we said there before, better prompt engineering, better testing, being more efficient and understanding exactly what the code is doing, and then taking it offline. Not just relying on AI to build these applications. It's another thing.

But yeah, we're moving into that point, I think, soon. And to your point about that, most of the developers I know are, they're not wanton spendthrifts. They basically, if someone tells them that the cost of something is X and they should pay attention to it, they will actually do the right thing.

So I often wonder how much of this is just an issue of the fact that we don't put the right metrics in front of the developers so that they can understand what the cost of these things are before they go launch all this stuff, or are they all just pretty much drunken sailors at the free bar? Yeah, it's an interesting point, right? Observing coding agents, observability is coming around now.

We are seeing customers spin that up and demanding that. They want to see their spend in there. I still think we're in this period now where it's still enjoy token maxing, whatever, have your fun, but as I said, it will come back where this can't be viable anymore.

The amount, the cost of compute this is costing companies, it can't be viable. It has to come to a point where it's like we can't use a million tokens for this app every hour. You just can't keep on doing that.

It doesn't make sense. It needs to be taken off the agents and into actual proper DevOps practices, for sure. So, yeah.

So let me ask you this. " Well, look, having log data in frequent search when they hardly ever look at it. So in an index, which is crazy, still happens.

High card-only metrics that are sat in an account that are not being used. Metrics that are not being used. Stale metrics is another thing.

Stale dashboards. The observability gap is a massive thing. So I see companies keep adding observability data as we know now, more and more data.

They think that's better. It's also not. It's just causing more and more of a gap.

I, as an engineer, as an SRE, are not going to go through 90 alerts, 100 alerts a day, 1,000 alerts a day. It's just not going to happen anymore. I'm going to miss that.

I'm going to skip some of them, okay? I'm going to miss some critical things. People need to start optimizing their workloads and really looking at the observability data and doing a refresh.

Any companies that are coming, also migrating from one of the big players, they see it as a real chance to start fresh, and reduce the amount of alerts they have and dashboards. And yes, also the fact there is using something like Oli, or AI agentic solution, where it can take in all this data for you and do the work for you, of course. That's grand.

But I think as well, though, there is a massive observability gap there, and companies, yeah, keep adding data, but they do need to start thinking about their practices a little better. All right, folks. You heard it here.

There's going to be more telemetry data to collect and store than ever. Important thing to remember is it's not all created equal. Hey, Andre, thanks for being on the show.

Cheers, Mike. All the best. All right.

And back to you guys in the studio.