OpenTelemetry Graduation Sets Stage for AI Observability

Hey guys, we're here in Minneapolis at the Open Source Summit, and we're having a little chat about OpenTelemetry and OTel with my buddy Chris. How you doing? Good to see you.

It's been a little while. It has been a little while. We're celebrating something here.

OpenTelemetry- Yep ... AKA OTel, is graduating. Yep.

Now, I think if I recall, it accepted in 2019. Yeah, it's about seven years. Yeah.

Seven years. Yeah. " Yeah.

There you go. I love it. Yeah.

It's been a long time. Seven years is a long time for an open source project, but OTel had some interesting history that I don't think the whole world necessarily knows the origin story that we used to have. One of CNCF's earliest projects was a project called OpenTracing, which came from a gentleman called Ben Sigelman, sharp guy out of Google, out a company called LightStep, that was trying to revolutionize how distributed tracing was done.

There was another project out of Google called OpenCensus at the time- Mm ... which was mostly focused on logs and metrics, and there was overlap between these projects, and there was, I would say, competition, debates, a little mudslinging sometimes. They were both trying to solve similar problems.

CNCF, at that time, we actually worked and brokered meetings amongst those communities because a lot of our members and users were like, "We really just want one set of standards when it comes to observability. It's silly that these projects are fighting, competing," just like the early days of containers, where people just wanted one image standard, not multiple. So we brought these folks together.

I still remember this meeting, because it was in the early days of CNCF in the Linux Foundation office in San Francisco. It was like a U-shaped table. You had OpenTracing on one side, OpenCensus on the others, and CNCF refereeing in the middle.

And we're basically a broker, meaning like, "What's missing? What's not working? " And that genesis, that meeting was basically the genesis of OpenTelemetry.

The projects decided to merge together. Mm. OpenTracing and OpenCensus merged, became OpenTelemetry, and seven years later, after that meeting, the whole freaking industry, of observability vendors and hyperscalers, all support OTel almost natively at this point.

It's amazing how much the world has changed because of that. I mean- Yeah ... I remember back when we had APM systems.

Yes. Yeah. " Yeah.

So eventually they came to the conclusion, I think it took a while, but all the observability platforms now support it. Yeah. Plus, I've also noticed that it's getting support from traditional monitoring companies.

Yep. Anybody who needs to collect telemetry data seems to be now using it. Yeah.

And in fact, I think we've gone full spectrum. " Yeah. " Yeah.

That's a separate problem, though. It is. That's not an OTel problem.

That's a good problem for vendors if they charge based on True that ... data being collected. True that.

Yeah. So, how does our approach to telemetry data need to change or evolve from here, now that we've got it graduated? As you look over the horizon, what's next?

So, I think for OTel, OTel has definitely matured over the years. Initially, when OTel started, there was this saying around there's three pillars of observability, right? You have logs, metrics, traces, right?

That's kind of been bread and butter for a long time. Recently, they added a fourth pillar, profiling, which is at the CPU, GPU snapshot level, because obviously if you have a service that does a bunch of things, hits a database, that's something you kind of want to know underneath. What is it hitting?

What is pegging? So they added that fourth type to the pillar of observability, which is good. So OTel continues to evolve.

The other thing that's happening is you can't escape the rise of AI these days, right? There's agents all over the place, AI workloads. Those things need to be observed and instrumented, right?

Mm-hmm. And so I think OTel will evolve to go support those use cases, right? There's already some efforts actually happening within OTel and even outside of OpenTelemetry that are trying to expand OTel to support full traceability of, say you do a prompt that spins up an agent that does a bunch of stuff to your database, having full traceability of that, right?

What prompt, what model was used, and that's not an easy problem, and that's something that's being worked on. So I see OTel has dominated the traditional observability world and cloud native world. Now it's going to go work and evolve to support the future workload, which is essentially AI- All right ...

based. Speaking of AI- Yeah ... any time you have people talking about large amounts of any type of data, AI immediately comes up.

Yeah. So, is there going to be an opportunity or maybe a need for something that feels like an LLM for telemetry data, and is that something that comes together? LLMs already are really good at analyzing structured data, and OpenTelemetry is all about having structured data for you to look at.

So, for sure, I think these days, they're already pretty good at parsing out that data. But is a developer writing a prompt and piping in a lot of OTel-related data a good user experience versus a Datadog single pane of glass, like a more vendor good user experience? I'm not sure.

Underneath the covers, most of those vendors for sure are going to be using LLMs to help provide insights and analysis based on all that structured-... data for sure. It's already happening.

And how closed loop do you think this will all get eventually? Because in my mind I'm like, all right, there's this massive amount of telemetry data. Yeah.

There'll be AI agents monitoring the telemetry data. Correct. And when they see some anomaly or some issue- Yeah ...

they'll act on it. For sure. There absolutely will be remediation and so on.

I think it's tricky, you have to stitch everything together and have it all work, and there needs to be standard ways to do things. So I think it'll eventually get there. I think the hard part was first getting all the industry to agree on how to structure data to store logs, traces in a very standardized way.

And we've accomplished that. Data is being stored in all different types of sources, right? Some people will store it in S3 buckets, some people store it in ClickHouse.

Pick your thing like that. People store it everywhere, but how you analyze that and how you maybe link it to fixing problems in production, that full loop thing is still being done. There's actually startups that are playing in this space to try to figure it out.

When I talk to people- Yeah ... they're all generally familiar with logs and metrics. Traces is still a little bit of a- Yeah ...

dark art for them in their minds, and- Yeah ... profiles is even further down. Yeah.

Profile's a little bit lower on the stack. Yeah. So what will it take to push people further down that maturity curve and get them to understand where traces fit in the- I think the agentic world will hopefully bring that to the forefront because people really want to understand what things the agent potentially is hitting on a very reasonable timeline.

Like, oh, you do this, the agent goes, calls a database, that eventually goes call hits your CA. That whole trace thing is super critical, I think, to really understand what an agent is doing and what potentially it shouldn't be doing, right? Having that full traceability, I think is going to help build guardrails and other things that are required to really understand what agents are doing.

These days, I'm sure you play with Claude and Gemini, we all do, right? Mm. And you get your bill from Claude, it's like, I used a bunch of freaking tokens, what the hell?

What caused all that? Usage is very difficult. If you notice, your Claude bill is not very detailed, right?

And I think trying to trigger from token usage to maybe business value and that full trace is a very difficult problem, but someone is going to have to figure that out eventually, because no one wants to just go spend a bunch of money on questionable- So- ... business value ... OpenTelemetry becomes foundational to tokenomics.

Is that how we're thinking about it? I don't want to put that out there, you put that out there. But I do feel like eventually where that will go, it'll be very crucial to figure out how to optimize both your agent workloads and figure out what business value is.

Even these days, answering what is intelligence per token, figuring that out is not an easy problem. I think the analogy that people put out, just because you use a bunch of tokens doesn't mean that more business value is being generated at all, right? Mm-hmm.

It's just like a 1,000-page book versus a 500-page book, necessarily doesn't mean it's a better book because it has more words in it. The same thing with tokens, right? You've got to figure out how to measure business value in this thing, which unsolved problem, and I think OTel could be very critical in helping figuring that out, because to solve that problem, you need full instrumentation of the full trace of your agent and token usage.

So one of the other things that comes up about OpenTelemetry a lot is the folks are still finding it maybe a little more difficult to deploy than they- Yeah ... would like. It's not easy.

Yeah. So what can be done about that? So there's two ways to look at this problem.

One, if we use Kubernetes as analogy, is you could go roll your own Kubernetes. Some people do. They love it.

It's great. Is it easy? No, not always.

It's why there's a bunch of vendors out there that offer GKE, EKS, OpenShift. These are all places to package Kubernetes in a more easy, either consumable way via a third-party service, or something you could roll your own via distribution. OTel is going through a similar type evolution.

You have certain distributions out there put together by vendors. AWS has a OTel distribution. I think Grafana has one, Splunk.

So there's these folks that are trying to package OTel for easier consumption, and then there's others that just offer it as a service. If you're using Grafana or Honeycomb, it's already offering an ability for you to go basically ingest OTel related metrics, and it supports OTel natively. So I don't know if that perfectly makes sense for you, but in my mind, I consider it simply you could go roll your own, or you could now there's probably 50 vendors to choose from for OTel, so it's at that kind of Kubernetes level.

Some things are best left to the professionals. Yeah. " And what is the relationship?

I just see eBPF as a way to instrument the kernel at a certain level to go produce data that should eventually be piped into OTel format. eBPF just allows you to hook into almost every syscall level area of the kernel, and you could go get data. Someone opens a socket, or someone does this with this syscallSuper useful to be able to deal and do something like that, but that data should eventually be structured into a format that OTel recognizes.

So if you think of OTel as just a structure to emit logs, traces, metrics, eBPF is kind of a... The way to look at it is like the old APM vendors, everyone had their own agent, right? What do those agents do?

They just basically collected data in some ways. However they did it. Some of the new ones actually use eBPF to collect that data.

And then that data is eventually normalized into some standardized format, in this case, OTel. In a previous world, each vendor would go store their stuff in some data store of their choosing and choice. So eBPF to me is, it's an enablement technology to instrument the kernel at a very, very deep level, that you get valuable data that eventually should be emitted as an OTel- Yeah ...

based system. " Oh, man. Okay, a funny way.

So this made me shake my head a little bit, but in maybe a way that you're not getting to where I was visiting a large financial institution recently, been around for a long time. They have three homegrown observability systems that they've built for, let's call it technology that, call it mainframe era stuff. Stuff that's existed for a while.

They have different homegrown stuff. They have some newer observability products that they use that now support OTel. They went in and basically augmented their homegrown solutions to basically emit OTel-related stuff that they were able to go now send to their third-party vendor of choice tool- Mm-hmm ...

versus maintaining their own internal things. And basically, they're building a slow on-ramp to basically get rid of maybe their homegrown solutions. So in some ways, I don't think that necessarily was the original intention of OTel to basically augment a system built for the mainframe era, but that's essentially what is being done.

It's basically allowing an institution to modernize their observability system slowly. So I think that's a very cool use case that initially was not at least thought of . In a similar vein, I think one of the things that's been surprising is you see now the networking and security people- Yeah ...

using OTel as well. Of course. Yeah.

And if all that comes together- Yeah ... and the application people are using OTel and telemetry data, and even the mainframe and legacy people- Yep ... might there come a time where we're actually all talking about the same thing at the same time?

If we're able to correlate all this stuff, that would be amazing. That's always been the holy grail is can I get data from my application that is maybe emitting logs? And then I have stuff at the kernel level, which is very difficult to get to, so I have eBPF here, and then I need something that traces all this stuff amongst the full path of a service or an application.

And just be able to analyze that in an easy, quick way. It used to be just an exercise to just stitch all that stuff together, and I think it is going to be significantly easier in the future. So, and I do think our world is going to have to depend on something like OTel with the rise of AI, which is just enabling more software, more agents that are producing more data.

We need to have very structured ways to produce all this information and also be able to analyze it in a structured way. So I think OTel will help us get there. All right.

Folks, you heard it here. OTel may not be the sexiest open source project you ever heard of, but arguably it might be the most consequential. Yeah.

Awesome. Thanks for coming by. It's a pleasure.

All right. And we'll see you guys in a minute.