Scaling OpenTelemetry in the Enterprise

Hey guys, thanks for the throw. We're here with Ryan Goins, who's head of product for BindPlane, and we're having a little chat about OpenTelemetry and the state of the community and the challenges that people are running into at the moment. Ryan, welcome to the show.

Thanks for having me. OpenTelemetry, it's been a long time coming, and a lot of progress has been made, but I think there's still work to be done. But what's your assessment right now of the overall community in terms of its maturity, and how much are we actually seeing OpenTelemetry in production environments?

Yeah. I think it really feels like this year, OpenTelemetry is kind of finally starting to have its moment. We got involved in the project back in late 2020, 2021-ish timeframe, and I remember going to KubeCon and talking to people about OpenTelemetry and people had never heard of it.

We were constantly introducing the project. And now that's kind of all we're hearing about. So I think as far as the maturity of the project, a lot of the pieces are in a really great spot.

It's being used in production environments by Fortune 100 companies all around the world. A lot of those that we're working with every single day, we're helping companies deploy hundreds of thousands of OpenTelemetry collectors in their environment. And so we're seeing a lot of success with it in those environments.

I think there's hardly any observability company out there that doesn't support OpenTelemetry at this point. But there's- Yeah ... still firms who are using more of the proprietary collectors because, well, they perceive that they're easier to deploy, even though OpenTelemetry may reduce the cost.

Are there things to be done still about just making OpenTelemetry easier to install and kind of make it a little more, shall we say, pervasive? Yeah, I think there's a lot of work to do there, and frankly, that's a lot of what we're doing with BindPlane. OpenTelemetry is a massive project.

Of course, it has the collector, and it's got standards for logs, metrics, traces, GenAI. There's OPAM for agent management. There's all of these kind of sub-projects within OpenTelemetry.

And so I think because of how big it is, how comprehensive it is, it makes it challenging to even just keep up with the current state. The behaviors of different components are changing all of the time. New capabilities are being introduced, and so that really becomes a full-time job.

I think a lot of the proprietary agents that have existed on the market, one of the nice things about them is they're more singularly focused, maybe just strictly as, say, a log shipper, which makes configuration really simple and that much easier to manage. OpenTelemetry can do a lot more. Certainly makes it more complex, and so that's something we spend a lot of time thinking about how to improve that experience.

And the other thing I do hear from folks is, on the other extreme, they're like, "Well, now that we finally got OpenTelemetry," they're kind of looking at it going, "Well, maybe this is too much of a good thing," because they're pulling too much data out on the other side of it. Yeah. They can't store it, and they can't process it all.

" So- Yeah ... is there a way to think about this maybe a little more intelligently? Yeah.

I think that has really been one of the driving forces behind the project and why it's seen such rapid adoption over the last couple of years. Obviously, the project began focused on traces. We built and donated a logging agent called Stanza, which became logging with an OpenTelemetry.

And of course, you get into the Fortune 100s, and most of their telemetry is still logs today, and that's the huge amount of volume. So where we're talking about reduction, that's largely where the problem is. But I think there were a couple of drivers.

One is the vendor lock-in problem. People wanted to really control where their data could flow to make migrations easier. So as they're trying to save costs to switch to a different platform, they have the flexibility to do that without having to redeploy a new set of proprietary agents across their 100,000 systems.

So it's accelerating migrations. But also doing things like routing to lower cost storage so you're not getting everything in your super expensive back end. And then, of course, within OpenTelemetry, there's a ton of different processing capabilities that allow you to do really sophisticated sampling and data reduction to ensure you're really getting just the most useful data to the right place.

There is, of course, a lot of talk about all things AI these days, and I can't help but wonder if we might be able to apply AI to all this telemetry data that we're collecting. But I don't know if I can use traditional LLMs as we know them today, or are we ultimately going to need something that's a little more telemetry data specific? Yeah.

I think there's maybe two approaches that are starting to work really well. So within the collector itself, you're seeing additional processors being built with more kind of ML, like smaller ML models built in that are doing, say, like anomaly detection and real-time analysis on every event passing through the collector. And so, that's kind of more of a classical machine learning solution to the problem, not really utilizing the latest with gen AI.

And that's kind of hard to do in real-time in the pipeline when you're talking about sending 1,000 terabytes of data a day through it. The approach we've started to take at BindPlane is using gen AI to build the processing capabilities. So if we're talking about the complexity of the collector, the possible ways to configure your processing so it's highly performant, it's hitting the right events and dropping the right things.

It gets really complicated really quickly. And so what we're doing is using LLMs to basically evaluate samples of your telemetry and help you build the right processors to standardize, reduce your telemetry, and then deploy those changes down to the collector to operate on the pipeline. How soon might we get to something that feels like, I don't know, maybe auto remediation or maybe something closer to it than we are today?

Because I feel like we're collecting all the data and we should be able to analyze it, and we should be able to hand it off to something that then acts on it. But I feel like maybe that's one of those things that's easier said than done. Yeah, I think we're going to start to see a lot of that this year.

We're certainly getting closer to it on the BindPlane side. One of the pieces of OpenTelemetry that we co-developed and spent a lot of time on is something called Opamp, which is the agent management protocol. And, so that of course, has the collector as a client, and then there's an Opamp server that's managing the fleet.

And I think that server piece plays a critical role here because what we can start to do is have the backend platforms communicate back to the Opamp server. We're starting to do that via MCP, and then make changes down to the collectors. And so maybe you have an incident and you actually want to get rid of all of your sampling rules and just kind of open the floodgates for the next 30 minutes and get all of that data while you dig in.

And so that's something where the observability platform can then communicate back to the Opamp server. Hey, pause all of these processing rules just for 30 minutes and give me everything. And so that kind of full circle loop, we're starting to spend a lot of time on, and I think we're going to start to see that become a reality this year.

What happens to monitoring as we once knew it? Because it was kind of like this notion where we were going to just track a set of predefined metrics, but now we've got the metrics, and we've got the logs, and we've got the traces, and we use that to investigate particular issues. But does one obviate the need for the other?

I mean, is essentially observability a superset of monitoring, and maybe I can start getting rid of my monitoring tools because I got to figure out how to pay for all this observability stuff. Yeah. I think what we basically start to do is get kind of closer to resolution.

And so, you're spending less time on getting the right data, getting it structured properly, getting the right dashboards built, and I think as long as you kind of have all of the data going into a central store, then the problems are going to be recognized sooner and surfaced by AI. And then I think the solutioning is going to start happening with AI agents as well. So how do these things evolve?

I think they basically start to merge toward getting to resolution very quickly with a lot less effort. So what do you see people doing today when they first get started with OpenTelemetry that just makes you shake your head a little bit and go, "Folks, maybe we need to think about this differently"? Yeah.

I don't know that I really ever kind of am shaking my head. I think I have a lot of empathy as people get started with it because of the complexity. So, a scenario that happens to us really frequently is folks are deploying OpenTelemetry on their own.

They've got kind of this organizational directive maybe to transition the organization to OpenTelemetry. So they're on the open-source standard. And they get a couple of collectors deployed, and things are working nicely, then they start to get to 100, and then they're trying to create organizational governance policies across the entire org, and standards, and the complexity really starts to grow exponentially.

And so, I think I have a lot of empathy for users as they're trying to figure this stuff out. BecauseThere's kind of endless different ways you can configure your pipeline and get all of these things set up, and they all have a bunch of different trade-offs and performance implications and these types of things. " They see that we're a huge contributor to the project and contributed the logging agent and work on agent management and those pieces, and they'll reach out.

And so it can be a lot to wrap your head around, but I think if you get your first collector deployed and get going with it, it's something you can pick up pretty quickly. Well, what is your sense of just how open is OpenTelemetry? And I'm asking the question because part of the motivation was people were locked into various APM platforms back in the day and observability platforms.

And is it easier to swap them out if I have OpenTelemetry as a standard collector, or do all these things have little extensions to them that make it hard to actually do that? Yeah, I think it definitely should be, and it can be. One of the things that maybe folks are sorting through right now is within OpenTelemetry, there's a distro of the collector called Contrib.

A lot of folks are deploying that in production. It just has every component under the sun included in it. And that's largely just because they're not sure which distribution of the collector they should be using.

So sometimes a vendor will have their own distribution. Now it certainly depends on the vendor, but some of these aren't really helping the lock-in problem because they only include the exporters for that vendor, or maybe that vendor plus one or two others, and they're not including the exporters of their competitors. And so if you deploy all of these, sure, it's open source, it is OpenTelemetry, but you're going to need to switch out your collector if you want to switch to another vendor that supports OpenTelemetry because they don't have that exporter included.

And so even BindPlane, we have our own distribution. We're actually frankly trying to push people toward building their own distribution of the collector. So we built an open source tool called the OTel Distro Builder that lets you do that.

You pick the components you want, it manages the whole build process, and has some GitHub actions and things like that that are included to make this really easy because we think basically, certainly if you're a larger company, you should be building your own distribution. And if you want total neutrality, you're going to need to control that distribution. But there's overhead there, and so we're trying to make that really easy for folks.

So what's next for the community? What are you guys working on or what's at the top of the priority list? Yeah, there is so much going on and I would say we're kind of involved in just a few pieces of it.

The things we're really focused on are Opamp, which is the agent management protocol, and continuing to make that more robust and more mature. There's just a ton of new capabilities we're excited about there. OpenTelemetry has a supervisor component that we're heavily involved in that can manage the collector itself.

And so then an Opamp server can connect to and talk to that supervisor, and then the supervisor can communicate with the collector. This is a really, really nice architecture, especially when you're deploying to IoT devices, where you just want to maintain this persistent connection, but you don't necessarily need a collector running all the time. I think those are really the two big pieces.

And then I guess the third is in the security world, there's an open source project called OCSF, and this has become an open source standard schema for security data. And one of the things we're working on is bringing some of that into the OpenTelemetry project so security teams increasingly can use OpenTelemetry to collect their security events, and send those to their SIEM. So ultimately, how broad will OpenTelemetry be used?

And I ask the question because it clearly got started out of the DevOps world for applications and infrastructure, but I'm starting to see it in not just security to your point, but I also see the AI community is now starting to look to it as a mechanism for pulling in telemetry data. Even the networking people are at it. So will this- Yeah ...

ultimately become kind of the cross-discipline standard? I think it's certainly on that trajectory. Maybe it was a week or two ago, Anthropic just released some guides on using OpenTelemetry to monitor Claude code and your Anthropic APIs.

Google, of course, is all in on OpenTelemetry, and that includes Vertex and Gemini monitoring. And so, yeah to your point, we're seeing just enormous adoption. Just ourselves, we're working with 18 of the Fortune 100.

There's a lot of the Fortune 100 I know using OpenTelemetry that we're not working with, and so theThe acceleration over the last 12 months of adoption has surpassed all my expectations. It's really nice to see the standard forming and yeah. All right.

Well, open source's communities are by definition democratic, but if you were appointed king for a day, what's that one thing you would do? I think this is something that you just kind of like developing any product, and it maybe happens a bit slower with an open source, but you learn as things get battle tested and new and unique use cases and environments, and I think a lot of the open telemetry deployments have been focused on tracing. And on our side, we've seen a lot of the log deployments and certainly at scale, where we're talking maybe 100, 200 terabytes a day, sometimes more of throughput.

And the thing we're spending a lot of time on right now is different components in the collector getting pressure tested at those sorts of volumes and that scale, where there's a processor that's not as performant as it needs to be. There's different things that just they get pressure tested and we got to go fix. And so I think I would just try and magically get those to all be resolved.

But it feels like we're getting closer and closer every day. All right. Well, folks, you heard it here.

Open telemetry is well on its way to being everywhere. We just have to figure out how to manage it from here. Ryan, thanks for being on the show.

Yeah. Thanks, Mike. All right.

And back to you guys in the studio.