OpenTelemetry Project – Perk Stozek, Sumo Logic

This is texturing TV. Hey guys. Thanks for the throw.

We're here with perkstozik who is engineering manager for open source collection at Sumo logic, and we're talking about all things related to open Telemetry perk. Welcome the show. Hey Mike.

Nice to meet you. I think people have it in their heads that somehow or other open Telemetry is one. Uber thing when it's actually a bunch of different things that all are being developed under the common framework, so can you walk people through a little bit of what exactly is in open Telemetry as it applies to you know, laws and traces and how does the whole thing manifest itself?

So open Telemetry is a big project actually and it consists of a couple of things so it's not going to be on easy answer but it's not that complicated either. So there are to me personally there are like three biggest things here. First of all, there is specification for all the things about the open Telemetry that is like that those are guidelines for the apis for the sdks.

They're like semantic conventions all the important stuff about how do you want to model the data? How do we want to how do we want to model? The API?

Then there are those sdks us implemented into the languages into the Frameworks so that you can actually use those MP apis from from within your language Java or or scholar Ruby and then there's this third part which is a collector itself. So not only you can create the data using the instrumentation libraries this data. The same specification you can also collect the data processes and processes and send them to the vendor of your choice.

Now you guys have your own distribution of that so is or is that how we're going to see this manifested? There will be multiple distributions or how does this come together in a way that's consumable for the average devops team? I believe that many vendors will provide their own distributions and the important fact about them is that those distributions are not Forks in a way that when you have when you have a Linux distribution, that doesn't mean that you provide your own Linux kernel, sometimes you just do some patches but the distribution itself is like you package package many components and you say hey I am going to support those components in those versions.

So actually this is the same thing that happens here as a Sumo. We what we do is we use the community version of the components. We add some of some components that we actually create it and then we package that and say to our customers that hey this is something that we as soon more support and customers should they have any questions and a problems.

We are there to take and help with that. I believe that this will be for many vendors because It's not like you can you can support all of the components that are out there in the in the option Community because there's a lot of there and right now, you know, we can support our vendor-specific exporter for example, but we cannot say that we can fully support exporters for the other vendors. So that's why we just put pick some components and saying that hey guys, this is our area.

They're also seems to be some confusion as to what degree all of this is ready for Prime Time. Some folks will say the agents are still fairly large compared to proprietary agents others will say it's harder to install than proprietary agents. So what is your sense of?

Where are we on this journey and how far do we have to go? I would say that it is pretty fair to say that. It is harder to install than other agents.

For example, because right now this is this is mostly put somewhere on the GitHub as our RPM depth or just a single binary to for you to download but I believe that we will be here. We'll be there at some point. So it shouldn't be that hard actually at some point because when when the adoption it's it gets gets bigger.

Obviously the it will be easier to install this stuff. But then when it comes to the Being able being production-ready. So to speak that is about that is fair as well.

Like open Telemetry says that traces are metrics are actually stable and logs are not and that means that the API for the metrics and traces are stable for locks is not but what we found that's Sumo when we use our own destroy is that we started quite recently took food in the this as much as possible. And right now I can tell you that we have we have mixed mixed laws. We use both kubernetes and non-coverities and for the kubernetes workloads, when we dog food our own stuff our own kubernetes collection Machinery.

We use open Telemetry at full and we don't see that many problems. There are some from time to time and we are able to fix that fix them quite fast. We have a very good connection with the option community.

So when Find a problem we just fixed it and and it's actually delivered. And it is working. I must say it is working and for us it's working brilliantly.

What is your sense of how pervasive will we start instrumenting applications? If I look back in time apms were not widely used because it costs money to buy the agent install it into the application. Somebody had actually go do that and it became something you used.

Yeah, unlimited basis. Are we on the cusp of seeing pervasive instrumentation of applications and how long might that take? There is a couple of projects in the area.

For example, there's a lot of preserve around the outer instrumentation. For example on kubernetes. You just install your instrumentation operator for the open Telemetry and it is able to inject the instrumentation libraries interior projects, but it's not there for every language.

So there is some stuff to be done and this this is like ongoing Journey. Obviously you can manually instrument your applications. But as I said, this this makes a hard time for for some customers, it's not easy and especially just don't have control over the application that you're on because maybe you just bought it from somewhere.

You have no idea. What's the source code and you don't have access to the source. So it's called in the first place.

There is there's one very interesting project around that very that is trying to tap into the ebpf so that Be you don't need to instrument your application. You will get all of the information from the layer that is below that from the kernel itself. But then there's this question how how good this data will be for you because this is like a very very technical instrumentation.

I would say just like function calls. You're not always interested in in this kind of an instrumentation. It will be rather more interesting to you.

If you get the business domain instrumentation, like if you get information, like let's say you use the Ruby on Race framework and you have instrumentation created this but specifically for that then you will get information about all of the calls that make sense in the Ruby on Rails domain and not about every function call that is actually being made out there. As we go along it seems to me at least that A lot's writing on all this because we're talking about observability, but we can't really have observability unless we really instrument everything and we can have it to a degree and even further down the road. All these AI Ops platforms are dependent upon being able to collect data from everywhere and so is it your sense that maybe we're a little over our skis they say when it comes to observability and AI Ops simply because we have that instrument and everything to the end degree yet.

And yeah, this is going to be an extended Journey. I believe it will be yes at this point. But then there is this promise for the operator and the vendor support is actually very very good here.

And and the community is very growing and then if you see like what is the what are the projects that are most popular from the cloud native Computing Foundation then open Telemetry is the second just after the kubernetes. So there is a lot of stuff going on here and I am pretty sure that it will be up to speed. I mean it is right now up to speed but but it's not there yet.

I think that it will be there in not that long time. Right, once your best advice first on how should I go about using open Telemetry? Where should I start?

You know, there are small projects I should start with today or they're particular use cases. Where are you seeing the most interest or success? If you'd like to start with the open Telemetry and just you just have your project and don't know what to do.

I I would Advocate just go to the open Telemetry website first because the documentation there is amazing and there are examples there is a lot of like a tons of information and it is actually a useful information. So so that is awesome. And then that depends what is what is your use case whether you want to create the data in the first place whether you Your application already creates logs metrics maybe traces and that you just you just want to take take this data and just push them to the vendor maybe process them in the means in the in the meantime because that is what actually collector is doing.

I think that this is a very very big strength of it is that not only it can collect and Export it but it can all also process it in the in the same way whether there's a logs metrics or traces. There's a lot of processors that takes care of full for all of three of them. So you don't have to have like different configuration for those and but maybe maybe you want to do it differently.

Maybe you just want to use a library to create the data in the first place and keep because you can do that you can skip the collector better. You can just send directly from the library through the OTP protocol which is a native open telemetrical to send the Telemetry data. Hmm all three signals logs metrics and traces then I would advise as well to go to documentation.

See what is the caracal current status for your language and there that their information how to top it into a project how to use that. And so on. What about folks that one either contribute to the project there?

Help support it in some ways. How do they get involved? Oh, there's a there's tons of meetings around that and as I said, there's the Defenders support is is amazing but also people involved in that Community are passionate about at work.

So there's a lot of stuff going on and if you're interested in contributing that is awesome, then you can join the slack which which is hosted by this slack thread, which is hosted by the cncf actually a couple of slack threads or you can go to the GitHub. You can join the Sig meetings that are that are in the special interest groups meetings every every day. There's like four or five different seek meetings for different topics that is like for for collector is one for locks one for the specification is another you just pick your topic topic of your interest and you can join the seek and start contributing there.

You can at the very least go to the GitHub and see where you would like to help and see, what are the problems there are issues which are which have this like good first issue label and you can just pick them start working on them. You can start with reviewing stuff or just start with reviewing the documentation. Maybe the documentation is not good.

So when you read and you see that something is awkward or maybe not explained enough. You can just fix it Community is it's very Community likes contributions. So to speak and there is a lot of contributions.

So, you know also the there's no problem with contributing stuff. I think right now within the community. I've been to the open telemetric community that they a month ago and I've heard from the maintainers that the biggest problem actually for them is that there's it's not like they don't have contributions.

Is there other that they don't have time to review all review all of this stuff because there's so many contributions. So if you think like maybe you're not so good with implementing and with the code, but you can review stuff then reviewing PR for reviewing issues is a very very good place to start everything. And do you think ultimately we've been talking about observability as a core tenant of devops forever in a day, but for the most part that's been kind of continuous monitoring at best.

Do you think we're on the cusp of some new era here in terms of devops and observability and that we may start to be able to do things. We never imagined. I would say yes, you know and the reason for that is that and this is very cool about the open telematic projects in the first place.

Is that because because it started with the specification and you have you have the same specifications for logs for metrics from traces and it goes through through the from the instrumentation through the collector to the vendor at some point you probably will have this data in the same schema in a multiple vendors, for example, so there will be this opportunity for example for vendors to communicate with each other. Maybe one vendor is good with one thing another good with another thing. So maybe you can mix them together.

Also, this is this is actually easier for vendors. I would say because right now, you know, I would say that everybody was definitely inventing the wheel for themselves and then you have different schema here different schema there. Everybody was working.

On the on the same stuff right now, you could actually adopt open Telemetry and don't reinventy wheel and just focus on the stuff that matters for customers, which is I want to just observe my application so not focus on the technical things, but then Focus rather on how how to use open Telemetry to make use your user successful so that they can they can see whether they applications and their infrastructure is working or not. Yeah, I would agree with that. Yeah.

All right. Awesome. Hey perk thanks for sharing your knowledge and insights.

That was awesome. Thank you, Mike. Thanks for letting me steer here.

All right back to you guys in the studio.