Elastic Common Schema – Ken Exner, Elastic

This is texturing TV. Hey everyone, welcome back to Tech strong TV. Our next guest is Ken Exeter.

Ken is the chief product officer and elastic and I think it's his first time here on Tech strung TV. So Ken welcome to Tech strong. Thank you Alan.

It is my first time here. So thank you for having me. I pleasure to have you on you know, we usually don't Haze the first time it's too bad can of ever since they suspended us for hazing.

Dean warmer and those guys but anyway Kent why don't you if we start off with a little bit of your background your own personal, you know backgrounds history sure. So I've been in elastic as the chief product officer for I think seven eight months now, but before that I was at AWS and it was AWS for over 16 years so much from the beginning of AWS. I was there very much Z2 and had a long run there.

It was a lot of fun. So I got to see the birth of the cloud and play a role in it. So that's what number of other companies on both the engineering side and the product side dating back to like when I left school was a database programmer.

So but yeah anything how many people you know started their careers as DB people. And the internet came and they pulled me into the internet, but that's pretty much they it's pretty much a story. Yes 16 years, I guess that's about right.

I keep forgetting how far along we heard 2023, you know, and so it's been yeah, it's I you know the cold Cloud thing for me kind of burst on the scene around 2005. So that that's 18 years, so. Yes, what 16s, right that's three launched in the spring of 2006 cc2 launched in October 2006.

So it's 17 years now. But yeah, it was a lot of fun. I got to I got to learn a lot.

I got to participate in that wild ride. It was it was amazing seeing the the birth and the excitement and how it sort of transformed the industry. Absolutely it did change, you know, just a quick side note with all this craziness and excitement around Ai and GPT and Generator of AI and stuff.

I tell people, you know to me the difference is the internet changed civilization right? Think about I mean even your mom Grandparents out here. They they're using the internet to shop to communicate everything else.

the cloud it did change civilization, but it was really a technology industry. change agent right in other words Your mom or your grandma's out there, you know that point up to the sky and see. Oh, yeah, I'm getting my information from the cloud.

It's only because they saw a commercial that said that they really don't understand the cloud. Right and the it's not up in the sky. It's not one of those white puffy things where AI is another thing when people use it, it's sort of this magical to them, you know, it's like primitive men with fire, you know, they think it's sentient or whatever but it sort of one of those civilization things not just technology industry things.

Right? I think that was more indirect impact on on lay people. It was weren't using the cloud directly, but they were absolutely impacted by it.

Like when the absolutely well you used any of the modern SAS products or anything they were using the cloud indirectly absolutely just don't see it. Regenerative AI has the potential I think to be both sort of direct and indirect. I think with Scott GPT people were are seeing like conversational AI generated AI very directly and they can use it themselves and it's not just sort of an indirect relationship with the technology.

So yeah, I think cloud impacted in transform things by being sort of a change agent that impacted businesses that then impacted consumers some of the AI stuff has the ability to do both. Absolutely, but that's not what we here to talk about Ken. Forgive me for going down that rabbit hole.

Why don't we talk a little elastic? Yeah, so I joined a elastic, you know elastic is of course the company behind elasticsearch and the famous elk stack, you know, great technology for search that's been used to power search applications, you know from Netflix to Uber to all these other Technologies, but it's also a company that has been going into observability and security and sort of using the, you know, the ability to do search analytics to do log analytics and stuff and expanding broader into observerability expanding broader into security. So today, Elastic has not only a traditional search business that is also being transformed by generative Ai, and we're working on stuff there.

But it's also a complete observability platform and a complete Security platform. So we we provide an observability side not only login analytics but but tracing and APM we provide metrics we provide profiling we provide synthetic monitoring for a complete observability platform. Now the security side similarly a complete Security platform for your Knox.

So if you're wanting to do Sim more security analytics, if you want to do endpoint protection, if you want to do Cloud security, we have a complete a complete platform for doing that. So I was excited to join elastic because I saw how they had this great Foundation Technology, but they were going into these really big and growing Industries with with big problems to solve. Absolutely.

So to me, you know parking back to this conversation. We had about cloud and Ai and everything elastic to me is what is kind of one of these. Secret technologies that really empower the cloud.

Yeah, everybody's heard of AWS and it's Amazon and all of that, but the last thing is sort of a lynchpin. for a lot of the scalability performance that we see in in these cloud-based apps that we all use now, right and it really You know it it's not a secret. I think those of us in the industry are very familiar with it.

But if you're not in that Circle, you may not realize the true. breath of its impact of the technology there, but Right. co to look up information about not only elasticsearch but also our solutions for security and our solutions for observability.

And yeah just to you know, reiterate and sort of add to what you were saying elastic searches. I believe it's the most popular Java open source project of all time. It's it's like ubiquitous.

It's used in yeah replications. Not only to implement website search. But any anytime someone like anytime you want to add some kind of look up or or search capability to your application, you know, whether it's okay how to find a car an Uber or how to you know, match you to a potential, you know date and I'm in a in a match app, like any of these things are Search application.

So it's become this ubiquitous technology for helping match and look up and look for relev. Is and those kind of keeping those kind of features are like ubiquitous and tons of applications. Absolutely, no doubt about it.

All right, let's talk about a little bit about elastic common schema. Sure, ECS is they're calling it. There's been some recent news a lot of you know, well, let me back up elastic also has a very long history in the open source Community huge player in that Community supporter contributor and and you know, the this news is in line with that but can't tell us the story.

Yeah. So elastic common schema ECS is an open source specification for how to how to define a event data that's consumed by elasticsearch and other other tools. So it defines sort of a common set of field names and data types for how to create.

You know, how do you should structure your event data being like metrics or logs? So that different applications can understand it and can create a visualizations on top of it can can provide some analytics capabilities on top of it. So it's a lingua Franca Franca for how to communicate with the different tools.

And the news we have today is that we're we're merging elastic common schema with open Telemetry. We're contributing elastic common schema to open Telemetry to become the basis of the logs and event data in open Telemetry. So I think this is this is a great, you know, merger between open Telemetry and elastic common schema that's gonna allow us to take one of the most widely used schema elastic common schema for event data and make it available to open Telemetry so that we can build upon this as a community for How We Do metrics and log type data in open telemetry.

makes sense I think most of our audience is familiar with the open Telemetry project. But of course, you know it that and I'm drawing a blank. Permit Prometheus.

There. It is open to laboratory and Prometheus are probably the backbone of a majority of the observability tools. Out there in the market today.

Yeah, open Telemetry. I think it's the the second most active second. Most popular cncf project behind kubernetes widely adopted we're big supporters.

We've been investing in and supporting open Telemetry for for several years now inside of elastic and you know, when open the open Telemetry started it was a way to it was a merger I think of open tracing opencensus that defined tracing and APM data and it has since expanded beyond that to other types of other parts of observability. But the area that I think we've had this sort of the most ambiguity around has been around login event data. So being able to merge ECS, which is the most populous schema for events and log data together with open Telemetry really completes the story for open Telemetry, and we're able to move the industry towards a common schema for all observability data.

I think it's really important for the industry. Like when you have tools that talk different languages when you have proprietary schemas, it creates friction for customers. So I think this is a win for customers.

It's a win for the industry that we can stop so we can now sort of back open Telemetry as the as the common schema as the common language and we don't have to create friction between the different tools. Everyone can sort of speak the same language. If you have an open Telemetry client or collector or open Telemetry agent, you can know that it work with elastics search and work with every other tool that supports open until Elementary.

So it creates less friction so that you know that the agents and collectors will work with the different tools that that consume that data. Absolutely, and it's an important. It's an important.

Point that you make here is it really does complete an aspect of this, you know can't not everyone out here is an expert on open source licensing and the how IP is handled in open source and so forth. So when we say elastic contributed the ECS to the open Telemetry project, let's just it's kind of do a chart if you will an org chart, you've got this open telemarket Telemetry project which in and of itself is Managed and I guess owned right by the cncf cloud native Computing Foundation which in and of itself is a daughter foundation of the Linux foundation. So ultimately the Linux Foundation.

right through cncf owns and manages the open Telemetry project now that doesn't mean there aren't people from companies like in elastic or service now or you know various other companies that are part of cncf here and open Telemetry that aren't involved in the day-to-day the maintainers of the project the toc's right and and so forth that helps sort of guide the project to make sure that it's delivering what the community needs and what you know, what the market needs. so when when a company like elastic contributes in this case ECS to this open source project, they're actually transferring the ownership and day-to-day management of it too in this case the open telemetry Project but you're involved the more more than that, you know, there's like the copyright that we were. But what we're really doing it is an open source project today.

It's it's backed by elastic but it's also you know our user Community has contributed to ECS. What we're really doing is committing to work on this in open Telemetry. We are committing to make this the de facto schema that we use and that we will continue to invest in so part of part of the news here is not just that we are merging schemas and we're contributing the elastic to Common schema to open Telemetry.

We are committing to working in open Telemetry going forward and making sure that we continue to evolve this schema going forward as part of open telemetry. So and that That implies your support not just as a user of it, but as a supporter maintain their contributor. In every sense of the word, right?

You may not own the copyright or IP. You know who won't the IP and open source, but You you guys are a hundred percent behind it have been will continue to be as it's part of this bigger project now. Yeah, this this is this is our schema and we are merging it together with open Telemetry.

This is because this this is how we will you know, how our tools will work going forward. So it is it is as much about the commitment. We're making to open Telemetry to working on this there not outside of open Telemetry.

Is anything else? So I think this is good for open Telemetry and it's good for us and for our customers too because they can know that they're that we are going to be working with open Telemetry across all signal types and observability to make sure that they have an open set of clients open sort of agents that they can use and that we will fully support it. excellent so if people are already using the open Telemetry project.

Right. This is in there. Okay, using the open Telemetry project.

You're gonna benefit from having these some of the new capabilities of ECS merged in it's a very mature schema for event and log data. It's going to improve the quality of the Opera of the overall up until Elementary project. You can also know that all the clients that have been developed for elastic common schema are going to be more we're gonna be evolving to support open Telemetry and you're going to be able to have one set of clients.

So I I think it's great. Like we know why I don't want proprietary clients. I want open clients.

I want people to be able to you know, choose elastic or choose someone else. I want to win their business every day. I don't want them to be having to be, you know be locked in because of proprietary languages.

I want I want to be forced to win customers business every day and I think that's a good thing for consumers. Absolutely. Where can we get more information on on this stuff though?

Kind is should they go to the open Twitter? You can go to the open Telemetry project. There's information about it there and you'll see the latest information on ECS has merged into the open Telemetry project and can learn about it there.

Ken thanks for coming on today. Good luck. Keep up the great work.

Thank you pleasure. Alrighty, we're gonna take a break here on Tech strong. We'll be back in a moment.