Enhancing Kubernetes Monitoring with Grafana Labs’ Tom Wilkie

This is Textron tv. Hey everyone. Welcome back here to Tech Drunk tv.

I'm happy to be joined by Tom Wilke. com or Cloud native now, or digital CXO. But, um, I'm assuming most of you know Grafana Labs there, the people behind some of the biggest open source projects out there, as well as, you know, a leader in observability and many new things.

And, and speaking of new things, we're gonna talk about some new Grafana Labs things here. But first, let's welcome Tom. Welcome you welcome here to, uh, tech Trunk tv.

Hey, thanks for having me, Alan. Yeah, nice to, uh, nice to be here. So I don't think I've had you on my segments before, so No, I don't think so.

Yeah, no, I, I think we've had Raj in the past and some others, but not you. So I don't know why we've missed up to this point, but thanks, thanks for joining us. Now you are CTO Tom, why don't you, if you don't mind, share a little bit of your journey with, with our audience.

Yeah, Of course. Yeah. I, I, um, my background's actually coming from the Prometheus community, so, um, I've been a maintainer on the Prometheus project for best part of eight years now, I think.

Um, mm-Hmm. I started a piece of software called Cortex is a, this is another, um, open source CNCF project, um, to build like a horizontally scalable version of Prometheus. And, uh, started a company around that.

And actually Grafana Labs bought my company six and a half years ago, back when Grafana was like 25 people, and my company was two, um, mm-Hmm. So, like, you know, it sounds much brander than it actually was. Um, and I joined mostly as an individual contributor building cortex into like Grafana Cloud's hosted Prometheus offering.

Um, I also, uh, built the Loki project, our log aggregation system, built the first few versions of that, and then over time started hiring a team, became like effectively the leader for our databases department. Um, I remember when the founders of Grafana Labs kind of tapped me on the shoulder and said like, Hey, can you start the Grafana Cloud team and turn Grafana Cloud into the product that it is now? Uh, and then, yeah, like about nine months ago, um, got the same tap saying like, could you, could you take on the CTO responsibilities and, um, and step up and lead all of all of r and d for the company?

Love it. What a great story. You know, uh, regular Horatio Alger, hard work and perseverance pays off, right?

And, and smarts and Luck as well. You know what I always say, sometimes it's better to be lucky and smart. It's good to be both.

Um, hey, Tom, you mentioned a whole bunch of what makes Grafana Grafana as part of your story, right? One of the, one of the big folks behind Prometheus, which is, I don't know, it's probably still top five or top 10 in terms of open source projects, right? It it is A-C-N-C-F project, Right?

Yeah. Second one after Es, you know, you look at Prometheus and, and, uh, open Telemetry, and that really is the backbone of so much of the observability industry today. Um, you know, across multiple, multiple vendors.

And then you mentioned Loki log aggregation, another great open source project that the folks at Grafana, um, you know, brought, brought, brought out. And then the Grafana Cloud project, though it's not really an open source project. Grafana Cloud is the product of Grafana that incorporates a lot of these open source project, you know, feeds into a enterprise level commercial hosted offering.

Exactly. Yeah. Yeah, I got that right.

Yeah. No, more or less, I mean, the company Grafana, I think, you know, I don't think people realize how long Grafana Labs the company behind Grafana, right? Has been around, right?

We've been doing this for over 10 years now. Um, you know, we spent the first part of our life, right, really focused just on the Grafana project. Um, you know, and we had a, um, you know, an enterprise version, a self-managed version of Grafana that we used to sell, or we still sell.

Uh, but yeah, like six or so years ago, we got into the Prometheus and database space, and we built Loki and what became MiiR Cortex became MiiR and, and eventually tempo our distributed tracing system as well. And in hindsight, we really think about that as kind of the second act for the company, right? Um, we built, you know, we built Grafana Cloud, which is the collection of our open source projects, you know, hosted run for you as a SaaS platform.

Um, but I think what's kind of more, more exciting is recently, like our projects have become or started appealing to the mainstream, right? And we used to be very, you know, very popular with influencer, not influencers, sorry, innovators and like early adopters, right? Um, but now like a hundred percent gone mainstream and our products have had to change as part of that, right?

So the last two or three years at Grafana Labs, we've been focusing on building like solutions on top of, we call it LGTM, like Loki, Grafana, tempo, and MiiR, right? Building solutions on top of that, right? So like we've built a, uh, an infrastructure observability solution, which realistically like sounds fancy, but it's really just a set of curated dashboards and alerts for things like MySQL and Docker and Kubernetes, you know?

Mm-Hmm. We've built, um, an IRM solution, Grafana Oncall, like, which is really a, you know, helps you make sure your alerts get to a human, get on the right device at the right time, you know, scheduling, escalations, rotations, things like that. Think a competitor to PagerDuty.

Sure. That's what I was thinking when you said it. And then, uh, you know, we, uh, a company called K six joined Grafana Labs a couple of years ago, and they have one of the most popular open source load testing tools, which is another kind of pillar, uh, solution in Grafana Cloud for like load testing at massive scale, and then storing all of the results back into Loki and MiiR and Tempo.

And then finally, one of the big announcements we had, um, about three weeks ago was the launch of an application observability solution. And you mentioned open telemetry, like our application observability solution is open telemetry, native uses open telemetry for auto instrumentation, uh, also uses a lot of eeb PF technology for auto instrumenting, like go and rust and c and packages us all up in a kind of really easy to use, easy to get started with a solution for application observability. Um, so yeah, our mission right now is to bring the really cool technology in the LGTM stack to a wider audience through these kind of more curated, like opinionated solutions, uh, and make it like super easy to get started with the tech.

You know, as we sit here, Tom, I'm listening to you and, and so I have a very interesting seat where I sit, right? I get to talk to all you guys and, and I'm listening to you describe the journey of Grafana Labs and, and some ways this is the journey of, of today's software stack, right? A lot of and, and keep, I look, I've been in tech 30 something years, right?

I've seen cycles. Um, you know, a lot of times we develop software and, you know, and in classic crossing the chasm, right? First the cool kids like it, that 15% of the early adopters, and you aren't the cool kids to like it because a, the cool kids sometimes they're cool kids, they're influencers, they're leaders.

They, they make a lot of noise for just 15% of the market. And, and so, you know, it's important to get the cool kids to, to use your stuff, but at some point you want to cross that chasm now to the mainstream. And, you know, and the mainstream's not homogenous.

It's, it's segmented, right? There's earlier versus laggards versus later mainstream, earlier mainstream. But regardless of that, to get to that mainstream, generally a product has to mature.

A company has to mature. You, you, you, you start moving from, you know, the Rell guy with the funny hat or the purple hair to the bag carrying suit wear. Well, not a suit, but you know, a briefcase carrying salesperson.

And, and you're not just selling necessarily self-serve, though self-serve is the big model today in the SaaS world, but you, you need more support. Your customers are demanding more support. They want a throat to choke or a, you know, a butt to kick at three in the morning if something's not working.

And, um, and so you as a company have to mature and, and offer that. The thing that I think is really unique about Grafana though, is a lot of companies, it's all they can do to do that for one product, right? It's a whole nother thing here where you just rattled off maybe six products or more and, and, you know, that's a feat right now, you know, you know, a million here, a millionaire before you know it, it's real money.

Uh, that, that's a lot, lot, lot to do. I guess my question to you is, at what point do you pull this all together and say, look, this is the Grafana platform and these are the modules that play on this. We're yeah.

Unified platform, And just to your, to your like, uh, companies having to grow up with like, for sure. And we do, you know, we have a professional services team and a customer experience team and you know, a sales team. But like, one of the things I'm very proud of is like, we, I don't think we've lost the scrappy startup culture in, in r and d, right?

We still have a lot of small, independent kind of empowered and autonomous engineering teams building cool stuff. Like most of the projects I rattled off, they came from our internal hackathons, and they, these weren't built as a response to customer demand. We were scratching our own niche.

Uh, we were solving problems that we had that we saw, and then we're like, oh, wow, customers actually want this as well, and, and turned them into products. So yeah, we, we, we've done a lot of growing up as an organization, but I don't think we've, well, I hope, but I really hope we've not lost that spark that makes it kind of exciting and fun and authentic. But no, to your point, yeah.

At what point does this come together to a coherent platform? Mean the observability space is, is a very broad space, right? And, and often like the, the capabilities that make up a platform, it's not really dictated by us, it's dictated by our competitors, by the market, right?

And so I would say like for many years we were playing catch up on those capabilities. Um, we, one of the things I'm quite flattered by is we've started to see some of our competitors copy some of our capabilities, and I think we're starting to reach more of that kind of inflection point where we have like a really good, um, working set of capabilities, inana cloud, that are coming together into something that's a lot more coherent. Like we've got a lot of growing to do and a lot of, you know, a lot of integration into the product to do, to get there.

Um, but like, I think, I think we're at that point now, and I love that you picked up on the crossing the chasm reference. I didn't say it because I don't like to, you know, I don't like to be that person saying we're crossing the chasm, right? It sounds a bit too self-serving, but internally that is a hundred percent how we, uh, frame that, the growth.

Absolutely. I Well, I'm glad I was able to play shill for you then. Um, but, but agreed.

So let you know, you mentioned some new stuff coming out. Just came out about three weeks ago. Uh, there, there was a lot of launches right around major new infrastructure.

Course monitoring is another capability kind of being rolled out here. Share, if you don't mind with the audience a little bit. You know, we're just coming off the back of conference season, you know, off the back of and reinvent.

Um, we had our own conference Observability Con in London, um, and we, we, yeah, you know, we, we try and avoid doing the whole conference driven development, but it's very tempting. So with massive series of, uh, of announcements, right? So our Kubernetes app, which is this experience we've built into Grafana Cloud for helping app app developers understand how their software runs in Kubernetes, uh, that got added kind of cost monitoring capabilities, like, you know, baby finops capabilities, um, mm-hmm.

Which we're, you know, very useful for developers to understand the financial implications of changes they make, right? Um, we, we started previewing, um, the third version of Loki, right? Focused on some pretty major, um, performance improvements and a real kind of loki's five years old now.

Like, and we started to understand, you know, I think we got the developer experience with Loki, right? But we've started to learn there are other use cases for log aggregation systems. 0 will be making a particular class of queries, we call them these kind of needling a haystack queries.

Uh, we're making them like thousands of times faster with a set of really cool technologies. We're using bloom filters. Everyone, uh, everyone's excited when you find a useful bloom filters.

Um, the, uh, the other thing we are previewing is, um, what was I, as I said, we, uh, launched our application observability product, um, you know, based on Open telemetry and EDPF, we also, uh, announced the acquisition of a company called Asserts and Previewed their technology and asserts is super cool. It builds this entity relationship graph of your infrastructure based on your metrics and other telemetry signals, and then has some really cool experiences that leverage that graph to like accelerate root cause analysis or help navigate within your Grafana instance or help query that data. So it's something we're incredibly excited about.

Um, Manoj the founder joined us on stage and kind of gave a live demo of a search to the audience. It was incredibly exciting and really excited to have him join the team as well. And then the third big, um, set of announcements, fourth I'll lose count, uh, was all of our kind of large language model, you know, gen AI features, um, in Grafana Cloud.

So we're taking a bit of a different approach. You know, we're not rebranding as an AIOps platform or anything like that. We're just adding thoughtful kind of reduction of toils features to the product that help engineers do things like automatically summarize incidents or natural language query for Prometheus or, um, we have this cool project called Dash GPT, which uses, um, OpenAI to help you build dashboards, you know, all of these kind of different capabilities baked in across the platform.

Um, you know, yeah. Like to help not to kind of be too flashy. I get it.

I get it. Um, sounds like nothing's going on over there. We've Been busy.

We've been busy. Yeah, we've grown like crazy. I hear you Tom.

We we're running a little low on time, but you know, we, we've covered a lot of ground here and I, there might be people in the audience there, head spinning, right? Where can they go track some of this stuff down and, and dive deeper? com.

Um, you know, our website, honestly, if any of this has sounded interesting, I'd encourage you to go and watch the keynote on YouTube, um, from OSCON in London, right? So that's got Raj, myself, Manoj, Merl, who am I missing? Like a bunch of the team on stage explaining this in a lot more depth.

And all of the session content from the con uh, from the conferences is online. So you can go and, uh, you can go and get a lot more depth there and, you know, join our, join our community Slack or reach out to us on Twitter, where we'll come to one of our many kind of meetups. We organize them, you know, seemingly there's multiple Grafana events going on in disparate cities around the world every week now.

So yeah. You know, get in touch and reach out if any of this is interesting. I love it.

And by the way, in case you're wondering, it's Grafana, G-R-A-F-A-N-A, no, double F. Sometimes I do the double F myself or P Right? Sometimes.

Yep. Or pH some people do that. com.

Check it out. Hey Tom, I want to thank you for coming on today and sharing with, uh, with us so much going on, continued success with Grafana Labs and, uh, enjoy your holiday and we'll to Thank you Very much. New Year.

Alrighty. Tom Wilkee, CTO Grafana Labs here on, uh, tech Drunk tv. We're gonna take a break.

We'll be back.