Splunk Finds Most Teams Cannot Control AI Costs

Hi, everyone. " My next guest is Greg Leffler. Greg is the Director of Developer Evangelism at Splunk, and he's joining us today to talk about a big highlight of the year in the observability world is the Splunk State of Observability.

This is the 2026 edition. But first, let's welcome Greg. Greg, great to see you, man.

How you been? Thanks, all. I've been great.

I was really looking forward to KubeCon and to re:Invent. It's a busy season, but really glad to be here. It's going to be busy.

I was just telling you in the green room before we came on, I've been on the road pretty much since Splunk Conf. Yeah. Constantly.

I've gotten a little bit of time off, though. Did you, good for you. No, look, I didn't.

I've just been on the road. And then, like you said, next week we've got KubeCon, then I think we're home, maybe for Thanksgiving, right? Mm-hmm, yeah.

And then boom, re:Invent, I'm flying out that Sunday of Thanksgiving week. Yeah, me too. It comes with the territory, right?

Yeah. First world problems, Greg. First world problems, right?

Yeah, very much. So Greg, you're the Director of Developer Evangelism at Splunk, but why don't you give people a sense kind of your journey, how you came to be the Director of Developer Evangelism here? Yeah, it's a long story.

I'll give you the abridged version. I started off my tech career in the NOC at eBay Advertising. And, for those of you people that don't know what NOCs are, back in the day, there used to be people that babysat computers, and when something went wrong, they would try to fix it, and then they would call a real engineer that would come fix it.

So, that's how I started. It was basically like triaging alerts and manually figuring out what the likely root cause was, and who the right person to call was, and getting them on board. So, that's how I started.

I moved while I was at eBay. I became a systems admin, and then I did a little bit of security work for them as well. Then I went to LinkedIn as one of their early SREs and rose through the ranks pretty fast at LinkedIn.

I left LinkedIn as a Senior Manager of SRE. Well, I left engineering at LinkedIn as a Senior Manager of SRE. And then I was actually on LinkedIn's editorial team for about a year, creating videos and writing articles, and trying to make LinkedIn a content destination for software engineers.

And so after I did that for a year, I took a little bit of time off and then came to Splunk as a Product Marketing Director, and then the role kind of evolved from product marketing to evangelism. And using my experience and helping people, I can talk credibly about observability because I used it for a decade. I wrote a bunch of observability tooling.

I ran really large scale stuff. So observability is both what I get paid to do, but it's also something I'm really passionate about personally. I really care about observability and have the background to know what it's like to be up at 3:00 in the morning, on Thanksgiving or on Christmas, trying to fix something that's gone wrong.

And so yeah, now I talk to the media, I write, I go to events and talk at breakout sessions and do workshops, and anything that can help people understand more that Splunk is in observability, that we have an observability product, it's not just security. And that observability has to evolve over time, which is one of the things that we talk about in the report, but it's also just true. Every year that I was working, there were completely new things that I had never done before that I suddenly had to start doing, and that's really exciting to me and that has really kept me interested in the observability space, with the benefit now of not having to carry a pager.

So, it's a win-win for me, for sure. There is something to be said for not carrying a pager. You realize our kids are probably not going to know what a pager is.

Yeah, I should have explained that as well. Yeah. " Yeah.

And you didn't know what you were walking into half the time, but yeah, funny. That was before the BlackBerry, something else people have never heard of, right? Yeah, exactly.

But Greg, I wanted to talk a little bit about, before we jump into the observability report, about the relationship of observability to security. And this is so core to the Splunk mission, isn't it, right? Mm-hmm.

Of course, I've known Splunk a long time, before they were part of Cisco, obviously, and longer than that. But a lot of people were, "Is Splunk a security tool? Is Splunk a security solution?

" Well, all of the above. Mm-hmm. But does it inherit, and more so now than ever, there's an inherent relationship between observability and security- Mm-hmm ...

between observability and DevOps, and AI, how it's playing in that. Before we jump into the State of Observability Report, Greg, talk about that inherent relationship or those inherent relationships. Yeah.

I think it's something Splunk has been trying to say the whole time I've worked here at least, which is five and a half years, is that these teams are coming together, the data's coming together, the workload's coming together. And I think finally, AI has been the catalyst to make that actually true, right? Splunk has wanted it to be true for a long time, right?

We've wanted people all working in the same tool and using the same data, but there's organizational problems. " There's been a lot of friction historically, but I think now that AI is helping level that playing field, and everybody is starting to play at the same time, and the security folks and the observability folks are working together. And I think Cisco's vision was forward-looking when they did the acquisition of Splunk, in my opinion, which was, "Hey, this is going to be one thing.

" And so you brought up at the beginning, is Splunk a security tool? Is it an observability tool? Yes, right?

It's both. It's everything, which is the upside and the downside of Splunk is it can do whatever you want. And so figuring out how to get the most value out of it is something that we keep working on, and we keep getting people to understand.

And I think the integration that Cisco's providing with Cisco Cloud Control and bringing all of your stuff into one place, and you can do all your troubleshooting in one environment. You don't have to learn different UIs. You don't have to learn about how to access things, how to write searches.

It's all handled for you. And I think that is really what's bringing it together, and I think we're really on the cusp of it being to the point where when something happens, you can more confidently say, "This has a security root cause," or, "This has an application problem root cause," right? And that's kind of been the Holy Grail for a while, I think, is when a problem comes up, who are the right specialists to help you fix it?

And that's often the hardest part when the problem comes up, right? It's like if you have a huge spike in login activity, right? Is that because there's a credential stuffing attack going on on your site, or is that because somebody mentioned you on social media, or marketing ran a commercial and didn't tell anybody, and now there's a huge spike of activity, right?

So figuring out who to engage is now a lot faster because you all have common tools. You're all in the same place. The data's all in the same place.

You can ask faster and find out the answer faster. So I personally have been a skeptic about that for a while, but I do think the AI has been the catalyst that's making a change. " And then it can pull in and run all the tool calls to find out across security, across observability.

Even if the data appears siloed, one of the benefits of the LLM is it can go in and look at all those silos and then synthesize it for you. And it can do it a lot faster than humans can. So I think the troubleshooting, the integration there is really going to be meaningful, and it's really going to change how people do troubleshooting.

Excellent. All right. Hey, I'd love to even dig into that more with you, but today we've got to save a little time.

You guys did come out with your State of Observability Report, kind of one of the foundational cornerstone reports in the space. Has been. Give us the scoop.

What's the deal this year, Greg? Well, as you might guess, this year we're heavily focused on AI. But every year we ask a pretty big audience, this year it was a little over 2,000 people, that are either practitioners or executives that work in IT ops and engineering.

So people that use observability every day, that rely on it both to solve problems and to figure out how to run the business. And last year it was the biggest report of its kind, so we'll say it is this year, too. We don't know for sure, the year's not over, but so far I think we're doing pretty well.

And we really want to try to figure out what are observability people actually experiencing on the ground, and then what makes a team a good team, right? So, the main theme of this year's report is that observability is trying to restore predictability to the world, right? And not that operating applications has ever really been predictable, but it's been more predictable, I think, than it is now, right?

One of the main benefits of AI is that it is not deterministic. You can give it the same prompt five times and get five different answers, and that can be helpful. " Well, you can't necessarily say, "Oh, we're going to change the prompt to say this one thing," because you might still get a different answer every time, right?

The way that AI makes you operate your services is inherently unpredictable, and so how do you fix that, right? And we tried to ask, and we got into some data about how do you make sure that if you're running applications with AI, which almost everybody is already, by the way, it's one of the other findings in the report is everybody's jumped on this really quickly. But how do you make sure that those applications behave predictably?

How do you tie their cost to business impact? How do you make sure that you're not just lighting money on fire, and that you can say, like, "Hey, this money we're spending has a revenue impact or customer satisfaction impact"? And then it's really how do you trust that you are able to deliver software with confidence?

And when you deploy a release, when you deploy a new LLM, when you change the prompts, when you do anything like that, how do you feel safe, and feel good that you can go to sleep at night and not worry about the pager going off at 3:00 in the morning because something bad has happened, right? I think all of these things are things we found in the report. And the last thing I'll say about it before we go into more detail is there's new operational demands about AI, right?

Nobody yet, well, maybe not nobody, but I feel like very few people would have a degree in AI at this point, right? And could be able to come out and tell you, "Oh, I know everything about AI troubleshooting. I know how to make sure the responses are accurate.

I know how to make sure that we're keeping track of the costs. " Nobody knows how to do that, so everybody's having to learn on the fly. And so I think those are the main things that we talk about in our report and how we try to clarify it and I'll tell your viewers to go read the report.

I don't want to give away everything, but I think those are the main themes that we had when we were doing the report. Absolutely. And now I've got to tell you, I have met a few people who actually have PhDs in AI.

Wow, okay. And so there are people who have been academically... But just because you have a PhD in AI doesn't mean you understand tokenomics.

Right. Doesn't mean you understand the finer points of using Claude versus Chat versus something else on this and that. I think just like we see in cybersecurity, people will go to school and learn theoretical cybersecurity, but practicing cybersecurity at an enterprise becomes a bit of a different course than playing around in school with it, unfortunately.

Yeah. Greg, I want to jump in, though, and I always like to ask for our audience out here, because I know we only get a few minutes with you, what are the top three takeaways from this year's report? Hmm.

I would say probably the top one to me is that over half of the people that responded said that they can't accurately control AI costs, right? That they don't know how to figure out how much they're spending. They don't know how to control it.

They don't know how to manage it. And- It's just a big black hole, man ... And right now, maybe that's okay because everybody's running to adopt it, and it's like let's adopt it and figure out the budget later, but later is coming, right?

Later is coming pretty fast, so. Later is coming, and it's got a bill in its hands. Yeah, and I think the biggest takeaway is that so many people still don't know how to control those costs and even how to measure them, right?

It was 54% of the respondents don't have clear business outcomes for their AI. They don't have it tied to revenue or customer satisfaction or something like that. So, if you don't know how it impacts your business, what are you even doing?

To me, it's like you've got to have something that tells you this is worth spending money, right? Well, that's the whole point is because you've got sort of a double whammy here. Number one, I can't even figure out what I'm spending.

Number two, I don't know why I'm spending it. Yeah. And then I ask you, well, why do you think AI stocks had a bad week last week?

Well, maybe that's part of the reason, right? Maybe that's why. Well, I don't know what I'm spending, and I don't know why.

Exactly. Those are two kind of fundamental questions I think you've got to ask yourself about this. Greg, let me ask you another one.

There's always something in these reports that you shake your head- Hmm ... " Right? What didn't you have on your bingo card this year, you think, that came out in the report?

The thing that I was honestly the most surprised about was despite saying 62% of the people said that they use AI agents either frequently or all the time to find root causes for problems. I expected that. I'm not surprised about that, right?

But what I was surprised about, that only 6% said that they use AI to write new code all the time. And if you read the news, if you paid, that feels very low to me. I was really surprised- Yeah, it feels really low ...

that small of a number, right? Because how are those 6% turning out a majority of the code, right? Exactly.

So that was a little bit surprising. And on that same question, 26% of the people said they never let AI write code, so- Oh, excuse me, 26, one quarter of the people, basically. Wow.

Yeah, correct. And which again is kind of surprising, but I think part of that is that we do a pretty diverse sample. We get across industries and geos, and so I assume there are some industries that are probably a little more behind on the AI adoption curve than others and some regions as well, where it's probably not as common.

But I was pretty surprised at both of those numbers, to be honest. Yeah. That seems like it doesn't add up.

But they might, you never know. It sometimes depends on, as you said, who's asked the question, where they live, how it's asked, what was the question before, what was the question after. Right.

There are variables that come in here. Greg, we're almost out of time. But for people who do want to maybe take a...

And guys, if there's one report you're going to take a good dive into that you'll find interesting, it's this one. Greg, for people who want to go grab the report, what's the best way to get it? com/state-of-observability will take you there, so.

com, I'll bet, or Google it even. Yeah. If you don't want to remember the URL, definitely Google it.

Well, that's an easy enough URL. You know what? Say it again, Greg.

com/state-of-observability. Love it. Hey, man, I appreciate you coming on here on Techstrong TV.

I hope to see you out in Salt Lake City for KubeCon. Well, I'm sure there'll be a lot of observability- Oh, yeah ... sessions and learning and announcements.

Yeah, we're really looking forward to it. We're also hosting an OpenTelemetry graduation party. So if any- Very cool ...

of your viewers are going to be in Salt Lake, come join us Tuesday night. We're music, beer. Oh, I'm looking for my invite.

Yeah, of course, absolutely. We'll get you there. I'll be there.

All right. It's on. Greg, a pleasure.

Greg Leffler, Director of Developer Evangelism at Splunk, talking about this year's version of the Splunk State of Observability Report. That's the 2026 edition. And they'll be, as he said, at KubeCon next week, as will Techstrong TV.

We're going to take a break here on Techstrong TV. We'll be back in just a minute with more.