Mastering Observability: Insights with Bob Quillin – KubeCon Europe 2025

This is Textron tv. Hey everybody. We're back at CubeCon plus Cloud Native Con in Europe, and we're talking with Bob Quillin as the CEO for controlled theory.

And we're gonna be talking about observability. Bob, welcome to the show. How you doing?

Yeah, Great to be here, Mike. Good to see you. We've been kind of working on this observability thing now for, uh, half a decade.

Yeah. At least. At least.

Yes. Um, and the one thing that seems to become apparent is the good news is we're collecting more telemetry data than ever. Mm-hmm.

The bad news is we're collecting more telemetry data than ever, and it's going into these multiple platforms and systems and, but we don't seem to have any way to kind of organize this stuff. So what do we need to do to like, make this observability thing mainstream for everybody and maybe simpler to manage? Simpler To manage?

Yeah. Yeah. You know, we've started this new company Control Theory and we're kind of focusing on trying to solve that problem.

Uh, and it's called controllability. And it's, it's a kind of a corollary to observability. And the idea is to put more control, uh, help you regain control of your observability, um, through three areas.

Cost control, operational control, and adaptive control. And for us, it's like putting a control layer on top of your telemetry, your traces, your logs, your metrics. Be able to control what goes into different systems to filter it, de-duplicate it, uh, maybe rerouted to appropriate places, mask, uh, secure data.

It's a whole bunch of different use cases that you can sit, uh, put on your telemetry in flight before it actually gets to your observability system. So putting more intelligence in how telemetry, um, how your observability is delivered, um, in advance. So really more intelligence, you know, outside of the current systems into the way things are flowing out to those systems themselves.

And one of the issues I hear about all the time is the cost of observability is killing folks who get down. Mm-hmm. To that point.

'cause they're like suddenly collecting every piece of telemetry data in the world, but maybe we don't need to collect every piece of telemetry data in the world. Yeah. And, and that's kind of what got control theory started.

We, um, you know, looked at the way the systems are architected today and it's like big fat, fat dump pipes going into, you know, big data stores. You're paying for all that. You're indexing it, you're ingesting it, retaining it, and all that's being paid for.

Um, and that's what makes the cost go up. 'cause you all the data's coming in, it's not really intelligently controlling what data goes there. So if you could control what you get, what you want, when you want it, and for whom all that can be built into the system using telemetry pipelines.

And they're built on top of, uh, open telemetry, which is, you know, a big theme here at, AT with CNCF and, and kon. Um, and the idea is really to put that control back in. We, we started looking at observability to solve a lot of issues that we heard around, I can't get my root cause analysis working.

I can't get my MTTR down, I can't get my KPIs. But then, you know, the more we talk to folks, really the, the first issue is cost. And that's kind of what you bring up is soaring costs, soaring telemetry volumes.

So getting the cost down is one thing. And then getting the value up is the other thing. And trying to rebalance the equation, the ROI of, you know, what you're getting from your observability and what you're paying for it, you actually balance that out by having this control layer, this control plane that says on top of open telemetry.

So any source, any destination to flow right through use what you have to make the observability better, more control, more intelligent to deliver the information you need and when you need it. Um, when we think about the observability and the telemetry data and as you describe it mm-hmm. I'm reminded of classic data engineering.

Mm-hmm. Oh yes. So are we kind of applying data engineering to telemetry data?

Very, very much so. And um, I, that's kind of what we went back to our, our roots in this control theory control systems. You think about even a thermostat controls your temperature based upon the temperature that's out there.

It's a feedback loop. And that really was one of the things that got the company started was this concept of a feedback loop. If you could actually can put some more intelligence in the pipes themselves versus being one way telemetry pipes, you could have a feedback loop that actually can control and have more intelligence of what you send when you send it.

Maybe only send the important things, maybe I'll tail sample and look at what's where the, uh, the issues are. If there's performance degradations, I could actually then send the data, uh, only the data you need. Maybe I'd go store the other stuff in lower cost storage, like AWS three or click house or something like that.

So be more intelligent about how you do this. And having those feedback loops, really that's the basis of control theory. There's observability going one way, but there's controllability coming back and those two things have to work together.

So that control element, I think that's the next phase in observability. You know, open telemetry gives us instrumentation, gives us collection consolidation. So it kind of standardizes all those things.

And those that has really matured to the point now you can start building controls on top of it. And that orchestration layer, that adaptive layer that manages and its policies, we kind of split it up into a control plane and a data plane. And the data plane has all the collection, all the instrumentation, it's control plane, then has all the intelligence to actually manage that.

And splitting that up and having that divisional labor divisional responsibilities gives a lot more scalability there. So Yeah, It's kind of like the way the cloud is managed, right? It's the same principles.

Absolutely. Yeah. And you know, our, our, our founders, we came from Stack Engine, um, we built out, we got acquired by Oracle Cloud and uh, we built out the managed Kubernetes service at Oracle.

It has a very scalable service. It's running some of the biggest AI workloads in the, in the world now, which is, you know, very cool. Um, but that had a whole data plane and a control plane separation.

So you can scale that up and it's a very common pattern. Um, but right now everything's in the data plane. So everything's very, you know, very dumb, very unintelligent how it's sending information.

Let's add some intelligence split, split the divisional responsibilities, divisional labor, put the control plane there. So you control what you're doing. That actually helps us move forward and take advantage of, you know, not only was, you know, solving problems.

Now you mentioned costs, uh, we mentioned, we mentioned KPIs and, uh, better root cause analysis, but also AI problems that are coming up. 'cause all those AI systems need data and a lot of the AI observability systems that they are having to go get their own data 'cause they can't get the right data they need and it's not curated. So in the future, having the right data set to the right place is another big use case that we see as people move forward.

They're bringing open telemetry in building more controls, more intelligence, and they wanna bring more AI systems. So you want to get the right data to those folks, curate it and send the information that's needed when they need it, just what they need. Alright.

Not to geek out a little bit, but we were having some conversations about this with some other folks this week. Mm-hmm. And, um, the issue came up about if I'm collecting this data and I'm put in a database, uh, do I need a schema or can I have a schema less approach that doesn't require me to have as much dependency on curation and a bunch of DBAs to manage the observability platform?

Yeah, Yeah. The, uh, there's this balance we think too of, uh, is retaining the data and having be have it available to do analysis later. Um, and how you do that, lots of new technologies that do that.

Part of our goal is to also, also just decide what data goes where. And a lot of folks are trying just to put all their data in the biggest data store they can. We think there's a, another approach is just be smarter about that, but the important data in one area.

But all the data maybe for, you know, compliance, et cetera, later analysis where you can do use analytics like that, but deciding where the data goes, you may not have to retain all of it. One of the biggest issues we see, we've heard about from 10 or 20 people I've talked to just yesterday, these log spikes. So they'll log spike from a developer who actually is logging all this information and they set the, the debug levels accidentally wrong.

And it runs its huge level for maybe three, four or five days before anyone notices, fills up their databases with unnecessary information. A lot of duplicate data from applications they don't really care about. You wanna catch that spike, get an alert on it, turn that off so you don't have to pay for it.

Um, maybe routed off to a lower cost storage. You can do the analysis later if you need it. But that ability to catch observability issues so you don't get those bill surprises and you don't have to retain, you only have to retain the data that you want and retain all the, the duplicate data and pay for all that want to filter that out, be and be smarter and more intelligent about that.

So that's a, you know, retention's a big, big problem set. A lot of people are gonna solve that. We want to actually help people decide and data manage that in advance.

Also, You cannot walk down this aisle without somebody leaping out to tell you about their great new AI thing. But it occurs to me, we, we could apply AI agents to observability, but what might that look like? Very much so.

So as this Lumberton is in flight, there's a lot of intelligence we could do this correlation. One of the things we're, we're working on and actually showing at the booth now, is the ability to correlate, uh, events with a developer level. So I have a, a, you know, a GitHub release, I've got a deploy that just happened and then my observability spikes up.

You know, I wanna be able to correlate what's happening at the developer level of the CICD level, um, and correlate that to know that maybe that spike is related to a particular release. And I can say, oh, that's a specific application. Maybe it's a feature release.

I wanna be able to do that correlation. So that kind of correlation, that analysis, that intelligence right now, it's like, you know, it's either in the operator's head, SREs, head platform engineering team has to know what's going on. So we can start doing more intelligence there.

There's automatic baselining, there's auto tuning. So how do I Auto-Tune my systems? Um, right now observability is like, it's up to you to tune it yourself.

So how do I actually then, you know, make sure I know where the data's go, if I have multiple observability systems, where, which system should it go to? And, uh, based upon, you know, the different use cases that I'm trying to solve, problem management over here to be compliance and security over here. Maybe logs go in one place, metrics go somewhere else.

So everyone has different policies and patterns. So having that intelligence built in for routing and distributing correctly and managing that, it's got this next set of problems. And that goes right, you know, to problems today, but also problems AI in the future too.

So, mm-hmm. So There's a lot of folks who are still finding their way in observability and then there's others who are oh yeah, very mature and have been at this for a while. The ones who are doing it successfully.

What are they doing early that's different than every other folks? And, you know, are there shortcuts that the newbies should hear about now? Yeah, there's a, there's a lot of best practices that we see folks who are, you know, they, they look at logs, they look at metrics, and a lot of people kind have a good feel for how the logs and metrics works.

You see a lot of people beginning to experiment this idea of traces, which is really good for application performance. Um, and they can be kind of tricky to work with. Um, and so a lot of it's the more advanced folks that we see are really beginning to leverage traces and, um, in some ways that can really reduce the amount of logs and metrics you have to collect because the trace is really telling you what the problem is throughout the lifecycle of an application, um, message and, and, and, and patterns.

So, um, what that allows you to do though is collect a lot of traces. Unfortunately, that's, that's the problem that people are dealing with now is once I turn tracing on, I get a lot of information. I have to now randomly sample that information, uh, to collect that.

So there's ways to be much more intelligent than we're seeing people that are storing traces for later for performance analysis. They put that off in a big database we've got later. Um, they also ask us in flight to tail sample and say, only send me, uh, the traces that have, you know, high performance degradations that have high response time to send me the important traces.

Don't randomly sample, actually wait for the trace to finish, see if there's a problem, then send that one to me. So it might be more intelligent of how they use the phrases themselves. A lot of people are doing this by hand now.

Um, but those, that's kind of one of the killer use cases with an observability that that's kind of more advanced and the more advanced users are, are, are trying to find ways to work around it. And now the, the instrumentation's getting to be much more intelligent and the platform's getting more intelligent. So you can actually use those kind of, that element of observability, which is kind of the next part of after you get through your logs and your metrics.

Now Trace are kind of the power user tool. So, Hey folks, I think it's pretty clear at this point that the devil is in the details when it comes to observability. Hey Bob.

Thanks Mike. Good talking to You. Good talking to you guys.

We'll be back in a minute.