Apica’s Andi Mann on the Future of Telemetry Data Management

Hey guys, thanks for the tour. We're here with Andy Mann, who's newly appointed chief product and technology officer for a pika. And we're talking about telemetry data because, well, there's more of it than ever.

Andy, welcome to the show. Hey Mike, it's great to be here. Thank you.

Alright, Congratulations on the new gig, but I have to ask you, have we somehow gone from, I remember, it just feels like a few short years ago, everybody's saying we don't have enough data to analyze. 'cause not enough stuff was instrumented to now we have instrumentation and more data that we know what to do with. So, you know, I guess is this becoming too much of a good thing?

Yeah, look, it is to an extent. Um, and this is actually a logical progression because we started with a lot of packaged applications that didn't actually give us a lot of insight. You couldn't look inside them.

And you know, when we talk about observability, that's what it's all about. It's being able to look inside a system and see what is happening based on the external sort of telemetry from that system. So we had package systems, NetSuites and SAPs and Oracles and Siebels and all these sorts of things.

Not to, not to mention all the financial apps and everything like that, which weren't particularly well instrumented. So we had to do unnatural acts, right? We had to do things like synthetic transactions.

We had to do things like, uh, uh, you know, real user monitoring, monitoring at the end point and looking at what's happening, uh, monitoring on fake signals that we are generating in order to get insight. This fundamentally changed around development on cloud. And we started to create these applications, you know, to start with custom developed.

Secondly, uh, built on these sort of, uh, uh, uh, atomized architectures. So they're all talking, you know, components and serverless bits and servers and virtual and all talking to each other. This created the data that we sort of needed.

We were sort of missing. So there's still a need for things like synthetic transactions, but we're getting a lot more information directly from the applications themselves. So this is all this telemetry data and obviously as we're building more and more and we're building more and more atomized applications, we're getting more and more data to understand what those applications are doing.

So for a single monolithic application, you might have a whole bunch of signals coming out of the server, the storage, the network, the app for an atomized application. You've got content and data log files, events, metrics, traces coming out of every component all the time. So as we continue to build and everyone's building new things, everyone wants to grow their business, do new things in new ways, beat competition.

So the way we do that is we create new products and services, we bring 'em into market, we have to have technology to support them. So we're creating a whole lot more applications. And AI is helping us create even more applications and even bigger ones.

And we're using different architectures and technologies and methodologies to create those applications, which are creating more data as well. So this is just exploding. Telemetry data is nuts and we do need it.

The more data we need, the more data we, we we have, yes, the better. But wow, mate, we are paying for that data. IT ops, devs, CIOs, CTOs, every year their bill for observability data goes up and up and up, and we're seeing data volume increases like 70, 80%.

And pricing increases are sort of, you know, a little bit linear from most vendors. So yeah, look, for a lot of the customers I talk to, this is a prime concern. They don't want to get rid of all the data.

They want the data to be meaningful and to be able to handle those increases without blowing their budgets. Mm-hmm. Theoretically, I think you can route a lot of this data to, I don't know, an S3 bucket to reduce your costs.

But I think the challenge then becomes, well, how do I get it back when I need it? So, you know, how do I manage that whole end, end, end process? Yeah.

Mean it starts with routing the right data. 'cause you're absolutely right. You don't want all your data in one place necessarily.

You want it where you need to to use it. And a lot of data is, I won't say meaningless or pointless or worthless, but it's certainly low value. You know, you think about a, uh, a verbose application, which is issuing log data statements, log file statements that are just saying everything's good, right?

Yes, transactions succeeded. Honestly, as an IT operator, I do not care about that. I do not wanna know.

It doesn't come into my event analysis, it doesn't come into my problem determination or prediction. What I wanna see is when it doesn't work. So start by filtering, you know, don't even send stuff that you don't wanna see that's not meaningful for you.

And, you know, every now and then you'll get a developer turning on verbose mode with logging, right? And all of a sudden you are sending terabytes. Uh, look, I, in one of my earlier lives, I saw a customer, uh, turn on verbose logging, and they were literally is is sending blob files, binary, large object files, screenshots of applications in p and g format being sent as binary log messages because logging was verbose, right?

We don't need that. So filter the data, make sure you are getting the right data, you know, do things in the pipe instead of the expensive ingest and storage layer. So, to do things like, you know, and this is what we are doing is filtering and, you know, masking, masking PII and confidential information and filtering out noise and, and, and, you know, crunching a single log line into a maybe a metric.

Uh, instead of sending a whole verbose line that says, I got a 4 0 4, just send a fail 4 0 4, you are already saving data, you're saving transmission costs, storage costs, makes it faster as well. And then yeah, absolutely select the right repository for the right data. You know, various observability platforms can get to multiple locations, you know, their own proprietary data stores, maybe an S3, maybe a Hadoop database, maybe a snowflake.

You know, these are all different repositories based on different data needs, access time, uh, uh, you how important it is to things like troubleshooting and triage, how important it's to predictive analytics and preventing problems. So yeah, look, there's, for me, there's three key elements here. One is get the right data in the first place.

Process it in the pipe to make sure you've got the, the, the, the tight data set that you can work on. And then make sure you're storing in the right location. I guess the first question that comes to mind is, which pipe?

Because, um, we have DevOps pipelines everywhere and no one's quite sure exactly where to insert the telemetry data management. And is it everywhere or is there a focal point? What, what do we need to think about?

Yeah, look, I mean, data telemetry, it is potentially everywhere. And look, I've done a lot of work taking data out of development pipeline and the software development life cycle to understand the activities happening in that work stream. So for example, taking log data out of bit, uh, out of GitHub for out of Jira, out of Jenkins or Puppet or share in terms of deployment, taking log data out of test tools, uh, test automation for example, code compliance tools and being able to take all that data.

So yeah, look, pipelines are pipelines, there's data everywhere. You know, I think the pipelines that we are most interested in are the pipelines coming out of the telemetry data systems on servers, infrastructure applications, middleware, that sort of thing. You know, we're looking at helping IT operations teams and development teams to understand things like performance utilization, capacity failure, root cause they're the sort of data items that I'm most interested in working with right now.

So we're taking data out of things like servers and stats, d open telemetry, uh, any open telemetry collector, for example, shipping that into our own proprietary processing system, a peak of flow so that we can then look at that data, understand it, and then forward it to the right repository. Whether it's a, a Splunk, a Datadog, a Dynatrace, a New Relic, uh, a Grafana or an elastic. Maybe it's most, because most customers tend to have three or four of these platforms, by the way.

So what I'm talking about is this a performance availability fault, A failure error, these sorts of data lines, both logs, metrics that say, you know, am I running fast or slow? For example, uh, uh, uh, traces, you know, intercommunication between different application components. Uh, so these are the data streams that are providing information about whether my application is up and running, running well, delivering responses or not.

That's the sort of stuff that I'm talking about. But you're absolutely right. There's data everywhere.

There's data coming out of IOT devices, point of sale devices out of mobile devices. These are all potential candidates for data streaming that CIOs and CTOs need to worry about because they're all gonna cost time and money to process. And so yeah, the same principles apply.

What I'm looking at is that telemetry data for IT operations and performance management, but there's so many different data streams you could think about in this conversation. Mm-hmm. So to your point, do I need a specific data engineer who knows about the nuances of telemetry data specifically, or is this something that can be incorporated into everybody's kind of day job as it is?

Yeah, look, it depends on, it's it, right? So it always depends. Uh, it depends on what you're using the data for and who's using it and how cluey they are and what tools you've got to make it easier.

If you're handling these data pipelines by yourself, then absolutely you're gonna need a data engineer to figure that out. You might actually need a network engineer as well, because that's, you know, pretty tightly related, but with good tooling, you take care of that, right? It's why we use tools in the first place to save ourselves time, money, effort, knowledge, uh, resources, all this sort of stuff.

So if you apply good tooling to this, then a lot of it becomes a lot simpler. You can use graphical interfaces to be able to define filters, deduplication, masking, these sorts of things become a lot easier because they're just straight through a gui. Maybe if you're a developer, you want to use the API or a CLI to do it, but you can still do that as a developer, as an operator if you have a knowledge of what the application is doing and why.

You know, everyone needs systems thinking, I think. Um, but if you don't have that kind of tool and yeah, look, it gets complicated. You're dealing with binary data streams and you're dealing the translations and network interfaces.

You're trying to get to layer what, I don't know, layer six, layer five maybe in terms of the data content and the payload to understand what's the contents are. You could also, by the way, if you're doing it manually like that, come against some pretty serious security and compliance implications around individual data scientists being able to look at confidential data, see it, access it, work with it, maybe leak it. Uh, but when you provide good tooling, then you get to do a, the CLI or UI set up.

You get to have security and compliance factor built in, including things like, yeah, role-based access controls and you know, mask, automatic masking of obvious PII like telephone numbers or credit card numbers or social security numbers, for example. Um, you also get the ability to see into the pipe, into the, into the, uh, payload without necessarily having access to the data itself. So you've got better compliance and governance.

So look, mostly this is the sort of thing that developers and operators would collaborate on. DevOps being the big thing, right? Still collaboration.

Um, you probably don't need a DI data scientist though if you have the right tool sets. Mm-hmm. So will, um, maybe I see something that looks like an AI agent emerged to help me manage all this stuff.

It seems like it's a good candidate for these jobs, of course, in my mind. Um, good candidates for AI agents or anything that I don't enjoy doing. So, um, is this something that's gonna be high on people's list of Yeah, I want an AI agent for that.

Yeah, look, I don't wanna give away what's on my roadmap right now, you know, watch this space, but speaking generally, think about you, you're, you're exactly on the right track track, mark, you, you think about, you know, this data is, is potentially complex. It's high volume for sure. Um, it's high cardinality as well.

So it means there's lots of changes in it. There's not a lot of this that's similar. Um, and so that's the sort of perfect candidate for automation and AI to take care of, right?

This vast data set of hard to detect patterns that do exist within the data stream that a human looks at it and goes, I just can't even pun intended grok any of this. And so you're absolutely right. AI in the pipeline is gonna be super interesting.

Uh, you get to do a bunch of stuff just based on pattern matching, which, you know, advanced machine learning AI let's, you know, see what it is. But then you can apply LLMs based on known knowns. So think about Windows system log as a very simple example.

Windows system log has a lot of known content, you know what it means and it's documented well. So now we have an LLM, right? We have documentation in the log itself.

Now we can start to apply ai, apply AI techniques and generative AI techniques to learn from documentation what is meaningful in that log and only collect and forward to these expensive observability platforms, those log messages that actually mean something. Or even better transforming those log messages into much smaller bys so that you are reducing traffic but still getting full accuracy on your data stream. So look, this is coming on, it's coming on strong.

I actually think that if you can't apply AI at the source, I'm talking within your application code to be able to restrict and limit what data you are sending out, the next best place has gotta be in the pipe. Right? True.

Well, let me ask you a follow up question on that. 'cause it seems to me therefore, based on what you were saying, that uh, I don't know, am I gonna see like an LLM for telemetry data sometime? Or is there gonna be specific ones for that?

Or are there gonna be these general purpose ones that I kind of bend to this purpose? Yeah, look, I don't know the answer that if I had a crystal ball, I would obviously be a very wealthy man and, and, and maybe sitting on an island Hawaii or something. But, uh, I, look, I think you're on the right track.

We, I don't think it's gonna be general purpose for this. I think it's such, I wanna use the term arcane knowledge. Uh, it's very specialized.

It's almost wizard like some of the knowledge that people have about these log messages, traces how they work together. Um, and so I think that it's gonna be based on maybe a small language model instead or maybe individual unique language models for each business, because this is a challenge as well as the compliance angle. If I wanna create an LLM based on, you know, something like system log is easy, but if I wanna create an LLM based on an application log, now I need to know what that application is.

We used to know that with packaged applications, now we've got non packaged custom applications. So I can see a pathway for an AI to get trained on both the application code, the application documentation, and be able to use that training in a small language model way to be able to filter out meaningless sort of verbose debug kind of stuff versus meaningful sev one error kind of stuff. The challenge is again, whether that data is available as a large public LLM or whether it's just unique and arcane to that one business because that their application and their uh, uh, intellectual property as well.

So it's gonna vary. I think I would love to see, for example, and we've got this for a lot of, of the traditional observability data sources, things like Windows system log or security log, things like, uh, stats d collect D data coming out of infrastructure. We've got good knowledge and that sort of content could be used as a training for an AI gen AI to be able to work with that data on the pipe dynamically.

But I think it's gonna be more complicated that simply because of the custom applications, proprietary data, confidentiality and honestly, people still do stuff OnPrem on premises. And so public cloud data doesn't come to apply to that use case either. So it's gonna be a varied, uh, uh, way of looking at it.

I'd love to see it sort of universal across the world, the way we do some of that stuff with security penetration and, and, and attacks in zero days with sharing that information through things like Mitre and other organization. It'd be great to be able to see that for at least package applications, but it's always gonna have to be, be ameliorated with a little bit of proprietary knowledge and internal training for those custom applications. So what is that one thing you see people doing with telemetry data that kind of just makes you shake your head a little bit and go, folks, I wish we were just a tad bit smarter about this.

Oh mate, dumping it, I think is the real bad one. Uh, people look c and especially at the admin level and the individual contributor and team lead level, they look at these bills they're getting and they look and, and you know, they don't wanna have to go to the CFO and try and get more money, so they try and take remediation actions to try and get their bill down to try and, you know, deal with this new applications that's coming online. I, I've got another, you know, 16 gig of data every day.

I've gotta get into my observability platform because of this new application. It's really important, it's competitive differentiation in the marketplace, but I don't want to have to pay a big observability storage bill and transmission bill and ingest bill. So look, I'm gonna have a look at what I'm already ingesting and maybe just dump some of it.

'cause it's not that important. It doesn't, you know, I haven't had any problems with that application in the last three months, so I don't need to hear from that application anymore, right? That is almost never true.

So they're dumping entire data streams. So just saying, oh look, this application, just turn it off. They're dumping it for certain times.

So this application, turn it off between 6:00 PM and 8:00 AM like, we don't have a global economy, right? Um, they're dumping data by cardinality. So just let's just summarize all these data points and they're dumping it by just doing sampling.

So I'll sample my data stream. So once every two seconds I'll take a data point. What happens if the problem occurs within those two seconds?

So they're, look, they're doing the best they can, Mike. Um, they haven't got more money, they haven't got more storage. They need to do what they can to accommodate these new, this explosion of telemetry data, but they don't always know what the right decision is to make, to be able to deal with that data volume increase.

Rather than use a sophisticated, you know, tool set which can manage that without dumping any data and reduce your cost. They just go for the quick and easy one, which is just like, let's get less data in, and that's gonna be problematic over the longer run. All right, folks, you heard here the only thing worse than looking for a needle on the haystack is knowing that you threw the needle away already.

Hey, Andy, thanks for being on the show. Hey, Mike, it's great to talk to you, mate. I really appreciate it.

All right, and back to you guys in the studio.