Applying Generative AI to IT Operations with BigPanda CEO Assaf Resnick

This is Techron tv. Hey guys, thanks for the throw. We're here with us, SAP Resnick, who, CEO for Big Panda, and we're talking about the next phase of AI and how it'll be applied to IT operations as sap.

Welcome to the show. Thanks For having me. I think if we look back, we have seen machine learning algorithms be applied to IT operations, and you could argue that was the first phase with predictive analytics.

And we've made some progress and there's still work to be done there. But now we're seeing the rise of generative ai. What is the difference in your mind?

How do these two things come together and we will one plus one equals six or more? Sure. Um, you know, I think the rise of generative ai, uh, does two things.

It, it adds a different way to process all this data and automates, uh, how you turn that into human insight, make it easier for human beings to, uh, interact with that data. Uh, it does not replace a lot of the other types of machine learning and AI that are very good at doing correlative type analysis. Uh, very good at doing, um, looking for, uh, u unique things, anomaly detection, things like that.

Generative AI will not help with that, uh, but it will bring meaningful, meaningful value, uh, to that, you know, human machine data interface and make that data much, much more, uh, intelligible and reachable for human beings. That's one. The second thing we, it will do, I think, is dramatically increase the, the velocity and the quality of the types of AI generated, uh, features you can drive to customers.

And I'll tell you why. You know, before when, let's call it, you know, earlier, earlier iterations of, uh, ai, um, AI was fairly res ar ai research and gen and development, or at least the research side, was very bottleneck to very, uh, SME data scientists paid PhDs from Stanford. There's very few of those people.

They're very expensive. You are, uh, competing with them with Facebook and Google and Microsoft. Uh, and so there's, there's, there's very few of them that are expensive.

The, the lead time to create new capabilities typically is a year to two. You have to take a bunch of data, you have to conduct a bunch of research on that data, create new types of AI capability capabilities, and hopefully those AI capabilities actually solve a problem for an actual engineer or an actual operator in the real world. Sometimes they did, sometimes they didn't.

Gen AI comes along and says, Hey, you can now use, you know, assuming you bring great data to the party without great data, you know, any kind of AI is, is, is, won't give you great outcomes, but if you can bring great data, uh, then you can now use prompt engineering to build some really interesting capabilities that were a lot harder to build before. So that means you can take a, uh, someone who's technical, an engineer, an IT ops engineer, a DevOps engineer, not a data scientist, but technical enough to do prompt engineering and they can really leverage this platform and do some amazing things. And so for us, it means, hey, we've got now a hundred x more types of people we can bring, uh, to develop, uh, on top of gene ai.

It doesn't have to just be data scientists. And these people they've been bringing, they can be people who are much, much more familiar with their customers, have much more empathy with the kinds of pains that they go through, and they can create and innovate with a much better idea of, you know, what are real world solutions to solve these real world problems. So, so gen ai, you know, certainly lets us use that kind of democratization to run faster and build better.

As we think this through, though, it seems to me that we have all these AI capabilities, but have we thought through how to operationalize all this stuff? Because generative AI will make some recommendations or suggestions, but how do I turn that into something that I can execute? Yeah, I mean, that goes back to, it's not about the ai, it's about solving real world problems.

You know, I can give you great gen AI that's got a really good chat interface that really seems like you're talking to a human being. But for someone in operations, their problem is, you know, the website that I'm maintaining is down and there could be 17 different problems, reasons that website is down and my customers are very angry and I'm losing a lot of money. Help me fix this.

And if the AI you're building doesn't help you fix that, well that's no good. And if you, and if what you're building does help me fix that, I don't really care if, is it based on ai? Is it based on predictive, you know, uh, uh, uh, deterministic algorithms?

Is it, you know, what does it matter to me fix my real world problem? And so, you know, you've gotta find the right balance of, you wanna leverage the latest and the greatest AI to make sure you're moving faster, innovating faster, but you can never lose sight on. And that's only interesting if you're solving real world problems that, you know, makes people's lives better or drives meaningful.

ROI, It seems to me there's a world of difference between a general purpose LLM such as chat, GPT and a more domain specific LLM designed for IT operations specifically. And do people need to be aware of this? 'cause it seems like the amount of hallucinations might vary widely.

So, you know, there there's two, uh, it's a great question and I think there's two ways you wanna look at that question. One is the accuracy of a general purpose of a loan versus a a, a domain specific LOM. And then there's also the cost and functionality.

So from the general purpose, you know, from a, from an accuracy perspective, you want AI that's gonna give you great insight. You want a low level of false positives, you want a low level of false negatives. And so therein you have to, you do have to make product, uh, trade offs when you're looking at, did I use a domain specific LLM for this particular set of, uh, issues, or should I use something domain specific?

And there's no one answer, there's some answers, you know, some cases where the output from a gen, uh, a general purpose LLM is perfectly fine and there's no need to specialize. And there's some areas where it's either the amount of heavy lifting or the amount of accuracy. Uh, I do need something domain specific.

Uh, and so that's part of the calculus of when you build something, what platform, what tool do you wanna use for that specific capability. Then there's also the cost aspect, which is, if I'm running a workload over and over and over again, do I want that workload to be run on a general purpose LLM where there's may be a lot of, uh, unnecessarily, uh, uh, overhead and I can do something cheaper with something that's much more focused than domain specific. And, you know, that's, that's also something you wanna think about as you scale.

What do you think the impact of all this is gonna be on the daily life of your average IT operations manager? I mean, are they gonna be able to manage things at higher levels of scale, less stress, or how does this all play out? Yeah, absolutely.

Uh, I think this is gonna have a huge impact for the IT operations vertical. Uh, you know, especially there, these are people whose, you know, job over the last 10 years has just gotten harder and harder. 'cause the amount of machine data, the scale of that machine data, the fragmentation of it has just grown by orders of magnitude as, as their enterprises move to the cloud and adopt microservices and do CICD.

That's just more moving parts moving faster. And, uh, ai that's one of those areas where it can actually bring a lot of help. And so our vision today, uh, or our vision is to automate as much of the workload of those IT ops, uh, individuals, especially the ones that are repetitive and low value.

And so today, you know, we call it a stage three autonomy where we're automating 20, 30% of the human workload. Uh, and so that's 20, 30% more time that these people can go to develop software, be innovative, do their day job, uh, with gen ai. That's certainly take, going to take things further down the road towards kind of stage four, stage five.

'cause it can add a whole new perspective of how do you take lots of data and turn it into really, really easy insights and, and hopefully automation. And so it's gonna take a lot of the drudgery out of the work that folks in IT ops ha have to do and give them much more time to, you know, build and innovate, which is what they wanna be doing. It's what their companies want them to be doing.

Do you think a lot of the silos that we have today will start to break down? I mean, we have ITSM specialists, DevOps specialists, data engineers, all kinds of folks, so we get to a more converged motion for how we manage all this stuff. Yeah, ly it's happening right now.

It's a great, great, uh, question. You know, it, it's, you know, to have through IT automation, you need to have two things. You need to have great data and you need to have great ai.

And before you have great ai, you have to have great data. And so one of the things that we really focus on is, is you have to put everything together into this kind of unified data fabric. And like you said, you know, uh, 20 years ago the average enterprise, you had one operation center with, you know, 50 engineers sitting in a room looking at everything across the business.

But then you started having, um, DevOps movements. And so a lot more of the application layer, the microservice layers owned by developers. You have SREs, you have cloud ops, you know, you buy new lines of business and pretty soon you've got six or seven different teams that are responsible for, you know, some business service to run.

And you know, me as a consumer, I don't care about any of that. com, buy some shoes, and have a great digital transaction for that to happen. All these different silos need to work together, but their data lives in different places.

The left hand doesn't know what the right hand is doing. And so part of what AI can bring, or part of what, you know, creating that great data fabric for AI is you gotta bring it all together. You gotta unify it.

So you can't have seven different tools speaking Chinese and French and Japanese and, you know, unstructured data. You gotta get it all speaking the same language and, and, and normalize it and enrich it together. And once you do that, you can really start to unlock, si unlock silos.

'cause people at the application layer can suddenly see what's happening at the bare metal network cloud and so on and so forth. And so that's actually unlocks a ton of efficiency across the business. You know, I can't tell, I can't tell you how many times I've talked to people and said, Hey, oh man, you know, when something, when we have a a a P one outage, a really big outage, right?

In our, in our most mission critical application in the business is going down, I've got a hundred people on the line and they're all running around, no one knows what's going on. They're just trying to figure out, is it me? Did I do something?

Did my team do something? And it can take hours just to get that kind of aligned visibility. All the meantime, you know, your customers can't use whatever core product.

You're, you, you're selling them. Uh, and so bringing that all together into a single data fabric where, you know, all that data really talks to each other, that's half the battle. And that, that unlocks so much value.

To your point, does that mean that the classic war room experience is gonna disappear and we won't bring everybody into a room and not let them out until we prove their innocence? Uh, I, that's a great question. Uh, more and more and more it will disappear.

You know, think of what we do. We come to a customer and say, Hey, we wanna be able to automate awareness. So you help you detect where you have issues.

We want to help you automate on, uh, insights to understand what is the root cause of those issues. And then we wanna help you automate remediation. And only if we can't do all three of those things, then we'll be calling in a human being and a war room.

Now, I'm not saying, you know, so, you know, will war rooms completely disappear in the next five, 10 years? Probably not. But when you do have war rooms, wouldn't it be nice if you can get those initial 50 people on the line and AI automatically tells you, here's what's going on, here's a timeline of how this event started to escalate across your organization.

Here's what I think the likely root cause is. You know, what a dramatically better way to start a war room than spend the two first two hours just trying to get situational awareness of what the heck's happening. And then every time a newcomer comes in, you know, there's a big problem.

The CMO or the CFO or their representative comes in 20 minutes into the call and they're a big wig, and you've gotta spend five minutes describing to them what's actually happening. Wouldn't it be nice if AI could do that for them and give 'em a readout, Hey, here's what you missed, and that that's what's happening. That's what AI can do.

And that that saves, you know, tangible in environments where seconds of minutes matters and you can save hours. That's a big deal Wise. Man once said, the future is here.

It's just unevenly distributed, as he said. Um, how long, how long before all this becomes pervasive? Uh, I, I, I think it's happening now.

I I think that, um, the pace of AI adoption has dramatically increased. You know, you used to have generic, like, you know, when we first came to market with AI ops and we said, Hey, folks in operations, you can use AI to automate, you know, your most mission critical operations. That was very scary for folks.

And they said, well, you wanna use some black box and, um, gonna magically, you know, automate my operations, which is the core of, you know, many of our customers run, you know, half the financial traffic in the United States or a third of the financial traffic in, in the eu. You know, these are people with a lot of responsibility on their shoulders and we're fitting in line. They're saying, wait, you wanna use AI that I inherently don't trust to automate a lot of this stuff?

Mm, that sounds scary. But now, you know, AI is driving their cars. AI is, you know, helping their children find better videos to waste their time on in TikTok.

And so they, they see it in their everyday lives. And so they're much more open to, well, I can drive my car, maybe AI can help me automate some of my, you know, enterprise workloads, including operations. And so now people are asking for it.

And I think gen AI in general has made the whole discussion around AI much more tangible and much more real. So like 15 years ago, CIOs would say to their people, like, I don't exactly know what we're doing in the cloud, but I want to go do it and go do it faster. We'll figure it out.

I think that's happening right now with ai. And so I do think that there's a ton of value in our space around operations to bring a lot of very tangible value. And, you know, uh, companies are waking up to that.

All right. These issues are as much cultural as they are technical. What's your best advice to organizations about how to get ready for this next phase?

Yeah, that's great. Um, uh, there's a lot to unpack there. You know, it's around unification.

So, uh, I'll lemme tell you a lot of the, the, the meta problem facing A CIO and then what do you do about it? So the meta problem is, you know, 30 years ago they brought an entire tech stack from BMC or ca or some large monolithic vendor that said, I'm gonna sell you everything. And that was nice 'cause you have a single throat to choke, but guess what?

Now, you know, they've got more power. The switching costs are huge, they've got more power in that relationships are gonna, uh, innovate really, really slowly. And over the last 20 years, Silicon Valley has brought a ton of innovation all across the tech stack from virtualizing bare metal to taking into the cloud, to turning into containers, to doing CICD, to observability, everything.

And there's a lot of different tools you can use and your team is using, uh, to choose the right tool for the right job. And, you know, a lot of the power for the operational environment is no longer in the hands of the CIO. It's now in the hands of the developer.

They own a lot of the operational, uh, responsibility for, you know, their microservices and scrambled egg of an infrastructure. And it's got a lot of benefits. And so what do you do?

You could either say, no, I wanna go back to a, I wanna consolidate back to a single vendor. The names are different today, but it's the same shtick. More and more of your power, uh, of your environment will, will be in their hands.

It'll be a closed garden and they'll innovate much, much slower than other folks. Or do I keep going a best of breed? Most of the folks that we see, you know, say, Hey, no, I want to continue to tap into the innovation of velocity coming outta Silicon Valley, but how do I unify that Tower of Babel with so many teams and tools and processes?

And so my advice to folks would be, as you continue to modernize, uh, your tech stack and optimize for velocity and agility, which is a good thing, you have to be able to unify your data and get everyone on the same page. And so, you know, anyone who touches a business service, which is another way of saying anyone who touches a consumer experience, you need to be able to see the, the same data as your neighbor. 'cause if you're not, you're gonna waste so much time just trying to get in line and you're, and you're never gonna do a good good, give a delightful experience to your consumer.

All right folks. Well, you heard it here. No matter how smart the systems get bailing the plan is planning to fail still.

Hey, has thanks for being on the show. Yeah, my pleasure, my pleasure. Back To you guys in the studio.