Blindspots in an Operational Technology Environment – Matt Kraning, Palo Alto Networks

This is Textron tv. Hey guys, thanks for the thrill. We're here with Matt Craning, who's C T O for Cortex for Palo Alto Networks and we're talking about cybersecurity blind spots in operational technology, otherwise known as ot, which is rapidly being connected to the internet.

Everywhere we go. Matt, welcome to the show. Thank you so much for having me.

Excited to be here. We've been connecting these systems to the internet for a while and sometimes we get ahead of ourselves From cybersecurity perspective, what are the blind spots that people aren't seeing and are the bad guys finding them? And what is the current state of cybersecurity in the land of ot?

Absolutely. So from our perspective, the two main things that have changed are one, just connectivity has increased dramatically. And also there's been a blurring of the lines between different kinds of systems that used to have well segregated, separated responsibilities and now all of a sudden, one they're getting connected more.

And two, even if policy in some cases says they shouldn't be in practice, many more connections are happening between things that had traditionally been only on the IT side of the house that takes, that had traditionally only been on the OT side of the house, including actually things that we did not think of as it now being the responsibility of it. I think of that as the proliferation of things inside companies that are now network computers, but that also control physical actuation in the real world that are not necessarily part of say a dedicated facility like a manufacturing plant but are say associated with a corporate headquarters or a building. And we're seeing this convergence just as it's actually become easier for attackers to find and exploit these systems at global scale.

It seems like it's a two level threat. One is aim at the OT environment itself to kind of take over something or manage a process or insert some random, random ransomware, whatever it may be. But then again, the malware also moves laterally and can confine its way up into those enterprise systems as well.

Do people really kind of understand the nature of this particular threat? I think that a lot of people say they do, but the people that usually have the highest confidence, unfortunately in my experience, tend to be the furthest away from the actual day-to-day of where the work happens. Um, I don't think this is unique in cybersecurity at all, but um, there was a great talk that Mr.

Rob Joyce who was uh, at the time the head of tailored access operations of N S A, so the United States' most elite hacking group. And he actually gave a very fun public talk in 2016 that was effectively how to defend yourself from a nation state actor like himself for a defensive audience. And the theme of the talk was actually if you want to protect your network, you need to know your network.

And very few places actually know every single thing and how it's actually configured. They know how they think their network's configured, but um, very frequently attackers will actually know better how the network is actually configured. And this has been true for it for, you know, over a decade now it's happening those that with the merging of these networks, things that had traditionally been segregated just for business purposes, why would you connect the factory to the internet are now starting to become connected for a lot of efficiency reasons and all of a sudden things that you've thought were security protocols in place that oh, that will never be connected.

We don't need to worry about that. We have security through say, segmentation, maybe true in policy but are not true in practice. Um, and happy to give a few examples of this as we pro as we progress, but I think that's kind of the 30,000 foot view is that how you think these things are configured and how you think they're segregated is not necessarily how they are.

And um, what I ask our um, our customers and you know, very senior security officials, how do you have confidence that your network is actually configured the way it is? How do you have confidence that the the reports you're getting are actually reflective of reality? Um, you get a lot of good answers but also a lot of candid answers about, we actually do have blind spots here because to some degree we're trusting how something was set up many years ago and that's not, and that nothing has changed.

Um, that's not actually convergent with the modern attack landscape. Where does the expertise come for all this? 'cause we've been complaining about the lack of skills for cybersecurity on the IT side of the house for better part of a two decades almost.

And do the OT guys have any better sense of cybersecurity or do they know even less? And so how do we kind of approach this if we don't have the resources? I think that almost everybody comes into work as a professional.

Not saying I'm looking to mess things up to today, but actually I'm confident and I know my job and I believe that about both IT and OT professionals. Where I think the difference now is that as networks get created and connected, the operating models no longer work the same way. So to give an example that um, we've highlighted um, in a threat report we recently released on kind of global attack surfaces, we should find that for large companies think global 2000 Fortune 500, um, about 14% of them actually have literally a building control system online.

So if you're in an O office, you don't think of it but you're actually in a computer in some sense. There's a computer operating system that manages say all of the badge readers that go to your doors that manages the elevators and where they go that manages security cameras, that manages fire suppression systems, all of the things you need to run a building. And that's an example of something that is very much ot but it's actually not how OT people think of that.

It winds up being a facilities problem and it usually has limited visibility of it because OT got defined down in this narrower way in general historically of oh we know the OT is at this one factory and the definition of people's jobs is not all OT in the company. It was all OT the company knew about and had the central foresight to have them go after and there's now this proliferation and the responsibilities are unclear. So I think it's more that environment is actually what's leading to a lot of problems because we've just had an explosion of things coming online due to de decentralized business purposes and there's not a central rationalized solution for that.

So I think that when I talk with a lot of our customers that look at ot, it tends to be more site specific and kind of prescribed ahead of time and that's great. They actually do a really good job protecting that ot. But there's other systems that the company have that we're perhaps less centrally uh, procured or less centrally understood.

And that's where a lot of the cracks start to occur. It's the things that were not mandated by A C I O or c E O level initiative but are still very real, are plugged into your network and pose very real risk to these organizations. Do we need to go back in and kinda upgrade a lot of these platforms?

'cause one of the things on the OT side of the house is stuff can be around for years and years and years and it really wasn't built with security in mind. So do we need to have sort of a massive upgrade cycle? I believe for me that in most cases there are certain kinds of technologies that organizations do need to buy net new, but in a lot of cases it's actually rationalizing their existing architecture and their investments but actually having it operationally apply to everything that they have.

So for example, what we Jones with our customers is they actually have good OT practices on the books. They actually follow them, they're just not going over all of the assets that they have. And this is true both in kind of the context of attack service management and also in a context such as zero trust of you actually need to have assets and inventory for everything you have.

It's not that your security policies that exist are wrong. It's not that your security controls are wrong, it's that the tools that you are applying them to are not applied over your whole estate because it has grown up historically in decentralized ways and there's not actually an accurate UpToDate repository of everything you have. So what we say is it's actually much more important to invest in asset and inventory and then you can actually apply a lot there because that's how you have confidence that what your network, how your network is actually configured is how you think it's configured.

Without something like that, you'd say, well I know about, you know, 60 70% of my assets and I feel like I have them locked down really well. Great, like that's a good start. Probably close to a hundred percent of your risk is in the 30% that you don't know about and actually don't, are not tracking.

Do you think AI will save us from ourselves? Can we get to the point where we can discover what we have and then b, assess its level of security? I actually think that AI and also automation are some of the few technologies that actually in the short term have favored attackers because it's kind of easy to scale certain kinds of, of more basic attacks.

I think in the medium and long term they favor defenders 'cause they actually allow us to um, very thoroughly search for kind of every backdoor and every way in. So at first what we've seen has been attackers that effectively have used AI automation to say, well previously you had to be a target of choice, right? To go after you, I had to do a lot of detailed research on you, I had to attack you.

Now everyone can be a target of opportunity. So, um, what we see in a number of breaches is that, or sorry, uh, zero days is that usually within hours if not minutes. Um, attackers have actually enumerated every vulnerable system on the internet and uh, are already actively exploiting all of 'em in a variety of cases.

This was true for things like the half ham attack against Microsoft Exchange servers and we see that as automation and then AI for follow on and lateral limit for that has helped attackers. What we're now seeing is that um, AI and the power of data is helping defenders because we actually can now catalog everything. We actually can protect everything and most of the ways that attackers succeed are by line in places defenders aren't booking.

And as soon as you actually know where to look, um, I always love the phrase everything is obvious, wants to know the answer, um, but knowing the answer's rather hard in cyber all the time, I think AI actually allows us to do that in comprehensive ways and we've been really excited to be a part of the Palo Alto networks. Do you think that the time it takes to respond has narrowed considerably? I mean it used to feel like if you were attacked you could think about it in terms of hours and days and are we now down in minutes and seconds and are people kind of prepared to be able to respond at that level of intensity?

I think, uh, the best example I have of this is our own soc, which uh, previously did take days and months to respond a few years ago, or sorry, not months apologize, but did take days to respond to instance and now we are down to um, single digit minutes and often a minute or less to actually respond and there's, uh, the key is there's not one silver bullet there, but I'd say the largest part of that was huge amounts of investment in automation, AI and data to say take, you know, over a billion potential events a day and then filtered that down to under two dozen alerts that actually need to be looked at by an analyst each day. And if you only have 20 things or less a day going off, you can actually investigate them thoroughly and quickly. The problem is when you don't have that and you've not invested both in the technology to do that as well as the domain expertise to know, um, what you can safely or automatically handle what you can avoid looking at you very quickly reach an operational explosion.

So I believe that done well. It's there. I like, we like to dog food our own products first and um, it's exciting for us to show other organizations what is possible.

And it's not that we were inherently the best place immediately. This took a lot of time and effort that we put in and we did it on ourselves first and then that's been reflected back with a great partnership in our products between our SOC and our product teams as well As the attack surface continues to expand, the number of alerts go up and a lot of the alerts are false positive. So how do I kinda narrow that down?

Because it feels like a lot of the cybersecurity folks suffer from fatigue eventually. And of course they'll wind up ignoring the one alert telling them that this is the most lethal attack in history. A lot of argue on that is that we actually need to go from reactive to proactive and in in particular, uh, many things that are deemed false positives when you actually drill down on them, they're not, they're just not such a burning 12 alarm fire at that time that you should actually spend a human in the grand scale of prioritizing everything.

But again, kind of by applying AI and automation argue view is everything that is something that could be considered, you know, a minor hygiene issue actually should be fixed. And the problem right now is that the capacity to fix that is not convergent with the scale and backlog of issues. But in principle, when uh, you know, a lot of places will have 10,000 plus issues backlog, the reality is eventually those should be fixed.

As a practical matter, if you're throwing labor at the problem, you will never solve that. So you have a problem that is this bad and getting worse. However, with automation, what we've been able to see is customers take over 10,000 bad issues and you know, in a matter of months actually take that down to hundreds or less.

And uh, that's kind of our overall philosophy is that it's great to start with something like we need to have alerts, we need to have prioritization, we need to have the ability to at least understand what's going on. But if you don't have the ability to actually fix something, then at the end of the day you are just gonna be contributing to more noise. So you need to both automatically triage things but then also be able to automatically remediate as well because that is actually solving the problem as opposed to piling on both, uh, more signal in some cases also more noise.

But if you actually solve the problem with AI and automation, that's our view of how this ends And think we can all agree there's probably no such thing as perfect security, but we are trying to narrow down our risks. So how do we go about doing that? Because there are so many attack vectors and sometimes I feel like the bad guys laugh at us 'cause we're defending against all these obscured type of attacks when they're using all the basic attacks and see no reason to do anything more elaborate.

So, um, how do we figure out what to protect when I always like to avoid being in broken record, but to me it goes back to asset in inventory. Um, if you don't actually have asset in inventory, how can you ever say that you're actually solving the right problems? Um, you're problems, but do you actually know where all of your risks are?

And I think, again, this happens a lot behind closed doors with trusted partnerships between us and our customers. But if you will ask a lot of people in public and they'll say like, okay, you know, how do you understand risk? You'll get these great answers.

Every CSO will have all these risk reports that will park them up. And then in a safe environment, once you've earned their trust, once you've actually shown that you can deliver value and are here to solve problems, you can say, do you actually have confidence in those results that they're reflective of the true risk reality? And um, I very rarely hear yes there.

And the reason for that is they know that it's not indicative of everything they have. So to me that's the first problem. And there's a lot of basics that people can have just, you know, are you ingesting and not just writing, but also reading log files from say, all of your gateways that you have across your enterprise.

Are they in one place? Are they normalized? Those sorts of basic questions if you actually have an answer to them and you're doing those basics correctly, um, yes, there are more sophisticated attacks that you will need some additional technologies to get through, but you're already gonna be in the top 10% of customers if you have those basics in place.

Um, one, one question in closing that I'd like to ask, uh, various information security professionals is how many routers do you have in your core network and why are you confident in that number? And that's a great question because it actually goes to how do they do as inventory? How do they understand where their blind spots are?

And if you can play that metagame, you actually get very good by doing basics. All right folks, well you heard in here, not only can you not manage what you can't see, you can't secure it either. So it all starts with visibility.

Hey Matt, thanks for being on the show. Thank you so much Michael And back guys in.