Ransomware Running Rampant – Andres Rodriguez, Nasuni

This is Textron TV. Hey guys. Thanks for the throw.

We're here with Andres Rodriguez is the CTO for Nissan. We're talking about data protection ransomware and cybersecurity and all those good things that go along with it Andre. Welcome the show.

Thanks, Mike. Thanks for having me on the show. We've been talking about this stuff forever and it seems like we're not making as much progress as maybe we ought to be because we still have an issues at ransomware people seem to have it difficult time getting a pristine copy of their data back.

So from your perspective, what's at the core of the issue the problem and what do we need to do about it going into the new year? Well in two words recovery times that is the one thing that everyone cares about and no one delivers. If you look at what people are sort of the vendors are promoting.

It's all about, you know, your data is going to be safe. It's gonna be protected. We're gonna be able to detect the attacks or prevent the attacks in some magic Manner and really it's impossible to anticipate and defend against every possible form of attack.

What matters though is once you get hit how quickly you can recover and you can bring your systems back online and that should really be the focus of the discussions and that should be the focus of how things are evaluated and that should be the focus of the strategies for you know, large companies that are dealing with the threat of around somewhere. So time is money as always so are people kind of hitting the stopwatch the minute they discover the attack and then they're trying to figure out well, how long will it take us to recover and if that's too long, will they pay their ransomware then and that's kind of what the mathematical equation is. That's right.

And that is exactly you know, that is the lever that this nefarious attackers are using the reason ransomware works and it's so effective is because no data is actually moved out of your system. The data is encrypted in place, which can be done very quickly and very quietly. And so what happens is at the time when you figure out my files are encrypted and the only way to see them again is to get the keys from the attackers you have to decide okay, if I have to recover the data myself.

How long is that going to take us versus if we just pay the ransomware and you know, that is exactly that time is exactly what the ransomware attackers are using as leverage over their victims. And therefore it is the thing that you want to be able to unwind you want to be able to you know contract the time that it takes to go from as you said if you started watch when you realize you've been attacked and you're trying to assess how much damage there was if you start a clock right there. How long before you have a perfectly pristine back to operations infrastructure.

That's what matters and if that time is measure in hours. You're gonna go with your recovery systems. If the time is measuring weeks or months, you're gonna pay the ransomware.

Do we need to modernize our data protection platforms? Because you know in my experience we've been at this for four or five decades now and we've never really been exceptionally good at it. It's always been a little bit hit or miss.

So do we need to kind of take a minute and say hey rather than just obsessing about what's the next greatest cyber security tool? Let's just figure out what our data protection platform looks like today versus what it needs to look like tomorrow. Absolutely, absolutely.

And you know, if you look at it back up is broken, especially when it comes to backing up large file Footprints, which is where ransomware you know has found its sweet spot, you know, if you're a company that works with a lot of files if you're a company that has a lot of file servers in a lot of locations you are susceptible to ransomware and if you're using best practices and the top, you know Gardener vendors for backup for the protecting those file servers you are going to be susceptible to ransomware because the entire back of model for file servers is broken and it all goes back to this question of time in order to recover a file server. You have to be able to copy the data from your backup server back to your file servers to make that data operational again, that could be operation for files can take a very long time. You know, it tends to be very Quick if you're talking about databases or if you're talking about applications because those are small Footprints, but when you talk about files, you're typically, you know in the order of tens to hundreds of terabytes.

And so that copy operation takes a really long time in if that's happening across many locations across your infrastructure. It's even worse because now you're trying to draw that data back out to those sites where those file servers are sitting what you have to do is you have to avoid the copying and you know, we've been doing this in the cloud for years now, but it's still, you know, a technology that's being adopted in the traditional kind of Enterprise. I team mentality, which is don't back up your file servers rely on the fact that because of the scale of the cloud you can have an immutable version stream or your file server history and you get on any point unwind history, you know file server back to a healthy, you know.

Time before the ransomware attack that is and not have to do any copy. It's all within the file system. It's not like you're trying to bring the files from some backup server back into the production systems.

The production system is able to self heal and to do so internally without having to do any copy of data operations, which is what slows everything down. Is also part of the challenge that the data types are much different these days is a lot more of them and they tend to be a little bit larger and I don't know if people are really thought through that before they started employing all these data types when they didn't look at the protection system. Absolutely.

I think two things made ransomware. I mean like you said at the beginning of the interview, it's not only that we're still dealing with this it's that it's gotten worse and worse and the the two things that I think made that happen was, you know, the file servers got much bigger because what you said that files got bigger and they're more files right the entire workflow of companies. Now, if you're an engineering company and factoring company legal firms It's All Digital and that's all files and all those files are going on file servers and you know that makes the file servers really really large.

Global companies again have many many sites around the world which compounds the problem. So the the footprint has gone, you know, they call the sort of our tax service is grown significantly, so there's more to attack and there was before Couple that with cryptocurrencies, which I don't know. They're gonna change the world of Finance, you know, given the time when you and I are having this interview, it looks like there's a lot of exploding and bloating cryptocurrency.

But one thing crypto has been very very good for is ransomware attacks until you know, the the introduction of Bitcoin there really wasn't a good way of monetizing ransomware attacks. Now there is and you know, there are sort of governments around the world who are nefarious as the attackers and they are uses kind of laundry mats or digital or cryptocurrency to go into real money and because of that there's actually organized crime making hundreds of millions and billions of dollars on ransomware attacking systematic ways. And so the question has stopped being whether you're gonna get attack by ran somewhere.

The question is when is it going to happen if your company has any any level? Of public, you know profile you will be attacked in time. And I think the difference between being attacked now and being attacked five years ago.

It's five years ago. You could you could beg and plead to your board and your CEO and tell them look we we were following best practices. We had the best vendors.

We read all the Garner reports, etc. Etc. Now that's not good enough.

If you don't have a recovery plan and your your company is down on its knees for weeks or it has to pay ransomware, which is not only the money going out the door, but it's the potential damage to your brand your firm if people find out about it both because You have lack security processes because you weren't able to recover and because you know, you're basically paying terrorists pain, you know bad guys for something they've done to you. So, you know, if you're if you're a brand this is not something that you want happening to your infrastructure. And so I think the excuse of sorry backup takes a long time.

That's all we can do. It's not good enough that that's gonna get you very quickly into your next job and it's likely not to be a move up. It's like to be a move in the downward Direction.

So I think everyone should be looking at this new systems that don't depend on back up. backup is broken the back of vendors don't want to tell you that the analysts that cover backup don't want to tell you that but they ran somewhere attackers are sure we're gonna make you aware of that and you're gonna have a really hard time explaining that you are reading the latest reports following best practices when you're down for weeks or when you have to pay ransomware, so, Are you seeing cybersecurity people get more involved in the data protection conversation used to be you know, the lowest person on the it totem pole was in charge and now maybe the conversation has been elevated someone. Definitely in both.

I can tell you the Cyber Insurance practices, which most companies have now demand that the conversation be elevated. So, you know cyber security is having cyber resilience, which is the correct term for this because it's it's not to say that detection and prevention are not important parts of the equation, but when it comes to files and when it comes to ransomware and files recovery time is this thing almost important variable in India equation that discussion is gone from being sort of a discussion that was happening. Like you said in the low levels of kind of it the backup world and now it's all the way up to at least the CIO level then the compliance officers the chief security officer, you know, there's awareness all the way.

I would I would argue that even at the board level. Once you start getting cyber insurance, which I would recommend anyone to you know, they're gonna come in and evaluate what your best practices are before they stand the insurance. I'll tell you a great story Mike.

We had we had a bunch of clients, you know, we do interviews on ransomware and you know, it's it's part of how we develop a roadmap. So we talk a lot to our customers about it. And one of my favorite stories was a one of our clients was actually head.

Before they had masuni and couldn't recover. I'm sorry and sort of they couldn't recover but the important thing is they were hit and then the ransomware that was asked was exactly their insurance policy. And what had happened is the attackers that actually compromise the insurance company gotten the list of clients a list of customers from them and their policies and were systematically attacking those companies and asking for the policy, you know amount for each of these customers.

So this is how sophisticated this attackers are. You know, these are well thought out you know, multi-month multi-year plan attacks and when they happen you you just have to be armed you have to be ready to do this fast recovery. You can't just be at the mercy of being the next company in the list that their systematically attacking and asking money from do you think they've also gotten better at targeting the backup files themselves and part of our issue is that when we go to recover we discover that the malware is everywhere.

So do we need to rethink that as well? You know, I think that I think the problem is the backup process itself backup is a bad idea, especially when it comes to files. I mean the whole idea of the backup premise was the primary system is not good enough to keep itself protected.

It's a bad idea. It's kind of like saying, you know, my door is not a very good door. So I'm gonna give you another door that actually a very good door.

No, you just need to fortify your door the primary door and that's the that's the issue. That's a, you know, people feel more comfortable because now they have two systems two systems is two vectors of attack. Like you said, I can attack the backups.

I can attack the primary and most importantly even if the backup is not compromised. It takes so long to get the data from the backup to the primary that is inconsequential the backup could be pristine but it's gonna take you weeks to recover this very large file servers or these many around the world file servers. I mean, we we've had clients that have that to Stage multi-month recovery operations where you have to triage which offices are going to come back online first based on their importance and you can bet that those offices that are sitting at the bottom of the triage.

They're really angry with their it Department by the time it's you know, the second month and they still are not back in production. All right. So you mentioned building a Better Door.

What exactly does that look like? And how hard is it to create that door? It's one door one system.

So you can keep all your eyes on that one system and that system has all the Telemetry you need for doing early detection for being able to most importantly every change in data that is made to that file system is actually kept in an audit Trail an immutable audit Trail and every single change to the files. It's actually also preserved in an immutable version stream of the files. And so when an attack comes in and you find out so first of all, if you can detect it early on awesome, you can prevent it from having any effect on your file systems.

But if you miss that, it's okay because now you have a perfect Trail. The other trail of every single file that was encrypted in your infrastructure and now you can automatically tell the system the one door. Unwind all of this encrypted files to just before they were encrypted give me the latest version before the attack and all of that happened in an instant.

These are operations that if you have a version file system, you can perform in a matter of minutes versus taking weeks or months to do the recovery operation again, the focus is on that mean recovery time. That's a single most important variable how long after I say whoops we've been attacked does it take me to go from a compromised system to a system that is now back in production and all the files are visible and all the end users and applications can look at the files and see the file system as it was before the attack. All right.

So a lot of folks are probably going don't we have that capability with operating system. So what was the hard part about doing it at the file system level? It's bigger, but you are exactly right operating systems have been able to do versioning and be able to unwind in in very grateful ways for for upgrade ability, you know reasons it makes that much safer because if the upgrade fails you can you can wind it back, you know, operating systems are very controlling environment operating systems are very small environment.

File servers across a large organization is a very big system scale, you know, I've been doing files my whole career and that this the difference between files and unstructured data in companies and everything else. It's scale. It's the scale of the problem, you know every other system in a company when when every other system in a company is something that people are talking gigabytes.

When you talk too far about file systems, you're talking terabytes when you're going terabytes on the other side, you're typically talking, you know petabytes on the file system side. And so that you have to be able to create a system that can do that version and that recovery at staggering levels of scale. And that is what the cloud file systems are all about.

It's basically about handling not only the capacity of the file system, you know in a in a cost-effective manner but the versioning of the past system the recovery of the file systems. I mean, you know for most of our clients their fastest into our too big to back up it's kind of like the banks too big to fail their chance to large and by that, I mean that the windows for backup start getting too long imagine what the recovery times over system like that would be on their just the the backup model where I have to grab, you know, 500 terrified. My backup server and bring those back to the file server anyone in it will tell you that that is an insanely long operation.

That is the sort of operation that you fear to do and that you typically get a partner and it's a multi-month operation to do so you want to avoid that, you know because you get attacked by ransomware, you do not want to be copying, you know hundreds of terabytes from your backup to your file servers. That is you're gonna be paying the ransomware because you know, if you look at the ransomware, it does exactly what a version file system does. Once you have the keys you can just unwind the encryption without having to copy data.

That's why it's so fast in the recovery. But if you have a version file system, you can do that recovery without having the encryption Keys. You can just go back to the previous unencrypted version without having to copy data.

All right, being data is death or scale. All right, folks. You heard it here.

If you're reached your backup system as you're ransomware recovery strategy, perhaps it's far too late. And it's just a question of how screwed you really might be. Hey Andre.

Thanks being on the show. Thanks, Mike. All right back to you guys in the studio.