The Growing Importance of Software Escrow with Wayne Scott

This is Textron tv. Hey guys, thanks for the throwaway here with Wayne Scott, who is the regulatory compliance solutions lead for S Code, and they are part of the NCC group, and we're talking about a thing called Software Escrow that's starting to come in Vogue, or maybe it's just been around and maybe it's coming more popular. We'll see.

But Wayne, welcome to the show. Hello there, Mike. How are you doing?

I'm doing great. What exactly is Software escrow for the uninitiated and, and what's driving suddenly this topic and conversation? Well, I mean, uh, uh, software escrow or cloud escrow or technology escrow, it, it's been around for a long time.

It, uh, was sort of bit like nylon was dual invented simultaneously between, uh, uh, the US and the uk. Uh, the, the initial instances of it sort of pop up around 1984. Um, and the, the whole concept behind it is there's a load of information behind business critical software applications that is important is, is necessary, but that you don't necessarily get access to it as part of your standard license arrangement.

So, you know, it could be source code or access credentials or build manuals or shopping lists for hardware or anything like that. Now, should the software supplier fail in any way, shape, or form, um, then you're gonna need access to that information to be able to carry on using that service. That was sort of the original, uh, version of it.

The original, uh, ethos Behind is, is really sort of a, uh, operational resilience, business continuity, sort of crisis management element to, uh, ensure that should somebody fail, uh, you, you can carry on, uh, carry on carrying on if you like Mike. Um, so it, it kind of comes in three stages. Uh, first of all, what you do is you establish a legal right to access all that information, and that's via an escrow agreement.

Um, and then you do, uh, a knowledge transfer. You know, you need to understand how to use that information. Uh, and the best people to tell you how to, you know, use that information is the people who've built that software.

So we get 'em to document the build process, uh, uh, rebuild the working application from the source code. And then the final stages of that, uh, process really is a scenario test understanding could the end user all the management of that service in-house, or pass the management of that service to another supplier? Um, with nothing really new here, as I said, it's been around since 1984.

Uh, escrow, uh, has been working here and the uk, uh, and the US for a, a very long time. It works. The vast majority of critical suppliers or material suppliers, uh, are very, very familiar with it, as are, uh, a large percentage of the, uh, financial services community.

The, the, the reason why it's becoming more popular, more prevalent, uh, uh, Deger, if you like, is that, uh, um, the, the, the, the world is becoming, uh, more aware of the risks presented by the supply chain to their end users. Uh, in the uk, Canada, us, Europe, uh, there's been recent developments around regulatory requires to mitigate those risks, say, uh, in the uk, uh, things like supply failure service deterioration, concentration risks are now named risks. So when, when a a risk is named, there has to be a mitigation strategy.

Uh, and that strategy has to be testable and, and ideally it has to work. Uh, and that's where escrow is named in uk, US Regulations, uh, uh, Hong Kong, Singapore, Indonesia, Philippines, New Zealand, and, and the places where it's not named, say, Switzerland, Canada, um, uh, Europe, through the, uh, digital Operational Resilience Act, a a scenario tested escrow, a great agreement will exceed the requirements of the regulations. Could be, you think about it, if they fail, what happens is you're getting given all the information so you can carry on using that service.

It's, it's, uh, fully workable. Like, Are regulators becoming more aware the need for this, and are they starting to pass or come up with more stringent requirements? Definitely.

So, yeah, definitely. So, you know, if you say you look at, uh, uh, Michael Grim Wade's work around, uh, the 10 rules of Operational Resilience, there's been a, uh, a direct correlation between, uh, financial shock and operational failure. Uh, and the bigger the shock, the bigger the uh, effect there is, uh, on an operational risk.

Uh, and that includes the technology supply chain as well. And, and, uh, if you look at what we've had globally, maybe over the last four or five years, are some of the biggest economic shocks in quick succession in the history of economics. So, uh, COVID, we have Brexit here in the uk.

We've had high inflation, we have lockdowns, we've got a couple of wars going on. We've had bank failures, we've had significant software supply failures too. Takes around about eight or nine years for these fully, uh, risks to mature and get recognized as kind of a lag in detection.

But these rapid quick succession risks and threats taking apart effect, uh, you know, if, if contagion is possible within a financial services industry, it's not too much of a leap and a jump to say that it's also possible within, uh, the technology supply chain. Uh, and also just take a look as well, you know, uh, uh, technical, say AI as as an example. You know, AI has the ability to be highly disruptive.

And when I say disruptive, I don't mean, hey, we've got a disruptive technology. Like everybody who sells technology says the proper disruption. Real disruption always causes obsolescence and obsolescence, uh, uh, is isn't something that you can really have, uh, when it comes to the technology supply chain.

Uh, if you look at, say, fintechs, uh, the financial technology companies, um, uh, uh, what's the main way in which we judged our success? It's their ability to scale and their ability to deliver return on investment. Now, what happens when a, a, a piece of AI comes along, uh, and, uh, makes a, a core banking service obsolete?

Are they workers gonna wanna work there? Are they going to, uh, want to, uh, what happens to the share price? Are they gonna be able to, uh, carry on servicing, uh, that industry and, and, and also the financial institutions themselves?

Uh, they're gonna wanna move away from that service. Of course they are, because this new piece of technology is gonna provide, I dunno, cost saving and efficiencies, our accuracy, uh, the improvements. Um, so, but that need to carry on using that service.

Uh, uh, and, and if the company no longer exists, uh, well be good. The ability to, to, to pull the management of that service in house. We don't know where the disruption is gonna come from when it comes, uh, to artificial intelligence.

You know, we could maybe look at, say, legal services. That's pretty predictable. There's gonna be some huge disruption there, but you can't predict where it's gonna come from.

Really what you've gotta do is you've gotta assume it's gonna happen and build out some necessary controls, uh, to limit the damage that it's going to cause. Is this becoming a bigger issue? Because in the Covid era, we seem to rely more on SaaS applications that are provided by relatively small companies often, and they are perhaps not as financially secure as some of the larger vendors historically have been.

So is it possible that they're gonna go out of business more likely, and we'd wanna be able to have access to that software if we're overly dependent upon it through some sort of escrow review? I mean, yes, it's highly applicable to SaaS services. Uh, it's that they're definitely a value there.

What, what, you know, what a lot of the public speaking that I do is to try and move people away from this, this idea that only small companies present risks. You know, there are concentration risks presented by all, uh, uh, all suppliers. I if, if you think, say, uh, uh, a bank may outsource, uh, 1% of its functions to a, a, a tech company at FinTech, that FinTech may provide that 1% of services across 10% of the market.

You know, that is a, a, a single point of failure, uh, and, and a clear and present, uh, concentration risk if, if you look, say in the UK and Europe and Hong Kong with Singapore, uh, the, uh, the, the, the requirement to build stressed exit plans, plans to cope with the supply of failure isn't limited just to risky suppliers. It's also expanded to all critical services, all material services. Uh, so, um, it's not just a matter of looking at, at, uh, SaaS services from, uh, uh, small companies, but all big companies are in scope as well, particularly in the UK and Europe.

Um, we've got a certain set of regulations called the critical third Parties regulations in Europe. It's called, uh, the Digital Operational Resilience Act. And those regulations will see, uh, uh, for the very first time, some of the world's largest tech companies being brought under direct regulation of the financial services regulators.

Some time off that, uh, away from that in the us uh, I was certainly over in New York recently at, at a conference called CefPro, a collection of, a gathering of some of the world's, uh, uh, uh, third party risk management experts, uh, and talking. And, and the regulators from the US were there, and they were expressing admiration for the UK and European regulations and, and gave some pretty strong hints that the, uh, these regulations or some kind of these regulations will hit the US at some point. So it, it's, it's not a, if, if something is critical, you should really protect it.

It shouldn't really matter on, uh, the, the, the size of the supplier. But, you know, a, an an escrow solution for a cloud service, a SaaS service is slightly different. You know, just having the source code is pointless.

Uh, certainly what we would look to do, say for a single tenanted solution is take the access credentials to the environment, pretty low tech solution. Uh, it always helps if you can just swap the credit card details on there, uh, and keep the service running. Uh, and multi-tenant solutions.

A little bit more complicated, but just as easy, obviously we still take the source code and everything like that, but we take, as the supplier concern to produce a fresh one down instance. We'll store that in a separate cloud account, all the, all the associated, uh, client data too, Historically. And it's one of the dirty little secrets of our industry is that we're not very good at documentation.

Um, do you think that, uh, in the age of AI though, especially generative ai, we might get better at documentation 'cause we can rely more on the machines and there'll be fewer excuses for not having the documentation required for the escrow? Uh, uh, having run my own FinTech, I could tell you yes, we're terrible at documentation. Yeah, we're terrible at it.

'cause it's boring, isn't it? Nobody likes to go back and document why they've done something. But, uh, think, think about that.

You know, you are, you are a, uh, uh, third party, head of third party risk management for a, a gsib, a globally systemically important bike, and some source code lands on your desk one day and it's like, Hey, John, can you, uh, manage this service for going forwards? And you open it up and there's no notes there. You know, we, we know there's a risk there.

We, uh, and that's a lot of it, uh, here, why the regulations are coming, uh, uh, are, are, and certainly be put together is everybody, there's no standardized mechanism for some developing or deploying code anywhere along the way. Uh, and I, I would never advocate one, but, hey, let's document it. You know, it, it, it's about, uh, you know, I talk about something called resilience by design.

It's not my term. It's been around as a concept by, uh, for quite some time. But, you know, the idea is build a service with your own failure in mind.

Thinking, can this be replicated? Can this be, uh, uh, done by somebody else? Uh, it's kind of best practice really.

If, even if you own a software company, you don't want the knowledge all city want with one member, one member, your staff. It would be idea ideal if other meds could, if the staff could actually repeat it. So a lot of the sort of knowledge transfers we do here, uh, are extraordinarily useful to the, the software supply themselves.

Um, AI could improve it definitely could improve it. Uh, we wouldn't necessarily look to be putting AI source code under an escrow agreement itself, though. We would now be looking at the ramifications of, of the AI as, as I mentioned earlier, around rapid obsolescence within the supply chain.

What challenges does open source present on this concept? Because sometimes, you know, I can get access to the source code, but there are so many forks and so many things they're added to software that it's not always clear to me what the providence of what is and where it came from. So, um, how do I incorporate that into my escrow adventure?

Well, I suppose it is a question. We, we, we do get asked, but I mean, uh, open source shouldn't, yeah. Anything that's deposited in underneath an escrow agreement needs to be owned by somebody.

Uh, there has to be an intellectual property rights owner, so it isn't really applicable, uh, to an escrow agreement. In theory, everybody should kind of a, have access to that anyway. It should be available on the open market.

I mean, the security concerns with open source as, as, as most probably alluding to there, but it's, it's not something that would really be in scope for espro. And yeah, if I go looking for the documentation for that stuff, I'll just get a note that says, you know, we use this module from this open source repository and send me a link to the repository. And the repository may or may not have any kind of documentation to go with it, but so yeah, E exactly.

And, and, and you kind of, in the US you're kind of tackling that through the sort of the software bill of materials work that's going on around there to be able to document it, kind of attacking it from the bottom up and making sure you've got a list of everything here. Whereas, uh, uh, uk uh, Europe attacking it from the top and basically documenting all the services that are used, uh, uh, assigning criticality ratings to their understanding the interconnected nature of them, then building these contingency plans and stressed exit plans, then that, that information is captured as part of the stressed exit plans building. Whereas in, in the US Canada saying, wh what is used?

Where is it? How do we get hold of it? Uh, I think that will, it's, it's highly complimentary.

It's just a different way of doing it. I think then when you throw in the regulatory change at the top end of it as well, um, uh, you'll end up with a more all encompassing sort of risk mitigation strategy. So what goes wrong with software escrow most often?

What should people be looking out for? What's the the kind of thing that, you know, people learn the hard way? That's a really good question.

I I, you know, I, one of the main things that goes wrong, and it's not necessarily, uh, a bad thing, but in this instance is, is that, that, that the, the procurement of software escrow in the US is kind left to the, uh, the software owner. They're in charge of the process, the intellectual property rights owner, the supplier, if you like. Um, and, um, what that can mean is that internally within the end user, it's not necessarily somebody that manages that process and an escrow agreement, an escrow solution's only as good as the information that's deposited underneath it.

So it always needs some money to own that process, even if somebody doesn't pay for it within the end user. You always need somebody to own that process, to check the versions, to ensure that what is deposited is up to date and is correct. Uh, and to really, uh, to ask the escrow provider to check, there have been no, uh, changes in the bill process, no, uh, updates in the hardware that's involved and to, to repeat those scenario tests at a reasonable level of frequency.

Uh, sometimes a lot of these things, uh, get captured in the procurement process at the very start through diligent procurement officers, um, but then are not necessarily passed over to somebody, uh, in maybe continuous monitoring or audit, uh, because it's not hitting anybody's radar. 'cause nobody on the, uh, end user side is actually paying anything for it. Certainly what we did, we did an exercise for, uh, uh, uh, one of the largest blank banks on the planet.

Uh, uh, a g said very recently, and they firmly said to us that, uh, they had enough escrow agreements in place, all their suppliers have to do a standard. Um, when we looked at this, uh, how many agreements were in place, that there was around maybe 15, 20% of the agreements that should have been placed. So procurement and contracts were doing their points and saying, these agreements need to be in place, but then no one was actually following them up to see if they're actually being established of the agreements that were in place when we checked them.

We should be able to rebuild everything that's underneath that, those agreements. That's the contractual requirement. All the information should be there.

So we took, took the top five most important suppliers on that supplier list and attempted to rebuild all of the software that was still over there. And these are five of the largest providers on the planet tech companies on the planet. Um, all five of them failed.

All the info and the information was incomplete. Uh, we've then gone back and become more diligent, set up and rewritten policy standards and procedures for the financial institution's concern to capture that information going forward. But to answer your question, in the most briefer fashion, uh, people just don't check that the information is true and correct.

I know I keep pace with the current innovation cycles because people are using Agile and DevOps flows to update their applications sometimes multiple times a week at least, maybe multiple times a month. And it's not clear to me that the repository that is in the escrow keeps track with that. Now, how do I kind of align those two things?

It's a really good question. It's a really good question because, you know, uh, uh, continuous development, continuous improvement, uh, it's not the opinion of NCC group, but you know, we used to call that unfinished. I'm really joking.

But you know, there are, there is risk there isn't there? There's, there's certainly risk there. You've gotta keep track of that process and, uh, and ensure the information's, uh, uh, definitely kept up to, kept up to date.

Uh, uh, first of all, you've gotta kind of attack it contractually. You've gotta put a contractual, uh, uh, obligation on the supply concern to keep moving forward with this process. Uh, if it's stored in a, you know, ub any sign of, uh, code repository, we could just set up some kind of link to pull it as frequently as necessary, usually 'cause it was deemed necessary.

Uh, we'd still advise that you do test that somewhere along the way and establish if there's been any changes in build processes. But, you know, those mechanisms that you're talking about there, those, the build, uh, agile and, and et cetera, um, you know, they do present their risks to resilience that you need to ensure that whatever it is they're doing, your suppliers, they're doing it in a fashion that's repeatable. All right, folks, you heard it here.

And the more dependent we come on software, the more we need to think about it as a business asset. 'cause otherwise, I wake up one morning and discover that, hey, the software that we're wholly dependent about that longer exists. Hey, Wayne, thanks for being on the show.

Nice to meet you, Mike. Thank you. All right, back to you guys in the studio.