Caroline Wong – The Practicalities of Pentesting at Scale

you Hi, my name is Caroline Wong. I'm the chief strategy officer at kovalt. I am thrilled to be with you here today at Cloud container Summit 2022.

I want to tell you about when I started my cybersecurity career more than 15 years ago what it was like to be leading information security teams at eBay and Zynga. These were super cool places to be working in cybersecurity and in both cases. We were running online operations 24 by 7 with millions of simultaneous users daily.

94% and is one of the first major electronic Commerce shops enabled strangers to transact with each other over the internet. Zing was growing incredibly rapidly as an early adopter of Amazon AWS and in 2009 the Zynga game Farmville launched and in just a few weeks the game went from zero to 10 million daily active users a few months later it Rose to 80 million daily active users. And Cobalt, we build security software and we do pen testing as a service.

I want to start this talk by telling you about the first time that I realized that most organizations do not do enough manual pen testing. Between 2013 and 2016. I live more than three dozen beesim assessments.

Be some stands for building Security in maturity model and it's a descriptive framework for real life software security. There are 128 firms in the latest besom data pool. This data pool can be thought of as somewhat representative of all the companies in the world that operate using software, but with one pretty big caveat.

It costs five figures to do a visim. If you can spend that much money paying a Management Consultant to interview software development teams and give you a scorecard on how well you're doing software security. You're probably relatively mature when it comes to software security.

So let's go ahead and assume that the be some data pool is above average compared to the rest of the world. I learned something really startling when I was doing this work flying around the world talking to teams about their software development practices. I found out that a typical Enterprise with 1,000 software applications was only pen testing about 100 of those applications.

Just 10% of most Enterprise application portfolios were getting manual pen testing that was sort of terrifying. There are a lot of reasons for this historically pen. Testing has been expensive and slow.

It's been hard to get access to Quality Talent whether you're building an in-house team or working with a third party provider. I do have good news to share with you today last year in the Cobalt 2021 state of pen testing report. We found out that organizations are doing more pen testing.

We talked to more than 600 International Security folks and found out that today on average folks are pentesting 63% of their application portfolios. This is a huge increase and I'm here to tell you why how and I'm also excited about promoting the idea that we could actually increase that number even firmware even further. I firmly believe it's the right thing to do.

Because if you're responsible for a hundred houseplants, it would be unacceptable to only water 10 of them. If you're in charge of caring for 100 cats, it would be unacceptable to provide food and shelter for only ten of them. Last year in 2021 our industry got a new owasp top 10.

The first version of the top 10 came out in 2003 more than 17 years ago. Can I do math? 17 20 years ago like 20 years ago when I put our latest OS top 10 next to the original owasp top 10, these two are alarmingly similar.

What this indicates is that the most common vulnerabilities found in web applications today are not so different from what they were nearly two decades ago. For fun. Let's talk about how long ago this was in 2003.

When the first OST top 10 came out Matthew McConaughey started in a movie called How to Lose a Guy in 10 Days. And in 2021 when we got our most recent owasp top 10. Which looks startlingly like the first version that we got in 2003 Matthew McConaughey played a singing koala in the movie sing, too.

It's been nearly two decades. I want to talk through a brief history lesson looking at the past decade or so in the cyber security industry and in software development. I want to present a maturity model for pen testing and I want to provide my advice for how to scale pen testing and achieve the five ideals that are written about in the Unicorn project.

So why is it that things have not changed? Why aren't they getting way best way better way faster? because I think that we know what to do the owasp top 10 and the supporting guidance provides information on exactly how to find fix and present prevent find fix and prevent the most common vulnerability types.

At risk of over making this point this information has stayed fairly consistent for nearly two decades. I think the reason that we're not doing it more is because it's just not particularly fun or cheap. Or convenient to do kind of like going to the dentist.

so In our industry for cybersecurity folks. We love our best practices our standards our Frameworks the graphic that you see here on the left is the table of contents only the table of contents for nist. 853 a nearly 500 Page Long document.

I think sometimes we do ourselves a disservice with these extremely lengthy documents. I think sometimes this makes cybersecurity actually seem a way more complicated than it actually is I actually believe that cybersecurity can be simple. Simple does not necessarily mean easy, but it does mean that it can be easier to understand.

On the right side, you see a simple model just four things that you need to do govern find security problems fix security problems prevent security problems. We as an industry have an opportunity to focus on the things that matter and to do them well and to do them often, Let's take a step back. What is the point of cybersecurity?

We're trying to manage risk security is about protecting value and so much of what we value today has shifted from the physical to the digital Realm. Value protection is about risk management. This list says we as an organization need to manage risk because we care about what other people think.

The second list says we need to manage risk because we actually want to build and maintain and operate software that is resilient to malicious attack. We want to secure our software. Unfortunately, we are not getting this right.

My former manager and very good friends. Sammy migas is a brilliant person and he says that historically the industry has not gotten risk management right budget driven risk management is simply not good enough if we are only able to cover 10% of a software portfolio with appropriate security testing. That's not good enough.

Every time we don't do risk management. Properly meaning appropriate security testing for the assets based on a risk analysis then each time. We're gambling.

And sometimes it turns out not the way we want it to. Let's do a fun and brief history lesson cybersecurity a decade in review. Oh wait.

Let's talk about cybersecurity two decades in review. So in 1998, there was this group of young hackers that went to Washington with a warning for Congress. Software and computer networks are insecure.

during that now Infamous hearing one of the hackers said Any of the seven individual seated before you could take down the internet and just half an hour? And in 2018 20 years later the same hackers offered a similarly Bleak assessment. Digital security is hardly any better.

Joe Grant who went by the hacker named Kingpin in his Loft days said it Loft. We tried to be the voice of reason and raising awareness or problems nearly. All of what we said 20 years ago still holds true.

Yes, there have been improvements but the general class of problems is the same. Back then said Chris my sopal the threat was the teenage hacker now. It's nation states.

So every vulnerability got a lot more risky. 20 years ago why soap will continued the threat of nation states and foreign governments with skilled hackers seemed so theoretical. But we all know that 20 years later.

This is happening constantly. 20 years ago. We got our first off top 10 today looks kind of the same.

last year I read a book. and it's called 4,000 weeks and it basically says look the average human lifespan is only 4000 weeks long and if that's the case, how do you want to spend your precious time? this year I crossed the 2,000 week Mark.

I've got 2,000 weeks left. And in my 2000 weeks, I want to see some change in the cyber security industry. So how do we make change happen?

How do we fundamentally change the cybersecurity industry in order to get to a point where we are making meaningful progress? We in the cybersecurity industry have an opportunity to look at our friends and colleagues in software development. Because whereas the state of cybersecurity has remained somewhat stagnant for the last 20 years software development has made tremendous strides in the last 10 years.

In 2011 puppet released the first state of devops report and by 2013 the state of devops report had established a relationship between a devops practice and high performance outcomes. Organizations that do demops say they do more frequent deployments have shorter lead time to change lower change failure rates and faster mean time to recovery. These results have a direct impact on business outcomes organizations that can deploy on demand and have shorter lead time to create change have comparative advantages by delivering solutions to their customers faster.

Shorter mean time to recovery and lower change failure rates mean they have more stable systems. So if companies doing devops have better outcomes, the naturally we want to know what they're doing and the state of devops report provides some prescriptive advice about what kinds of things work best in today's environment. So from the state of devops 2021 report.

Security campaign afterthought or the final step before delivery. It must be integrated throughout the software development process to securely deliver software soft security practices must evolve faster than the techniques used by malicious actors during the 2020 solarwinds and codecuffs software supply chain attacks hackers covertly embedded themselves in the infrastructure of thousands of customers of those companies. Given the widespread impact of these attacks the industry must shift from a preventive to a diagnostic approach where software teams should assume that their systems are already compromised and build security into their supply chain.

It is easy to emphasize the importance of security and say teams should prioritize it but doing so requires several changes from traditional information security methods. What does state of devops 2021 say about security? Here's their advice number one conduct a Security review for all major features.

Number two invite infosec early and often number three use high quality documentation. so if I'm in cybersecurity And I'm observing. That the industry is stagnant and then I look over at software development and I observe that things are growing and getting better and improving really quickly.

What could we take? As inspiration and try to incorporate into cybersecurity. SAS Cloud SAS companies have fundamentally transformed enterprise software at this point.

There is no going back. SAS companies don't necessarily have all the answers, but they are inherently flexible and they can iterate according to Market feedback and business needs. It's all about speed of innovation design and usability.

The faster. You can go the less you spend on product development and the fewer person hours are required to deliver a complete solution. Every iteration is an opportunity to deliver greater business value.

So what if we took SAS? and we applied it to security testing at Cobalt we have taken Historical traditional pen testing and now we call it tasks. So I want to present this pen test maturity model and ask you to take a look at it and ask yourself.

How would you describe where your organization is along this maturity model. I also want to encourage you to look up the 2022 state of pen testing because we've got a lot of really amazing data-driven information there for you to peruse that information is at no cost and it is Data driven. It is informed by thousands of pen tests, which were conducted in 2021.

If you want to go from ad hoc or structure to strategic, here's my advice on how to do so. First things first do it faster and more often. When do you need a pen test?

Usually it's because of one or more of a handful of reasons. Number one you have a customer who insists on seeing a pen test report before they commit to buying your product. Number two, you have a regulator who insists on seeing a penis report as part of a Regulatory Compliance exercise number three, you're getting Acquired and the company that's buying yours wants to see a pen test before the complete the transaction and add your risk profile to theirs or number four.

You've just deployed a new feature or an entirely new product and you want to find and fix security vulnerabilities before they can be exploited by malicious actors. In any case one way to get strategic with scaling your pentist program is simply to start faster. I mean when you've decided what you need to pen test get the technical experts performing manual pentastic activities right away, like within 24 hours the sooner you start the sooner you have the finalized pen test report in the hands of the person who is demanding it.

It also means that you're that much closer to knowing about exploitable vulnerabilities in your software and the sooner, you know about them the sooner you can fix them. And the sooner you can start your next pen test because the product teams are just going to keep building new things. And if those new things are not getting proper security testing then they're probably vulnerable.

One super silly thing that I've observed throughout my 17 plus year career working in cybersecurity is that a lot of compliance regimes and best practices Frameworks will recommend that you do some kind of defect Discovery like pen testing or scanning for vulnerabilities. But then they don't actually require that you fix what you find. So when I talk about scaling pen testing, I don't just mean printing a PDF handing it over to a regulator checking the box and moving on with the rest of your day.

I mean finding security vulnerabilities in your applications and fixing them. Now don't get me wrong fixing vulnerabilities is hard. Most of the time if you're a security professional fixing vulnerabilities actually involves getting someone else to do something and getting other people to do stuff is hard.

I would like to tell you about a way that doesn't usually work. If I get a 50 page PDF in my email and I forward it to a bunch of engineering managers and I say here are the results from our pentest that we did. Please work with your teams to get the findings from mediated then maybe.

The other person opens the email gets really stressed out and the security findings still get fixed. It might be better. If as soon as a vulnerability finding is discovered, the engineering team finds out through slack and then goes into jira where ticket has been submitted that can be worked right away.

It would be cool. If there were a direct line of communication between the engineer and the pen tester who found the issue in the first place, so they could ask questions and give advice and work together to get security vulnerabilities addressed. I have been obsessed with security data and metrics since the start of my career.

I believe that better data enables us to make better decisions and Achieve better outcomes. So with this late stage in the cybersecurity green in the cybersecurity game, let us remember that hackers testified before the US Senate in 1998. We should have a ton of pen test data.

And in theory that could help us out a lot. But it takes grunty time-consuming effort to physically parse through piles of customized Consulting reports to get actual pen test data and format it in such a way that it can be collected and analyzed over time. I have known consulting firms who tried to do this because the data would be very interesting, but it was simply too much work.

There's a better way. If pentastata can be standardized in a format that can be organized and accessed through an API, then you can actually analyze the data you could actually use it for years. I've heard people say yeah, they all step 10 is great.

But what's really important is my organization's top 10 or top five or top three. but I don't actually know that many organizations who have their security metrics to a maturity level where they even know what they're top three classes of security vulnerabilities are now there are some companies that do have super sweet pen test management and data and dashboards. Everything's custom built and very specific to that Organization for this type of company an API to transfer pentas data from one platform to another can be super helpful.

This is a picture of my hard copy book of the Unicorn project. And I really think that cybersecurity has an opportunity. To think about the way that we work with these five ideals in mind.

If you're interested in learning more about PTS. I wrote a book. It is called the pitas book get the book read the book.

The book will tell you exactly what you need to do in order to achieve pen testing at scale. But spoiler alert. This book is not going to do the work for you.

It's kind of like a 12-step program. If you follow them your life is gonna be a lot better just do it, but not everyone chooses to do it. I want to talk to you about an analogy for the cyber security industry.

I think that some people think that security is like a vitamin or a Band-Aid. Some people think that it's something you can inject or do at the last minute or add on after the fact some people think that security is a feature. I think that security has always been the result of decisions and actions made by many different people.

It's actually the outcome of an unpredictable dance between many different people and so for SecOps to be successful. We've got to build a collaborative approach that brings us together. When I'm with six years old, the first ever ransomware attack directed users to mail a hundred and eighty nine dollars to a PO address in Panama.

Last year in the first half of 2021 the average ransomware payment climbed to more than half a million dollars. When the first ransomware attack happened in 1989, I was six years old. This the ATV shot is a photo of my daughter.

at six years old When she's 40? I don't want us to be talking about the same types of security vulnerabilities that we're talking about today. If we don't start turning this around right now, it's just gonna get worse.

I want the world to be a safe place for my daughter and for my grandchildren. I want their energy sources to be reliable. And I want their food processing plants to be safe in operational.

I want their computers and the internet to be a place where they can connect and create. It is time for us to do better and I believe that we can I have tremendous hope but no one's going to do it for us. There is no software program or machine learning that can do this for us.

We have to decide that we want it and then we have to do it. It's not easy, but it is simple. We need to work together security folks and Engineers to collaboratively decide that it's important enough to get asset inventory.

Right? We have to decide that it's a important enough to update our software and install patches when software is vulnerable. We have to decide to look for security vulnerabilities that we know our exploitable and find them and fix them.

This is something that we can only do together. It is time. Thank you so much for joining me today.

I would love to connect with you and continue the conversation.