Caroline Wong – Security Testing at Scale

My name is Caroline Wong. I'm the chief strategy officer at kovalt. I am thrilled to be here virtually with you today at the appsec API security event 2022.

I started my cybersecurity career more than 15 years ago leading information security teams at eBay and Zynga. These were super cool places to be working in cybersecurity in both cases. We were running online operations 24 by 7 with millions of simultaneous users daily.

94% and is one of the first major electronic Commerce shops enabled strangers to transact with each other over the internet. Zingo was growing incredibly rapidly as an early adopter of Amazon AWS in 2009 the Zynga game Farmville launched and in just a few weeks the game went from zero to 10 million daily active users a few months later it Rose to 80 million daily active users. At Cobalt we build security software and we do pen testing as a service.

I want to start this talk about security testing at scale. By telling you about the first time that I realized that most organizations do not do enough security testing. Between 2013 and 2016.

I led more than three dozen beesim assessments be some stands for building security and maturity model and it's a descriptive framework for real life software security. There are 128 firms in the latest be some data pool. This data pool can be thought of as somewhat representative of all the companies in the world that operate using software, but with one pretty big caveat, which is that it costs five figures to do a decent.

And so if you can spend that much money paying a Management Consultant to interview your software development teams and give you a scorecard on how well you're doing software security. You're probably relatively mature when it comes to application security. So we can safely assume that the bezum data pool is above average compared to the rest of the world.

I learned something really startling when I was doing this work flying around the world talking to teams about their software development practices. I found out a typical Enterprise with 1,000 software applications was only doing manual pen testing on about a hundred of those applications. Just 10% of most Enterprise application portfolios.

We're getting proper and appropriate security testing. this was Terrifying there are a lot of reasons for this historically security. Testing has been expensive and slow.

It is hard to get access to Quality Talent whether you are building an in-house team or working with a third party provider. It's for you, which is that last year in the Cobalt 2021 state of pen testing report. We found out that today organizations are doing more security testing.

We talked to more than 600 International Security and it folks and found out that on average folks are pen testing 63% of their application portfolios. This is a huge increase and I'm here to tell you why how and to promote the idea that we could actually increase that number even further. I firmly believe it is the right thing to do.

Because if you're responsible for 100 houseplants, it would be unacceptable to only water 10 of them. If you're in charge of caring for 100 cats, it would be unacceptable to provide food and shelter for only 10 of them. Last year in 2021 our industry got a new owasp top 10.

the first owasp top 10 came out in 2003 17 years ago and when I put our latest owasp top 10 next to the original owasp top 10, these two are alarmingly similar. What this indicates is that the most common vulnerabilities found in web applications. They're not so different from 17 years ago.

17 years is a really long time in 2003 Matthew McConaughey Matthew McConaughey starred in fantastic film called How to Lose a Guy in 10 Days. Last year, he starred as an animated koala in the movie sing too. Seventeen years is a long time for very little change.

And so why haven't things changed why aren't they getting way best way better way faster when it comes to the way that our industry does security testing why are just 10% of enterprise software portfolios getting tested in 2016 and just a little over 60% in 2021 today. I'm gonna cover a brief history lesson with regards to cybersecurity as well as software development. I will present a maturity model for pen testing and I will give some advice on how to scale pen testing and achieve the five ideals that is a fun reference to the Unicorn project.

So here's here's the question. Why? If we know what to do.

Are we not doing it? the owasp top 10 and the supporting guidance provides information on exactly how to find fix and prevent the most common security vulnerability types and this information has stayed fairly consistent for nearly two decades. the problem Is it it's just not that fun?

And it's not cheap. And it's not convenient to do. So I think there's a difference between doing controls and managing risk.

This industry loves Frameworks and best practices and standards. This graphic on the left is the table of contents for nist. 853 a nearly 500 Page Long document.

I actually think we do ourselves a disservice. With these extremely lengthy documents. I think sometimes this can make cybersecurity and security testing actually seem a lot more complicated than it really is.

I believe that cybersecurity can be simple. Simple does not necessarily mean easy, but simple does mean easier to understand. On the right you see a simple model of just four things you need to do for application security.

We need to govern we need to find security problems. We need to fix security problems, and we need to prevent security problems. We have in front of us an opportunity to focus on the things that matter and to do them well and to do them often.

And security testing has everything to do with this. If I take a step back and I ask myself what is the point? Of cybersecurity.

What is the point of application security? We're trying to manage risk. Security is about protecting value and today so much of what we value has shifted from the Physical Realm to the digital Realm.

Value protection is about risk management. this list says here are reasons. We need to manage risk because we care about what other people think.

This separate list says we need to manage risk because we want to build and maintain and operate software that is actually resilient to malicious attack. We want to secure our software. By preventing the same cybersecurity problems from happening over and over again.

By reducing the probability that malicious attackers can cause critical systems and applications to stop functioning and we want to require fixes for security bugs for which well-known attacks exist. We are not getting this, right. my former manager Sammy migas is a brilliant guy and he has also said that historically our industry has not gotten risk management, right?

prioritizing compliance or features over a comprehensive process that increases resistance to attack. And also gives us compliance and better security features is not the risk management. We need historically this industry has not gotten risk management right budget driven risk management is not good enough.

If we are only able to cover 10% of a self-reportfolio with appropriate security testing that is not good enough cybersecurity. They brief history lesson. Let's talk about the past decade better yet.

Let's talk about the past two decades in 1998 a collective of young hackers went to Washington with a warning for Congress. during that now Infamous hearing they told Senators that any of the seven individual seated before you could take down the internet and just half an hour. In 2018 20 years later the same hackers went back with a similarly Bleak assessment.

One of them in particular much has recently offered some pretty specific observations on his perspective with regards to Twitter security. And in 2018. The group said look digital security is hardly any better than it was 20 years ago.

Joe Grant who went by the name Kingpin in his early hacker Loft days. He said at Loft we tried to be the voice. Of reason and raising awareness to problems nearly all of what we said 20 years ago still holds true.

Yes, there have been some improvements but the general class of problems is the same. Chris weisel said back then. The threat was the teenage hacker.

Now it's nation-states. So every vulnerability got a lot more risky. in the last 20 years in cybersecurity things are mostly not changing despite the fact that we know exactly how to find and fix and prevent these most common.

Application security vulnerabilities. I read a book last year. It is called 4,000 weeks.

the basic premise of the book Is that an average human lifetime consists of 4,000 weeks? I happen to be this year in 2022 crossing my 2000 week Mark. I have spent my entire career.

focused on cybersecurity And in the next half of my life. And in the rest of my career, I want to see some change happen. I would like to see some fundamental changes happening in the cyber security industry and I would like to see us get to a point where we are making meaningful progress.

So for as much as things have not changed. In cyber world we can actually look at our friends and colleagues over in software development land and see that. in the past decade they have made tremendous strides.

In 2011 puppet release the first state of devops report. two years later That report had established a relationship between a devops practice and high performance outcomes. Organizations that do devops consistently report more frequent deployments shortly time to change lower change failure rates and faster mean time to recovery.

These results have a direct impact on business outcomes organizations that can deploy on demand and have a shorter late time create comparative advantages. Because you can deliver solutions to customers faster. Shorter mean time to recovery and lower change failure rates mean your systems are more stable.

So if companies doing devops get higher performance than naturally we want to know what they're doing. And the state of report state of devops report continues to give us great prescriptive advice about what sorts of things work best in today's environment. So from the state of devops 2021 report security Champion afterthought or the final step before delivery.

It must be integrated throughout the software development process. To securely deliver software security practices must evolve faster than the techniques used by malicious actors during the 2020 solarwinds and code Cove software supply chain attacks hackers covertly embedded themselves into the infrastructure of thousands of customers of those companies. Given the widespread impact of these attacks the industry must shift from a preventive to a diagnostic approach where software teams should assume that their systems are already compromised and build security into their supply chain.

It is easy to emphasize the importance of security and suggest that teams need to prioritize it but doing so requires several changes from traditional information security methods. State of devops 2021 says that you can improve security improve software delivery and operational performance and improve organizational performance by doing the following things. Conduct a Security review for all major features invite information security early and often use high quality documentation.

so what is one big difference that our friends in software development land have made happen that we in infosec land can observe and learn from we've gone from on-premise into the cloud SAS has benefits SAS companies have fundamentally transformed the enterprise software model and at this point it seems like there's no going back. Peter Levine in His science Manifesto says it doesn't happen often every 10 to 15 years or so, but we are in the throes of the reordering of the four trillion dollar corporate it market and depending on what side of the transformation you sit. This is either the best time to be an Enterprise technology company or reason to start looking for a new line of work SAS companies don't necessarily have all the answers, but they are inherently flexible and can iterate according to Market feedback and business needs.

It's all about speed of innovation and design and usability. The faster you can go the less you spend on product development and the fewer person hours are required to deliver a complete solution. Every iteration is an opportunity to deliver greater business value.

So we have an opportunity. Our opportunity is to take traditional information security testing and turn it into SAS. I present to you a pen testing maturity model and if you find yourself in the ad hoc or in the structured part of this model, and you want to get to the Strategic part.

I have some advice for you. Here's my advice on how to scale security testing. first thing do it faster.

And do it more often. When does an organization do security testing usually it's because of one of the following. Number one.

There's a customer who says I need to see a security testing report before they commit to buying your product. Number two. There's a regulator who says I need to see a security testing report as part of a Regulatory Compliance exercise.

Number three you're getting Acquired and the company that's buying yours wants to see a security test report before they complete the transaction and add your risk profile to theirs. Number four you've just deployed a new feature or an entirely new product and you want to find and fix security vulnerabilities before they can be exploited by malicious actors. In any case one way to get strategic with scaling your security.

Testing is simply to start faster. When you've decided that you need to do security testing get the technical experts performing manual testing activities right away, like within 24 hours because the sooner you start the sooner you can have a finalized report in the hands of the person who is demanding it. It also means that you're that much closer to knowing about exploitable vulnerabilities in your software and the sooner, you know about them the sooner you can fix them.

And then the sooner you can start your next security test because the product and Engineering teams are just going to keep building new things. And if those new things aren't getting properly security tested, then they're probably vulnerable. Advice number two remediate smarter one super silly thing that I've observed throughout my 17-year career working in information security.

Is it a lot of our compliance regimes and best practices Frameworks? Well recommend that you do some type of defect Discovery like pen testing or scanning for vulnerabilities. But then they don't actually require that you fix what you find.

When I talk about scaling security testing, I don't just mean printing a PDF handing it to a regulator checking the box and moving on with the rest of your day. I mean finding security vulnerabilities in your applications and fixing them. Now don't get me wrong fixing vulnerabilities is hard most of the time if you're a security professional fixing a vulnerability actually involves getting someone else to do something.

Getting other people to do stuff is hard and I can think of a way that does not usually work if I get a 50 page PDF in my email and I forward it to a bunch of engineering managers and I say your results here are the results from the test. We did. Please work with your teams to get the findings for mediated.

Then the person probably opens the email freaks out and I don't know for mediation actually happens. A better way might be if as soon as a vulnerability finding is discovered the engineering team finds out through slack and then goes into jira where a ticket has been submitted that can be worked right away. It would be so cool.

If there were a direct line of communication between the engineer and the tester who found the issue in the first place so they could ask questions and give advice and work together to get security vulnerabilities addressed. advice number three use data I have been obsessed with security data and metrics since the start of my career. I believe that better data enables us to make better decisions and Achieve better outcomes at this late stage in the cybersecurity game.

Let's remember that hackers testified before the US Senate in 1998. We should have a ton of security test data and in theory, that would help us a lot. But it takes grunty time-consuming effort to physically parse through piles of customized Consulting reports to get actual security test data and format it in a way such that it can be collected and analyzed over time.

I've known consulting firms who have tried to do this because the data would be so interesting. But it was too much work. There is a better way.

If test data is in a standardized format that can be organized and accessed through an API. And you can actually analyze the data. You can actually use it.

for years, I've heard people say yeah, the OS top 10 is good and all but what's really important is your organization's top 10 or your top five or your top three, but how many organizations have their security metrics to a maturity level where they even know what their top three classes of security vulnerabilities are not alive not enough. Now there are some companies that have super sweet security testing management and data and dashboards. Everything is custom built and very specific to that Organization for this type of company an API to transfer pentas data from one platform to another can be super helpful.

I think that we as cybersecurity folks can take a literal page from our friends in software development and transform what we do in our industry from the way things have been done traditionally to the Sass way the cloud way and we can begin to achieve. The five ideals I wrote a book. It is called the pitas book.

It is about scaling security testing. Spoiler alert, the book is not going to do the work for you. But it will tell you exactly what you need to do in order to achieve security testing at scale.

Here is a thing. about this cybersecurity industry I think that we as cyber security folks love analogies. I also think we use a lot of analogies that are not quite correct.

I don't for example. Think that security is like a vitamin. I also don't think security is like a Band-Aid.

I don't think it's something you can inject or do it the last minute or add on after the fact security is certainly not a feature. Security has always been the result of decisions and actions made by many different people. I think it's actually the outcome of an unpredictable dance.

Between many people let security dance for deaf-secops to be successful. We must build a collaborative approach that brings us together. If you know how to SLSA or Lindy Hop you can extend a hand invite a partner and step onto the dance floor, but security is not always invited to the party.

Too often development security and operations stance alone. We've got to invent a dance style to sync our movements and create a beautiful partnership. Life is short people when I was six years old, the first ever ransomware attack directed users to mail a hundred and eighty nine dollars to a PO address in Panama.

And the first half of 2021 the average ransomware payment climbed to more than half a million dollars. When the first ransomware attack happened in 1989, I was six years old. This is a photograph of my daughter.

Who's six years old? When she is 40 years old, I don't want us to be talking about the same types of security vulnerabilities that we are talking about today. If we don't start turning this around right now, it's just going to get worse.

I want the world to be a safe place for my daughter and for my grandchildren. I want their energy sources to be reliable and I want their food processing plants to be safe and operational. I want their computers and the internet to be a place where they can connect and create.

It is time for us to do better and I believe we can I have tremendous hope. But no one's going to do it for us. There is no software program or machine learning that can do this for us.

We have to decide that we want it and then we have to do it. It's not going to be easy, but it is simple. We just need to work together security folks and Engineers to collaboratively decide that it's important enough to get asset inventory.

Right we have to decide it's important enough to update our software and install patches when software is vulnerable. We have to decide to look for security vulnerabilities that we know are exploitable and then we have to and then we have to fix them. And we have to we have to decide to look for the security vulnerabilities.

We have to find them we have to fix them. This is something we can only do together it is time. I'm Caroline.

I love talking about this stuff. I'm so grateful for the opportunity to talk with you about it today. I would love to connect and talk some more.

Thank you.