Julie Peterson and Orion Cassetto, Cycode | DevOps Connect 2022

Thanks for joining us today. Our topic is effectively tackling hard-coded secrets with a secret management maturity model. A little bit about us real quick our speakers before we talk in about the actual content.

My name is Orion cassetto. I'm senior director of product marketing and psycho. I've been in the security industry for a little over a decade and a half notable past employment includes some appset vendors you may have heard of cell networks imperva armorize Technologies and most recently I've had a company called XD.

I'm a frequent conference speaker and security enthusiasts. So definitely Stokes to talk about today's subject with me is Julie Julie you want to run us through your background? Sure.

I'm Julie Peterson. I'm excited by excited to be here to talk about Harkin and secrets and I have been working and check my entire career. I've had work published in Forbes d-zone Security Boulevard and others and past employers include white Source now men IBM and IBC wonderful All right.

So today's topics first. We will generally we're going to be talking about hard couldn't secrets. So we're gonna start with an overview on the subject.

So what is it why you should care? What danger do they present? We'll talk about some incidents that have made it into the news.

Then we will talk about how you can effectively put together a secret Management program with a maturity model. We'll walk through that step by step. Then we'll talk about some capabilities that you should look for in tooling that can help you to you know, bring that maturity model to life.

And then finally we will go through a Q&A. So we'll be on and you can use the chat to ask us questions and we'll get them answered for you. By the way.

Feel free to use the chat box and ask questions as we go along. So with that said I'm going to hand it over to Julie Julie. You want to kick us off with the topic of hardcoded Secrets?

Yeah, so we're gonna start with kind of an overview to set the stage. And you know, so what our secrets and code. Well the vast majority of them are authentication related.

So things like username and passwords or authentication tokens. There's also things like API keys and tuna database credentials. We're also seeing a lot of secrets that are encryption related.

So certificates or encryption keys are very common. And then finally, we're seeing a pretty big rise in the number of custom secrets. So that could be something like an ID number that correlates to a customer which perhaps on its own is in a big risk, but if you combine it with Insight or knowledge you have you know a lot more information to go on and it could result in some some risk, you know other things that we see frequently and say the retail space are things like activation codes or discount codes which might be hard coded and you definitely don't want exposed to the public.

You know, you don't want somebody getting 50% off across the board. And then more and more we're seeing GitHub used for non-code file storage sort of like a Dropbox, right? So in addition to code they're storing other things in there.

So organizations need to be aware of what those things might be. So we are seeing a definite uptake in the amount of secrets that are being hard coded and why is that? Well, the the short answer is that the way in which we develop code has changed dramatically over the last number of years.

The long answer is that you know development has shifted to relying on things like sauce and third party services. So we're not talking just like open source, but in sdks but you know things like apis and all of these things need authentication to talk to each other microservices as well as we develop in smaller and smaller chunks those things like containers need to be able to communicate and that again requires more Authentication. And then finally you have things like infrastructure as code, right?

We're developing a smaller and smaller chance. And so we are creating the infrastructure via code so that we can deploy them quickly and again that creates more opportunities for Authentication. And you know our secrets is the can then kind of a long standing practice.

So why are they a problem and why are we seeing more and more breaches that involve part for the secrets? Well, first of all lateral movement is a big issue. So the sdlc is comprised of tightly connected and integrated tools, you get the credentials or you get into one of them you can find credentials into the other tools and you can move you know, And shift left into source code or you can move right into production environments.

But basically what we're seeing is secrets are allowing attackers to live in systems for far longer than ever before and to move laterally using legitimate credentials making them harder to find secrets are also problem for compliance issues. So there's a number of different compliance protocols that require things like separation of Duty. So something like servings Oxley rate, you have to demonstrate a clear separation of Peace.

If you have a developer who can find a hard code a secret that will allow them to log into a database. Then you have a clear break in that separation of Duties between developers and say a DPA So, you know, we're not seeing a ton of these just yet, but it is something that is on the rise and we expect more Auditors to be asking these sorts of questions. And then finally, the exposure of Secrets is just increasing exponentially.

There's a number of reasons for that. First of all. Attackers are targeting developers account specifically.

So if you look back at this office attacks from just a couple of years ago was a really sophisticated attack that targeted developers specifically because their accounts are typically very over provisioned and they have access to a lot of great things that attackers want including Park could have secrets. You know the second problem and this one is really huge is that we're seeing more and more leaks Happening by mistake. So the way get systems are designed you have developers bringing their own credentials to their organization.

And if they mistakenly save their proprietary code to their personal developer account and oh by the way that developer account is set to public like many are all the sudden you have, you know lines and lines of code being leaked. And then finally, you can't discount the problem of malicious insiders. So somebody who you know has keys like, you know, it's supposed to be in your systems, you know as a developer or what have you and perhaps and you know, they have access to other things that they want to take with them one interesting thing that we've seen recently with.

The great resignation is a developers are leaving their jobs sort of three times more than they were just a few years ago in that it is not that uncommon for them to take some of the code that they've developed with them. If that code contains Secrets. Then you have a problem and you know the human part of trying to change this insecure coding practice is really big.

It's it's quite an ass to try to get people to developers to change the way that they've always done things. so here this image is githubs digital Millennial Copyright. Act takedown notices over the past few years and what this line is showing you that the trend is obviously increasing right?

So this means that more and more organizations are asking for their code to be taken down. So I think in 2015, the number was under 150 requests last year. It was over 2000 and oh by the way, each request is not a one for one per repo but often is request to take down hundreds or thousands or more of repos.

So again, the problem is pretty big and the big thing here is that you can't assume that secrets are only going to be seen by the people who are authorized to see them with the rate of leaks increasing you have to assume that Your code at some point is going to be exposed when your code is linked and you want to make sure that there aren't Secrets there to make, you know, believe itself even that much worse. And then, you know don't just take our word for it. There are actually a number of high-profile instances that we've seen in the news that where her good secrets were at the very core of the week in the first one is something we've all seen and feared is the solarwinds attack.

Right and it was a very sophisticated attack by a Russian nation state and you know, the bottom line here is if that kind of hurt that kind of organization is going to attack you you probably have very little recourse but We know that sort of wins was reached through their team city server, but there's also evidence that secrets were involved. We know that silverwinds was leaking Secrets via an FTP server and is believed that because silver winds was so thoroughly compromised that Secrets played a huge part in the ability for the attackers to stay in the systems for as long as they did into move about and compromised solarwinds. So thoroughly, Now if we look at their Downstream customers, right?

So the attackers weren't necessarily just going after solar woods, but they were going after all the downstream customers including you know, US federal government agencies, but Microsoft was one of them and Microsoft was infiltrated by the attack but one of the things that Microsoft was able to do sort of post-mortem was see what they were after and it was very clear that the attackers were scanning for and looking for specifically hard coded secrets. And Microsoft said that they didn't lose anything because they didn't have any hard coded into the code that was exposed. But again, you can just see that Secrets led to the breach and then were the main target.

I think Orion you've got another one. Yeah, absolutely. Another interesting incident Comes To Us by way of the New York State Office of it and essentially here they had a leakage that happened because they're gitlab instance was misconfigured.

The end result here was that there were you know gigs of data that were leaked out into the public including you know secrets and you know, those Secrets also in some cases were for production environments. I think the takeaway here and what makes this so interesting is that This these secrets were exposed potentially to attackers by way of something that had really nothing to do with coding right this this exposure happened just because of a misconfiguration of you know, complete other system the misconfiguration by the way allowed it. So anyone could log in anyone on the internet to log into their get lab create an account assign themselves admin access and then just view anything.

So it really illustrates how easy this can happen and how it can happen from systems that aren't necessarily, you know, coding related, you know an SCM is a repository, but it's not actually a coding practice. yeah, that one blows my mind every time I hear about it another, you know, big breach was the code called and this one was Secrets all around so The attackers were able to compromise khokov. By finding a secret in a Docker image.

So we have Decor image that was publicly exposed. The attackers got it logged in and what they did was they actually altered A bash uploader script which was then pushed to codecoff's customers and what the change in that script was was that the attackers had instead of sending the get credentials back to Coco. They was sending it to the attacker server.

So the attackers were able to get those legitimate Cricket credentials and then log into the downstream customers. And from there. They were specifically mining for hard-coded secrets to get into other systems.

So again Secrets all around impacted a lot of great companies, you know twilio hashicor Monday in more but again secrets were The end all be all here. Another great company that had a secret incident is Amazon. So essentially what happened here was in Amazon web services engineer inadvertently made public a lot of data over a gig of data and then quoted not only their own documents but also secrets that were credentials that could gain access to AWS environments now, I think there's a couple of noteworthy things about this one first and foremost is it's Amazon who is you know, a company known for being kind of at the Forefront of cloud technology.

So if it can happen to Amazon, it could happen. Anyone two is really related to the time that this incident took from the time that the secrets were released out into the public to the time that they were discovered and reported back to Amazon was a mere number of minutes. We're not talking about our so days.

It was a couple of minutes and you know, luckily the people who found and reported this back to Amazon, you know, it was a benevolent party, but it could just as easily have been an attacker and it really shows you, you know, the risks that exposure poses, you know, it could just be a couple of minutes or a couple of hours of exposure that could result in, you know your secrets getting in the wrong hands. cool That is our last incident. So what we're gonna do here is shift here.

So we've talked a little bit about what secrets are we've given you some good examples of you know, how they've happened out in the wild and some takeaways and now we're going to be talking about how you can build an effective Secrets Management program using a maturity model and the goal here is for you know us to identify, you know, kind of different levels of maturity here and ideally it offer you approach, you know, a framework or rubric that you can find yourself in and to identify some proactive steps that you can take to further your program. and the model that we've put together looks like this so it's essentially five levels with them the least mature on the bottom and the most mature on the top and so essentially Our goal again is for you to identify where you are to move up this this pyramid with actionable steps. So we're going to start at the bottom and we'll walk through each of these one by one.

So level one is awareness and you know by Nature by virtue of you being here at this this session. You essentially are are in this level or higher and people who are in level one character are characterized by you know, understanding that essentially the needs of your development and your architectures moving into microservices and Cloud, you know using SAS and apis is driving up your authentication needs. That means that you have more and more need for you know, essentially secrets.

And so therefore you can recognize that those Secrets may end up being hard coded unless you provide an alternative. So at this point, you know that there is a problem because you know, there is a problem. You're likely putting some form of training in place as a first step to deal with that and you may be thinking about policies that you can put in place to deal with this however, a level one organization you do not have any detection mechanisms or en.

Mechanisms like a really good example an anecdote that I can give you here. We were talking to one organization and they were trying to tackle their hard-coded secrets Problem by finding developers $50. If a secret ended up hard coded into a commit or a pull request, you know, that's there's no, you know automated detection mechanism.

There's no enforcement mechanism other than a $50 fine now. Narrow is a lot wrong with that as an approach. It doesn't you know go back into version histories and look for hardcoded Secrets it you know things can get by 50 dollars is a lot of money but not a lot of money for some people and so it may not be a great deterrent.

And so essentially this is less than a Band-Aid in the world of our good Secrets, but it's a starting point right people start here. All right moving to the next level of basic detection. Here's where you're starting to take a program of like hey, I know there's a problem to hey, I'm gonna do something about this problem and it starts with adopting a scanning tool to look for Secrets now for a level two organization that effort and that tool is a laser focused.

On your Source control management system and it's usually focused hopefully in all of your repos, but definitely on your repos that are used to produce your production environments. So things that you know, go in your main branch essentially, right? It may also include your version history.

So looking in the past if you had a secret that existed in the past in the version history, which took it out now, but people still have access to that version history. Guess what? It's still exposed.

So you need to make sure that you know, you look backwards and clean that up as well or revoke those Secrets, which we'll talk about later. Here you're scanning for very common secret formats of the ones that Julie talked about earlier API Keys, you know authentication credentials passwords Etc an important thing about scanning at this level. These tools are typically running on an ad hoc basis or on a timed basis so that we may see some different approaches as we move up the maturity model.

Okay, level three is Advanced detection. So it's building on the same Concepts and tooling of a level 2 organization, but it's increasing in terms of breadth depth and Fidelity for breath in depth. What we're really talking about here is looking for more types of secrets in more places and the big one is that secrets are not just applicable in your Source control management system.

In fact, they can show up all throughout your sdlc. It can be things like, you know infrastructure as code. It can be in, you know build logs.

It can be in personal developer repost. It can be in public repos. They can be in artifacts and container Registries that can be in public Cloud infrastructure so they can be in a lot of places you need some ability to be looking way more broadly than just the SCM in order to deal with this in a kind of a holistic and complete fashion.

The next thing is about depth in here. What we're talking about is looking for more than just basic a secret types and look expanding into those custom categories that Julia was talking about so this could include things, you know, like maybe something that's specific to your business logic or your application logic like a discount code or an activation code. and then at level three we bring in the idea of prioritization.

So here you're going to be thinking about where is a secret. What type of secret is it? Is it exposed who has access to it?

Is it in the production system or a staging system? To do that you're going to need a lot more information. We'll talk about what that looks like and what where you can get that information, but you're going to need context about the world that the secret is from and living in and what type of secret.

Okay. Level four is where things get really interesting. So at this point we've essentially given you some mechanisms for finding Secrets.

Now, we're gonna be giving you some mechanisms to enforce your your secrets program and your policies to prevent secrets and then also to provide developers with Alternatives so that they can adhere to those policies. We're going to talk about these two things separately one on each side first, we'll talk about enforcement. The idea here is that if we want to say, you know, essentially we don't want hardcoded.

We don't want secrets in our code. Then we're going to need to to enforce that somehow the first way to do that is to integrate scanning into developer workflows. Here what you're going to be doing and you can see a diagram of it here.

So we've got our main branch. We've got our feature Branch. We want to bake scanning in.

You know pre-commit hook and also at the time of your pull request or your merger request and you're with the idea is in either of those opportunities. If you find a hard coded secret you want to tell the developer about it and provide them some remediation suggestions so that they essentially, you know can do something about it before it ends in your main branch end up in your main branch taking that same logic one step further and a little bit more, you know kind of Iron Fist approach is to say all right at time of pull request if the test is failed. So if there's a hard coded secret present, we want to block that code for being merged into your main branch and you're kind of sanitizing the main branch and protecting it you're preventing it from having our credit Secrets now not everyone is comfortable with this.

Some people are some people aren't but if that's the approach that you're gonna take. Well, you're essentially roadblocking your developer, right? And so that means you need to give them an alternative to hard coding secrets so that they can easily overcome this Roadblock and then rectify the situation and get their pull request into the main branch.

And that is where a secret management tool. It becomes an effective complementary solution. It's like a one-two punch.

Secrets management tool essentially what it is is think about the concept of a password manager or a privilege activity monitoring tool. And apply it to secrets and so what a developers essentially going to be doing is they're going to be using this tool. They're going to call a code such that their application is going to reach out to the secrets manager anytime that it needs to authenticate or anytime that it needs a secret to gain access to a resource.

It's gonna get a secret from the secrets manager and can then use that secret to access the resource and question. And then essentially that means that you don't need a hard code you're hard coding a call to the super manager instead of heart cunning a secret. The benefit here is that the sinkers manager allows you a single point to manage those Secrets, like if a Secret Gets exposed or burned you need to rotate it.

You can do it at the secret management without going back to the code and it's also, you know not exposing the secret directly in the code. So the code gets exposed. You're not enough bad spot.

So this is a really powerful kind of one two punch and again, it can help to increase adoption of your policies. All right, the last level the highest level in our pyramid here in our in our management or maturity model is risk reduction. The reason at the highest level is it's taking all of those Concepts that we've had in the past.

In building on it by thinking about you know, the idea of Secrets through a risk reduction model and okay. Let's look at this picture here. So the idea is you've got a threat which is your hardcuted secrets and it becomes a risk when it's a threat plus you're exposure.

So how do those Secrets get into a place that becomes dangerous or it becomes in the hands of a hacker and that happens through three primary means code leakage, which we've talked about compromised accounts. So an attacker gains access to an account and then they gain access to any secrets at that account has access to or you've got a malicious Insider who's abusing their privileges. So essentially the idea here is if you you know thought very holistically about how you could reduce those exposure on the right then you can reduce the risk.

So, how do you do that? Talk about these one at a time starting with compromise accounts. But essentially the idea is you use complementary Security Solutions in your sdlc to address each of these different areas for compromise accounts the problem.

As I said a second ago is an attacker get you know steals are credentials are gain access to your developers account. Then they have access to all of the code that that developer has access to it if there's hard coded secrets in it and boom they have access to that. Now what you can do here is two things, you know, first and foremost, you need strong authentication for your developers accounts.

This makes them harder to compromise. The second thing is that you want to aim Implement a concept of least privilege. So reducing access to code to only the code that developers need to do to do their day-to-day policy or day-to-day activities.

This is different than what happens in most organizations where developers are kind of over-provisioned and have access to lots of stuff in case they might need to use it. So we want to trim access and privileges down to just what you need. The reason this is important for compromise accounts is essentially attackers now need to compromise and account and it needs to be the right account that has access to code that has Harkins secrets in it.

You're just making it harder and harder for this to happen by putting kind of more and more barriers in front of attackers. Some of those same Concepts also work for your malicious insiders. So your malicious insiders are looking to you know abuse their access privileges they gain access to you know, that database credential now they have access to something they shouldn't and then you have risk, right?

Well it by implementing these privilege what you can do is essentially make it so that that developer account is not super over provisioned and has access to everything they only have access to what they work on on day to day in their day-to-day activities and you know essentially thus you can reduce the likelihood that that one developer who is malicious in going rogue has access to code that has hardcoded secreted. So again, you're just reducing your exposure and reducing your risk. The last way that you can do this is by working on to detect and prevent source code leakage first in terms of prevention.

The idea is by implementing strong security and governance policies in proper configurations to prevent things like we saw with the New York State Office of it. You can make it harder for people to or for your code to get leaked out into the public. So this is really a complementary security approach where you're looking at security controls and configurations to just Harden things and lock things down so that you know, you reduce the risk of this happen.

The second thing that you need to do is to if your code ends up out in the public like we saw, you know with Amazon you need to bring you down very quickly. You need to understand that your code is out on a public sharing site and you need to you know have the ability to you know, find it and take it down quick to that. And essentially what you need is a set of capabilities where you can fingerprint your code and scan on a regular basis the code sharing sites that are out in the wild looking for those unique indicators and then to be notified so you can bring it down quickly now is this 100% full proof that you can say?

Hey, you're gonna find it before attackers. No, but it again reduces the likelihood, you know, you're just the time it's out in the wild. Then you reduce the likelihood in an attacker can find it.

So it's just taking you know, this like multi-lens to approach to reduce risk. With that we're going to go ahead and talk about some of the capabilities that you would need to look for in tooling. You know, that could support the program that we just we just described so with that we're going to go through a checklist, Julie.

Do you want to kick us off sounds great. We would love to. So the first thing you need to do is scan right the foundation of any solution.

So a good solution has complimentary scanning approaches. So that includes looking for new patterns, which are the most accurate way to scan and that includes something like, you know, what a Google key pattern looks like so being able to scan for things like that. Second way is to search for a high entry strings.

Now. This is a great way to find Secrets but unfortunately can also result in a hot pre-high false positive rate. And then finally sort of specific application usage.

So how your application is built to kind of think of it as like custom Secrets, but being able to scan for things that are specific to you. So once you've done this scanning you need to actually kind of make some sense of these the results and a good solution allows you to improve the results by saying giving developers the ability to report false positives, right? So high entry strings lots of all positive developer can mark this one as something you can ignore forever because it's not a secret.

So that is when we can improve your results and cut down and noise. And then finally you need to be able to scan across your entire sdlc from your code creation through production secrets are all around Ryan mentioned, you know having it in in logs and path places. So you need to be thoroughly scanning your entire pipeline to find all of your secrets and This serves another purpose in that if you have a secret in a test environment, it's less critical in resolving that then say something in your production environment, which is a good segue into our next thing of prioritization Ryan.

Here you go. Awesome. Yeah, thank you.

So the next thing you're going to be looking for in your tooling is the ability to prioritize your secret remediation efforts, you know, essentially you may end up with hundreds of thousands or tens of thousands of violations or secrets that you found throughout your entire sdlc and you need a tool that can essentially use context from everywhere and every tool that's in your stlc so scms build tools container registries, you name it and to help you to take that information and tell you what's important so you can focus on the most critical things first here some of the things some capabilities that you may want from your tool is understanding where secret is used test or production, you know, staging or runtime how critical is the resource being protected? Is it, you know password to your S3 buckets is an API key. Is it something that is unimportant How likely is exposure who has access to it?

How many people have access to it? How strong are your you know? At least privileged model or your identity model how Strong's the authentication, you know, all sorts of things like this really come to Bear to how critical an issue is and so being able to take in into account.

All of these things can really help you prioritize on. the most critical things first truly back to you. Thanks.

Yes. So, you know one of the next things that's really important to think about is workflow Integrations right changes hard. And so we feel the best way to Have developers stop hard-cutting Secrets is to give them feedback on potential Secrets where they live and breathe in in their repo like basically integrating into their workflows rates.

So you're giving them the results of a skin for secrets so that they can remediate it right away. So they're not looking back two weeks later trying to figure out what it is. They're working on they can do it in the Moment by giving it to them in their workflow your limiting context switches and essentially your Protecting your main branch.

So this graphic here shows a notification in GitHub and it shows you that there was a potential secret that was found and it gives the developer the ability to then do some sort of action on it, right? They can sort of globally ignore it or they can you know, resolve it or you know somewhere in between. So there's a number ways you can integrate into workflows, you know, pre-comment hooks or even talked about in a previous slide pull requests Integrations learning and then again, Integrating into existing ticketing systems.

So, you know could be something like servicenow or Jira, and then again, you also want to confirm that these fixes have actually taken place so that when you scan the next time, you know that it's been resolved and it's something that the developer no longer needs to worry about. awesome Our last major capability that is required in order to implement the model we were talking about or one of the last ones is Secrets management. So, you know essentially secret Manager Tools, like I said earlier essentially allow your developer to code the application to ask for a secret when they need it they get their secret from the secrets management tool and they use it to access resources.

You can get them from a number of sources and including infrastructure providers like AWS Google Cloud Standalone vendors, like a keyless Pam vendors like Secrets managers and even more past one that's very popular called. So any of these are others may work, but it becomes you know, essentially a very great one to punch or complementary solution to you know, the enforcement or the the scanning that that you're going to implement. In summary, we've talked about quite a lot today.

So secrets are a large and growing problem being fueled by architectural changes and how we make applications today things like microservices and API calls to to third party in SAS Services. A maturity model can be a very effective way for you to plan out how to grow a Secret's program from something that's very nice to something that's very mature and if effective Secrets Management program needs tools that can help you to scan for secrets to prioritize remediation to do enforcement by you know, essentially integrating into developer workflows and that protects your main branch and then providing Alternatives by implementing a secret management tool you need to be taking into account to get to the highest level other things that you can be doing with other security initiatives that can have an impact on your secrets program things like, you know, proper security and governance deal with malicious and compromise. First and preventing and detecting code leaks to reduce code exposure to potential attackers out.

What? And then last slide before we go into a Q&A just a quick note about SCI code and we are a complete software supply chain security solution that provides visibility security and governance and integrity across your entire sdlc. There are a number of use cases that fall out of that including security and governance Koh tampering hard coded Secrets detection, which is today's topic code leakage IAC security software composition analysis and security and compliance governance.

com, otherwise head over to the chat box and feel free to ask us questions. We will hang out here for a little bit answer your questions. And yeah, thank you very much for your time.

Yeah, thank you.