Accessing Resources in the Cloud: Infosec War Stories and Architectural Lessons Learned – Techstrong Con 2023

Hi, I'm Sharon Goldberg. I'm the CEO and one of the founders of Bastion zero, which is an infrastructure cybersecurity company based here in Boston. I'm also a professor at Boston University.

I've been in this industry of information security and infrastructure for the last I guess 15 17 years, then a bunch of work on core internet protocols and cryptography written a bunch of papers and now founded this company So that's me. What we're going to be talking about today is how do you think about securing the access that your employees have access to your employees have to your resources in the cloud right as we all move to the cloud. We have important Assets in the cloud and the risk that's associated with granting human access to those resources can be significantly high if those humans are compromised.

And so what I'm going to be talking about today is historic set of War Stories starting from 2001 and going up to 2022 of how people have built out infrastructure access systems over time and how adversaries have attacked them. And so in a sense, it's the classic infosec cat and mouse game. We build a system.

It gets attacked. We build a better system. It gets attacked and we're constantly improving the way that we design these systems that said a lot of organizations are still using the kind of systems that we're appropriate in 2020.

Let's say 20 2001 and not in 2021. And so what I hope to do in this presentation is walk you through some of the different architectures that you'll see in the wild and in different places and help you think about the way they've been attacked in the past. So you can apply the architectural lessons from these previous incidents to the designs that you're using in your organization.

So that's the goal of the talk. The tldr is we're going to go through some really fascinating classic Watershed infosec security incidents and try to see what we can learn from them so we can build better systems in our organizations. Okay.

So this is basically a history lesson in three acts. We're going to start from the 2000 to the 2010s where we had a lot of standing credentials in our system. Basically Keys passwords secrets that were held by people that could be attacked by adversaries recently.

There's been a big movement away from giving humans access to long-lived credentials that they can store like SSH keys or passwords and instead moving to a zero trust security posture. I'm going to talk a little bit about what is zero trust security posture is and then talk about some of the gaps in this security posture and what you can do to protect yourself against the type of breaches that we've started to see in the last couple years, like what happened with Uber at the end of last year and with the solarwinds incident. So again, I picked the sort of watershed incidents, but these are not the only times that these things have All right, so let's get started.

So in the early 2000s. You know what you would do if you had infrastructure usually it was sitting in a data center. Usually it was a bunch of servers or databases and you would have Engineers that would need to get into these servers and databases in order to do their jobs.

And so how did you do that back in those days the most common architecture was to use a Bastion host and what a Bastion host is it's just another server that sits in front of your servers or whatever your infrastructure may be at that time. It was servers and databases and that server is what you log into first. And from there you jump from the Bastion to whatever other Target you wanted to get to right.

So the user has a key to the Bastion then there's the the Bastion has a key to all the other servers and the user will log into the Bastion and from the Bastion log into the other servers. This is a classic architecture. You still see it used in a lot of organizations.

So this is an incident that I learned about from my friend Andy Ellis who was the chief security officer at Akamai at the time that this happened. There was an adversary called Fluffy Bunny. These were the early hacker days of the internet and what fluffy bunny did was they compromised this user watched the user log into sorry watch the user log into the Bastion host and then got on the Bastion host.

Then what they did on the Bastion Host. This fashion host was running a protocol called SSH which is what's used to log into servers secure shell something SSH was what used to log into servers. And with the adversary did was they swapped out the SSH client that was sitting on this Bastion host with its own malicious SSH client and what that client would do.

It was copying the credentials that that the SSH client was using to access the other servers. So essentially there was a malicious client here that was exfiltrating credentials every time a person logged in and put in a credential to log into the next box. The adversary was stealing that credential and able to then log into the Box and at this way, they were actually able to harvest all the credentials that were you know in this environment.

So this guy was eventually found and kicked out of the art of the architecture, but there is an important lesson learned from this right which is that, you know, this Bastion is an extremely powerful point in your infrastructure. I'm going to call it in this presentation a single point of compromise. If you can compromise this Bastion this single point, if you can compromise a single point of compromise, you have a lot of control to be able to access other more interesting parts of the system like all the servers or the databases that might be behind the Sebastian.

And so lessons Lessons Learned Bastion host are single point of compromise and don't give your users standing credentials, especially passwords because the adversary was able to exfiltrate this passwords and use them later right logging in from somewhere else with those passwords because the passwords were long-standing were long-lived. Okay, so 2001. I'm sure everyone in this in this watching.

This is familiar with vpns. What a VPN is it's like a perimeter-based defense around your systems. The way typically works is the user will have a key the login to the VPN and once they're in the VPN, they can move around to all of the targets that are inside their VPN and access them right typically you'd need credentials to the individual targets.

But once you get behind the VPN, you would be able to access these targets. If you are outside the VPN, you could not access these Target. So this VPN architecture historically has created a lot of sort of security feelings for for designers of these systems because the idea is if the service is behind the VPN or if the servers or the databases behind the VPN, it's not exposed to the Internet.

So we don't have to worry about someone accessing it. It's very different having a server on a public IP that's exposed to the internet and having a server that's behind a VPN right now the issue and what's become very apparent in the spoiler. Is that these days just assuming that something is behind a VPN is very insufficient, right?

Because what we've seen over time is that just because the server is behind a VPN does not mean that an adversary can't get into your VPN and move laterally through that system, right? So this is an example from operation Aurora. What happened here was that the adversary was actually able to compromise one of these servers and from this server move to the next server and start moving laterally through this environment.

This is another Act of my story that I got from from Andy. Right where they actually had an adversary in their system jumping from server to server for quite a long time before before Akamai was alerted to this activity by Google what happened was once these things were getting owned. They were phoning out to a command and control server and the only reason that they found out that they were owned was they saw the command and control channel to that server?

And realize that something was wrong. But once the adversary was inside the VPN the fact that they were in that VPN allowed them to move from one box to the other and that this shows the weakness of relying on vpns as your main security mechanism because once someone gets in they can start to move around. This actually this concept has become very popular and important recently in January of 2020 the White House released a memo that said that the federal government is no longer should no longer rely on Perimeter based defenses.

In other words. The government should no longer assume that security is following from your VPN that a VPN is not enough to assume that your assets are secure. And so what I'm going to show later on in this presentation is sort of the more modern approach to think about securing assets and access to assets which involves segmentation and zero trust architectures, but but really just looking at this, you know this incident in knowing that the adversary could move around I think should give enough of intuition of why this is a big deal.

Um the other piece here that we typically see and that we think about is that you know back in the day you may have had one VPN for all of your production infrastructure and it was just one zone. So once you got into the zone you could kind of move anywhere these days that's a more unusual. It's more typical to have segmentation and to have the types of vpns that will allow you to create different segments in your infrastructure.

So that access to one of them does not necessarily mean you can access another right in segmentation can protect, you know, less less important assets from actually being used as a pivot point to access the more important. Okay. This is another example.

When we have servers when we have computers, we sometimes need to administrate them, right you need to log into the machine. You need to update software and things like this. If you use active directory domain and administration, what would happen actually is that there would be an administrator credential that allows you to log in as the admin and do things to the machine like update the software and so on and so forth.

So, I'm not sure how many of you remember what happened to the shipping giant mask in 2017. They were attacked by ransomware. This non-petia not pet Gem ransomware and what happened there was that the adversary was able to basically compromise a machine inside mask the way they were doing that is actually very interesting.

It wasn't the typical approach of first compromising user and then, you know compromising a machine which is basically what we see in all of the rest of these incidents that I'm going to talk about in this incident. What happened was there was a server that was using Ukrainian tax software and what happened was the Talk software which probably didn't have the best security practices was hacked. And so the tax software effectively became malware that the adversary was able to use to own a single machine inside marisk from there.

They were able to extract the admin credentials from that machine and what happened at that at marisk at that time. They had the same admin credential to control access to all of their computers. So if you remember the story this was a story of the shipping giant effectively going offline, they were not able to ship.

They just turned off shipping. They were these stories of you know, the executives putting cargo on airplanes because they couldn't put them on ships and just spending money just to get them on airplanes to keep the business going. So this was a massive meltdown where basically people's computers servers everything just shut down because the adversary had the admin credentials to the servers and was worming its way through the network and just turning them all off.

So this is just sort of like the the most poignant example of one password. Controls them all is a really bad thing one password to control them. All is a very dangerous situation because that is what happened.

They extracted a single admin credential from one server and use that to hack all the other servers and basically cause them to melt put ransomware on them and they were all just melted like brick. so Um, you know the lessons here are you know, admin credentials whose scope is very broad. If everyone has the same admin credentials to their machines.

It's very very dangerous and as a corollary to that with a lack of segmentation in your infrastructure can create a lot of risks for your system because if one segment is compromised you want to keep it isolated and reduce the blast radius to that segment rather than have, you know, all of the infrastructure compromise just because one segment was compromised. The other thing for me is that this is like very interesting at least from the perspective of domain control and active directory. So domain control is AD is a popular way of controlling large fleets of servers.

It allows you to have a centralized location that you can administer these servers and that's really helpful so that you can keep them uniform you can keep them updated and patched you can keep software that you want on them and software that you don't want off of them. All of that is really really important but the adfs system because it's so power. Full is a very valuable singer point of compromise, right the system that controls the admin credentials to all your computers if you hack that system, that's a really valuable single point of compromise and it's a supply chain risk, right a software supply chain risk.

So what we have today in a lot of organizations where they use adfs or ad domain control, they actually have scaffolding around adfs to monitor it security and limit the risk that it gets compromised. And the reason for that is because adfs is such a powerful. It's such a powerful component of the network.

You really need to monitor its security right? So if you can identify positions in your network that are really risky if they get hacked then you need to have the security mechanisms put in place around those mechanisms and and that's an adfs is a great candidate for that. And the other thing that I would say is that if you actually look at a lot of the high profile incidents if you look for at Sea says right up on some of these Events, they do talk about adversaries going after adfs quite often because when they do they're actually able to log into a lot of other systems because adfs has admin credentials to all these other systems.

So if you act that, you can hack all the other things. Okay. So, you know, we we talked a lot about these early 2000s, you know, like more historical incidents and I want to talk about more of the architectures that we see today that a lot of organizations may not yet have but are on their way towards having a lot of folks have this a lot of folks don't have this.

Let's talk about what zero trust means. So zero trust effectively means that instead of using sort of perimeter-based offenses things like vpns, you're going to control access to individual targets individual boxes rather than just saying, you know, this is a VPN is it's behind a VPN and therefore it's secure right? So anytime you want to access server anytime you access a database.

You should authenticate you should prove that you are who you say you are your access should be limited to some amount of time and then next time you log in you need to be able to be doing that login again, and that as part of the access you're not holding any long lived credentials. So what do I mean by that? You're not gonna have a Long Live password sitting on your machine you're gonna have Something like a single sign-on and a multi-factor authentication with that creates a short-lived credential that allows you to access the box, right?

You're not going to be having SSH keys that live on your machine. You're not going to have a password that sits in a password manager that you can just check out and use whenever you want. It's going to be a short-lived credential every time you want to access something you're going to have to again prove your identity on authenticate and only then you will be given access that's the concept of zero trust and the whole idea here is we don't want adversaries walking off with our secrets and using them later to attack us.

We want the real user to have to prove that they are who they say they are every time they want to do something and only then can they actually go and do the thing that they're trying to do. So that's the concept of zero trust and I know that this concept zero trust has been thrown around in the industry quite a lot and it's a little hard to understand what it means because it's been applied to so many different places, but I really encourage you if this is interesting to you. There's the nist 8200 framework defining zero trust that I think is from 2018.

It's quite old and go have a look at that because there are several what they call Key pillars of zero trust and I think those are really really valuable ways of thinking about how do you set up access for users in a way that really adheres to the principles of making it very hard for an adversary to compromise a user today and then steal their credentials and use those credentials later or to laterally move through your infrastructure by compromising one single position and moving everywhere else from that position. Okay, so that's the high level idea behind zero trust in practice. These zero Trust Systems basically rely on an authentication system.

The user has to authenticate. So here's Alice. She's authenticating to this system.

She's asking let's say for root access to the server the authentication systems granting her along lived credential. She pretty short-lived credential a short-lived credential and she's presenting that short-lived credential to the server and the server is checking with the authentication system. Yes.

Is this really the right user? Okay. Great.

I'm gonna let her in right so this is the idea of zero trust you you authenticate you prove yourself and then you get a short-lived credential for access to whatever you want to access. The authentication system comes in lots of different flavors. It can be a certificate Authority that grants a certificate to you.

I'll talk about that in a second. It could be a single sign-on provider something like your Azure ad or your okta your G Suite admin could be one of those things. So One login is another example key cloak so any sort of single sign-on provider can be used as the sort of root for which you base your zero trust system on it's what the user will authenticate to and then another really popular type of architecture can be based on a Secret store.

So essentially you have secrets to all of these targets somewhere in in a Secret store a Secret's fault, you're going to authenticate to the secrets Vault that will give you a secret it allow you to log into this system and then usually that secret is rotated automatically by the system once you've once you've used it once So this is the basics of these types of architectures. And so what I want to do now is go back to some infosec War Stories and show you different ways that these systems actually may not operate in the way you expect. So let's get into that now so as an example of zero trust with the certificate Authority, let's look at an example.

Here's Alice. She's trying to log into server one as root. She proves that she's Alice by her username her password her MFA, maybe her context any number of things for IP address.

Any number of things can be used to prove that. She's Alice. The certificate Authority will say Yep.

This is indeed Alice. Let me list, let me issue a short-lived certificate. So what this effectively is is going to be Alice is going to hold a secret key.

There is going to be a public key that public key will be in the certificate and the certificate will be signed by the certificate Authority. That's what the ca does now this key can be used to set up a cryptographic authenticated Channel between the user and the server and they can communicate securely for as long as the key is valid which basically means if the certificate is good for two hours. They can talk for two hours at which point they can no longer communicate because the server will not trust this key anymore.

Okay, so that's what a certificate Authority would do. A lot of zero Trust Systems actually have a certificate Authority Under the Hood you may not be able to know that because it's not very much advertised. But this is quite typical.

I wanted to talk about how zero trust may look like with SSO. It's almost the same instead of having a certificate Authority. You would authenticate to something like an OCTA.

They wouldn't give you a certificate but they give you something like a saml token again signed by the certificate Authority you present that to the server the server validates this this token if it looks correct, then you can access the server. So it's very similar type of Technologies really at the core. There's not a lot of difference.

Although the implementation can be quite different. Um, so that sounds great, you know certificate authorities sound great. But the thing is that I don't really I have some issue with with certificate authorities and I want to walk you through sort of why we need to be careful when we think about putting certificate authorities into our systems and using them as the basis of our access system or as the basics of our zero trust system.

And so what we have here is our adversary and we have this certificate Authority. This is an incident from 2011 and this actually didn't affect zero Trust Systems this affected TLS the what we use to create https Connections in your web browser. If you remember in 2011, there was an incident where this certificate Authority in the Netherlands was actually compromised and they're signing key was stolen.

And so what the adversary was able to do was they were ish able to issue certificates? For any website, they wanted that were fraudulent. com pretending that they were really the digi note our certificate Authority and used those certificates to actually connect and sorry to intercept security TLS connections to Google so they were able to basically read people's passwords read their username's password reads their emails read whatever that they wanted that the user was doing with Google because effectively the adversary was able to impersonate Google because they had a fraudulent certificate for Google so effectively a total failure of TLS and https all of the things that we think about when we use the web browser.

We're no longer true. This happened in 2011 and what it led to was a huge change in the way that we use certificate authorities in the context of web browsers and TLS and https. Right, which is this thing right here, as you can see this lock over there.

That is https that's happening right there. So if you can no longer trust that lock a Lot of Things Fall Apart, imagine putting your credit card into a website where you no longer think that the connection is encrypted and secured people wouldn't want to do that. So this was a big problem and what actually happened was there was a huge change in the way that certificate authorities were used in your web browser.

There's a bunch of Technologies called certificate transparency certificate pinning things that were made to effectively trust but verify certificate authorities as opposed to what we were doing before which was just blindly trust right now. What's interesting here right caes if trusted blindly are a single point of compromise that is what we learned from this from this incident. And this was really taken to Heart by the the web browser and TLS Community.

We no longer trust ca's blindly, but I will say that, you know, I will say that we still see these ca-based architectures quite a lot in in computer Cloud security and to Center security and access systems because the fact that they're able to issue short-lived tokens really fits well into the ethos of zero trust people sometimes forget that the certificate Authority itself is actually quite a big security risk. And so putting in a certificate Authority means you really need to be careful and protect it because it is a single point of compromise. Okay.

So again certificate authorities single point of compromise, even though they do great things for zero trust protecting the user giving them short-lived certificates all that's great. But the certificate Authority itself is a single point of compromise. Okay, this is another incident that happened in 2020.

Another Watershed incident. What happened in this incident was a variety of things but I want to focus on the point at which the adversary was actually able to completely own their targets infrastructure. And what they did was they actually compromise the SSO provider of the of their victims and what they were able to do with that was actually steal the SSO signing key and issue their own Samuel tokens.

So effectively this is what's called a golden ticket or a golden Samuel attack where the adversary steals the key that's used to create Samuel tokens and can then use that key to issue Samuel tokens to itself. Now if all of your infrastructure trusts this yellow box to issue correct Samuel tokens and an adversary has stolen the signing key for this yellow box and is using it to issue Samuel tokens, then you have a security risk, right? So what we learned from solarwinds amongst other things was really that the fact that this these O systems are a very appealing Target for attackers similar to the not Petra situation again, and that it's very important to keep these protected because they are a single point of compromise.

Right. So again, SSO providers are a single point of compromise and it's important to keep track of you know, what they're doing in the kind of powers that they have in your network. Another incident more recent happened towards the end of last year the Uber breach.

So what happened in that case, we actually had no we have no animation. I think now we do so we have a situation where an adversary compromised a user's login to Uber. They got into Uber they moved around and eventually they found the admin credentials to the secret server that Uber was using now in this secret server.

This was a place where you had stored the credentials to lots of very valuable things like their jira services and like their AWS accounts and the adversary as far as we know was actually able to get into the secret server check out credentials for valuable systems and then try to access those systems. So again, what happened here the secret server is an extremely valuable position in the network that this that Uber had and so once the adversary got in they moved laterally found the secret server and then got into the secret server. And from there they could go anywhere they wanted.

So again, I think you get the theme Here Right This notion of like if you're going to control access with one of these types of architectures where you have this very controlled choke point that decides who gets access to what and stores all the secrets. You've got to be careful that that choke point that single point of compromise does not get hacked because if it does your adversary can basically log into anything. This is what you don't want.

Um, so so what do you do with this? So what do you do with this? Right?

I think that there's a couple of things that you can do. The first thing you can do is just be really aware of where your single points of compromise are and monitor and take care of them right treat them like the valuable positions that they are you have a secret server. You better monitor that secret server right?

You have an SSO provider adfs. You better monitor. That's that that SSO provider adfs because of those are hacked.

That's it. You have so much control from those points in your network that you really need to be careful if anyone is getting into them, right? So I I I'm trying to tell you what I'm trying to say here is that it's not enough to just worry about controlling access and making sure that it's coming from a well a well-controlled authentication system like in this picture.

Right. Let's go back to this picture. Where is it?

Right, we have this the system that's controlling access in a really clean way, right? It knows who should get in it knows who shouldn't get in and it makes those decisions for you. The issue is that this authentication system or the zero trust system is a point of compromise most likely and you need to be careful and monitor and monitor what's happening with that system if you have such a system in place in your Networks, Okay.

The last thing I wanted to do is say that you know, there are other things that you can also do in these situations Beyond just monitoring and what what what we do at Bastian zero and what you can do in your own networks is to have multiple roots of trust right? Here's a very simple example instead of having a single authentication system that decides who gets into what you can have a pair separate independent authentication systems that are that are checked before you give access to something right so you can be using let's say OCTA for your SSO and then you can be using a separate MFA provider that is being checked before access is granted to your to your services what we do at best and zero is we actually sell that sort of in a box where you can you can SSO with your existing SSO provider. And then we provide an independent MFA access isn't granted unless both of these roots of trust are being used.

But you know, really the key Point here is just around like Some people may call defense in depth, right? You don't want to have authentication depend only on a single root of trust. You want to have multiple places that need to be compromised before an adversary can own your networks.

And that's really my sort of final takeaway here about access systems because access systems are so sensitive thinking about the posture of the access system itself is really worth your time. And so let's just kind of go through the historical lessons and then wrap this up. You know giving users standing credentials is long deprecated.

We want to be moving to zero trust postures. Otherwise, we'll be in situations like fluffy bunny where someone hacks our Bastion host or not petia where they steal a single password and start to access all of our servers with that. Um, don't trust a user just because of where they're located in a network or behind a VPN with the example of operation Aurora, you know segment your infrastructure.

If one part of your infrastructure gets compromised, you do not want the adversary moving to the next part, right? You want to keep the blast radius contained within segments that are very hard to move across avoid single points of compromise and software supply chain risks, right? And and what I really mean there is like if you are gonna put these three guardrails these first three guardrails in place you want to make sure that the guard will itself is not so over privileged and so valuable that if an adversary hacks that single point of compromise, then they can own all your networks and we actually saw quite a number of situations where that's exactly what happened with fluffy bunny.

The Bastion host was a single point of compromise with did you know how the certificate Authority was a single point of compromise with solar winds. The SSO was a single point of compromise and with Uber The Secret store was a single point of compromise. And so if you take anything away from this talk, it's that you need to be thinking about kind of those pieces here the control and and security of your users but also control and security of your systems how centralized they are how valuable they are and if they are a single point of compromise, you should either create another root of trust that eliminates the single point of compromise or monitor your single point of compromise so that you can know it's not getting Yeah, and with that, I appreciate the time you can find my information below and if you want to talk more, please reach out.

I'm a real person. I live in Boston. I'd love to talk to you.

Thank you.