AWS Security Director Mark Ryland on How Nitro Architecture Protects Cloud Workloads

Hey guys. Thanks Smith Throw. We're here with Mark Ryland, who's head of security for Amazon Web Services, and we're talking about this novel technique that has emerged for, um, leaking, I guess is the best phrase for it, guest data from a hypervisor and AWS has some ways of thinking about how to prevent that from happening.

Mark, welcome to the show. Thanks. It's great to be here.

Appreciate it. Not everybody is up on this technique, so walk us through what's going on here and what do we gotta do to kind prevent it? Well, if you roll back the clock to, uh, I think 2018 when the member, member of the industry was rocked by the Specter meltdown research, which discovered that through these techniques, they're called speculative execution, where the modern CPUs will, in order to optimize performance, they'll actually go down a code path that's not a, a legitimate one, and then realize later they weren't supposed to do that, and then they can roll back and not express any of those changes to the, the next layer of the architecture.

But that speculative execution does cause changes in the underlying state of the system, and researchers found ways to use that, uh, reality to then do what are called side channel attacks, where you can extract memory without directly reading the information. You can still determine the information through the timing and other mechanisms so that that whole world opened up. And since then, there's been a lot of research on, uh, ways in which these side channels can, can impact customer, uh, privacy and confidentiality.

So the most recent one that we're talking about was a very interesting combination of two previously known bugs. One was called Specter, um, and the other was called a level one, uh, terminal fault. Uh, but no one ever thought to combine those two.

So even though the industry had created some mitigations for, for both those separately, um, there weren't existing mitigation specifically aimed at the two when combined. So that was really the innovative research, uh, that that happened. Uh, but fortunately because of our defense in depth strategy and the care we take in trying to design systems which defend against even, uh, vulnerabilities that we're not aware of yet, we are able to, uh, protect customer confidentiality through that design.

Before we get into that, has any of this been seen in the wild yet, or is this still in the land of researchers have discovered this and we're just waiting for the bad guys to figure it out? I think it's fair to say this isn't happening in the wild or hasn't prior to the publication of the research anyway. Um, it's very, very sophisticated.

These are, you know, advanced computer scientists working on your, or you're working on their doctorates or with their doctorates, um, looking for, for these very subtle and interesting problems in computer architecture. So, uh, once they're revealed, of course, then it is possible that, uh, bad actors will begin to try to exploit these. But I think, uh, it's very unlikely that, that this happened in a while prior to the, the research being published.

So at the core of the AWS platform is this nitro service, I guess that you call it, but, um, what is it about the way that that's constructed that kind of prevents these attacks from happening in the first place? Yeah, so as you, as you know, our core virtual machine service, which it is the heart of almost everything that happened on a cloud, although we build and others build many other layers services on top, in the end, you still need some virtual, some compute system in which, in this case, a virtual compute system, which you can, you can build sort of from the bottom up. And so for us, that bottom layer is, is the EC2 system and the nitro architecture, which is a modernization that occurred, um, gosh, now almost eight years ago when we first launched Nitro for, for EC2.

So Nitro is a complete re-imagining of how virtualization should happen, uh, on an X 86 or, uh, you know, armed processor. Uh, what we do is we, um, strip the hypervisor down to absolute minimum thesis software that only ex, uh, only splits up the PU and memory into the different, uh, compartments needed for the virtual machines. But we offload from the main CPUs all of the other software that you need for virtualization.

So we offload the virtual networking, virtual storage, all the other components run on separate computers co-located in the same physical enclosure, but they're actually different computers with different code, um, much more protected from, from the core compute that customers then run. Um, and one of the things that we did in that, uh, hypervisor design was to try to anticipating problems like this, I'll, we used a technique that's called secrets hiding, which is a kind of a, sounds a little bit more of a, uh, you know, spy game type of name, but essentially what it means is minimize the amount of information that the night, that the hypervisor even has access to in the first place. Because it's possible in the future that if someone can get hypervisor code to execute some instructions on their behalf that could cause problems, then that hypervisor, it's literally not able to see, uh, huge portions of the memory of that computer.

Uh, it can't, there's nothing it can do, for example, but pull the pages that you're trying to examine into the cache, which is how all these life hack work. So we developed this technique many years ago, um, and we modified our version of the open source Kvn hypervisor to do this, this secret hiding technique. And the, the, the good news is when this attack was, um, the researchers launched this attack and successfully exploited on other, both on premises KBM kind of off the shelf play BM as well as another cloud platform.

Our hypervisor already had this protection name, namely that when they were able to get the hypervisor to speculate into, uh, tried to get it to speculate into memory that belonged to other guests, not the one they were operating from, it simply was not able to do that because in the process of setting up the, the, the whole system memory map, the, our hypervisor had removed, uh, all of its act all of its references to the pages of other guests from its memory tables. So that was the reason that this, uh, speculative execution attack failed in our case. Yeah, Sometimes they say luck is the residue of good design, and we talk a lot about, um, building security by design.

Is that what is really isolating all the different components really at the core of that? And that level of isolation is what gives you the protection from not just these types of attacks, but others that might try similar things? Absolutely.

I mean, even the original spectrum meltdown, um, bugs that surprised it, I think almost everyone in the industry, uh, we already had a number of protections that we had just simply by creating a very conservative, very secure by design, uh, system. Even even in, you know, when it first launched, for example, nitro Hyper would never Co-Schedule two different guests on the same CPU, like alternating the running of the code on the same CPU. We'd always break up a hardware into specific CPUs allocated to specific, uh, customer dms.

Um, and if there was only 16 processors on that physical hardware, there could only be 16 virtual machines, so to speak. Um, so that would was one of the ways we already had a number of simple protections. And similarly, there was a technique that was used a lot in the, in the industry, if you have on-premises VMware and you figure, hey, everybody trusts everybody on this, on the system, you would enable a feature call, uh, page coalescing where the hypervisor, when it had fair CPU cycles would scan memory and look for pages that have the, they would, you know, do basically hash each memory page and say, Hey, this is identical to that memory page over there that belongs to a different guest.

And it would actually change the memory pointer tables to point all those references to the, that one same physical page, and you could save a lot of memory that way and, and run more efficiently. But the problem is that now you've created sort of shared stake, shared fate between two VMs and which would enable these kinds of, uh, speculative attacks to be much more likely and more successful. So those were kind of the kinds of design decisions we made early on.

Never, never do those page coalescing never Co-Schedule VMs on the same CPU. Um, and all those protections have stood us in good stead, but this is kind of a step beyond in terms of, uh, the secrets hiding approach, uh, that we were able to bring to market in, in this case was able to afford a pretty, uh, sophisticated attack. Now, there's a bit of a paradox in all of this conversation, at least in my mind, because every time you turn around out there, you'll hear people say, well, we're not moving stuff to the cloud because of security concerns.

And yet when I look in their on-premise environments, there's nothing quite like nitro in there already. And so are they more or less secure on premise? It's seems like maybe less, I don't know.

Well, maybe you shouldn't ask a cloud provider to answer that question, but I have spent a lot of time in my career at AWS, in fact, for many years I worked in our public sector, uh, sales organization as a technical leader there. And so, uh, people asked me, how did you get into the security world? And I said, well, I sold cloud to government for six years and then became a security expert.

That was the only thing anybody wanted to talk about. Understandably, it's a new paradigm, it's a new set of concepts. It sounds kind of scary when you say public cloud, which is kind of a terrible name really for the multi-tenancy architecture.

Um, but the, the truth of the matter is, I mean, I've worked with so many customers over the years and the first they might be reluctant or concerned and, but as, as they gain familiarity, as they experiment with less sensitive workloads and, and kind of set, you know, ramp up their own skills of their own teams, eventually they do come to come to, they frequently come to the conclusion that, Hey, I feel more confident in the security of my file-based workloads than my on-premises workloads. And that's a very common experience our customers have. So I do think that there's a lot to be said for that point of view.

Um, and I think there's a lot of facts and data to back that up. Mm-hmm. Has the mindset of the customer changed a little bit in your conversations over the years?

And I asked you to, because it's Changed a lot over the 12 almost. Yeah, I think 13 years coming up next month at AWS, uh, in an old timer year, big, big change in, in attitude. And, and very seldom now do we, uh, you know, encounter customers that think the cloud is like a scary place.

Now there's still, um, some reluctance or sensitivity if they're kind of new, um, to the environments. And understandably, it's, it's a learning curve. You have to understand the technology, get your teams to understand it, work with them to build the best practice around it.

And of course, there's no magic silver bullet. It's not like cloud is automatically better or more secure. It depends on how you use it, but there's so, there's so many security controls that we just take care of automatically that your experts can concentrate on a smaller portion of the overall, uh, attack surface.

And that is a, a big win for customers. And so even though, um, yeah, the conversation changed a lot, um, and people still have a, a bit of a, typically a bit of a slow start, but once they get rolling and once they get familiar with the technology, then uh, things ramp up and, and they're generally very, very happy with their security. Yeah.

And in that context, are they having that conversation upfront? Because, you know, at least a few years ago, it was always like, let's go put something up in the cloud and then we'll do that, and then we will figure out if it's secure or not. But I wonder if security's become more of a front end part of the process in terms of evaluating which platform to use in the first place.

I Think it has become much more upfront. I mean, in the early days it was what was called shadow. It was an issue, right?

Where teams eager to get stuff done would just open pilot accounts and start working and, you know, kind of outside the purview of the more central it, um, you know, security teams, what, what have you. And now I think that's, um, much less common because the central teams realize that, hey, you'd be better, have some policies in place that make it both relatively easy to adopt cloud, but still allow us to have some oversight and control. And I think the customers we've seen very successful have that, uh, you know, kind of best of both worlds where it's easy to get started.

Um, but once you do get started, they provide you with some central toolings and central compliance and configuration over oversight. Um, and therefore you're most less likely to make configuration errors and what have you. So it, it's, it's a good, it's a good evolution of the, of the industry in now regard.

Yeah. So as you look at all this, what's your best advice to folks? I mean, obviously besides you want them to put workloads in AWS but are there things that you wish more it people would think about from a security perspective that you sometimes maybe you just shake your head a little bit and go, folks, we can be a little bit smarter than that?

Well, I will say, you know, one other thing we didn't specifically mention about Nitro, but it's very important is that we have also removed all human access, no operator access to the platform. So literally, it's impossible for any of our administrators to, to log in and see customer content and data. And that provides a very strong assurance, especially in circumstances such as with our European allies.

We are very concerned about, um, sovereignty, data sovereignty and so forth. And we have these poor services such as our management service and piece two, nitro and others that allow you to build systems that are so, they're so locked down that there's no way for even AWS to see any of your content, any of your data. Therefore, we can't respond to things like port orders with regard to that data.

So there's a lot of technology that exists that, that, you know, people can, can feel, feel and really build that confidence on. Um, but my recommendation is to really think about, um, you know, obviously, hey, use cloud, we have a lot of these powerful capabilities, but even if you're not using cloud, you can adopt some of the practices that have become common in the cloud world, such as a secure development pipeline. Instead of, you know, pushing code or having people log in and configure systems directly, create a, a software pipeline which, you know, has a, a, a, a formal process for code check-ins, um, you know, uh, unit testing, integration testing, uh, deploying the code into a a gamma or staging environment, do some testing there, then deploy the code into production with easy rollback.

All these techniques that people use to, you know, get humans away from directly interacting with production systems. All, all these things increase both, um, operational, you know, quality and, you know, decrease es chance of, of the errors of humans typing the wrong thing, but also security because now you're keeping humans away from the data. The data's only in the production system that humans are working on these pre-production pipeline systems.

Um, and you automate that process end to end as much as possible, and that's possible on premises as well as in the cloud. Um, and so we do see customers adopting those kind of cloudy type practices that using, using those practices, uh, wherever they build and deploy systems. All right, well folks, you heard it here.

The bad guys are always evolving their tactics and techniques, so that's nothing we can do about that. That's always gonna happen. The question is how secure is the foundational platform to thwart most of those attacks even before they get started?

Mark, thanks for being on the show. Absolutely. Yeah, thanks a lot.

It's been great to chat and appreciate and taking the time with us. Alright, and back to you guys in the studio.