Steve McGhee – Building Reliable Services on the Cloud

Hi everyone. My name is Steve McGee and I'm here to talk to you about building reliable services on the cloud. This is actually a book that we wrote.

It's a very small book this big. And it was authored by a friend of mine Phil Phil tishler and then myself and Challenger helped him out. So a little bit about myself.

I'm a reliability Advocate. I was an SRE inside of Google for 10 or so years. Then I left Google.

I became a customer just like you guys and I was you know learning how to use all the cloud providers and how to make reliable systems on this crazy Cloud thing. And that's my job now is to help customers do exactly that you can see the link at the bottom to download the PDF of this report. But what we're going to talk about today is kind of an overview of the whole thing.

It's gonna be mostly like a visual overview. We're gonna go through a bunch of diagrams. It's not gonna be a super deep review.

If you want to go into the depth of it. Just click that link and get the PDF and you can read the whole thing. So we're gonna go over first year you if you want to design resilient systems or reliable systems, you have to know what it is that you're going for.

So you're going to Define your objectives. You're gonna then need to know what it is you're working with so you need to identify your dependencies. That's when you start architecting your service.

That's when you build it out. And then once you're running it, you want to make sure that you avoid some common failure modes. So first off like, why are we even doing this?

We know that outages in Internet systems your road trust, right and they get customers to leave or or adopt Alternatives like your competitors things like that. And that's that's no good. Right you want to make sure that you keep your users even worse than that is data loss.

Most of the time data losses really really bad, right? It can destroy trust not just erode it. So Google's been doing this kind of work for a while, which is, you know, we call it in general reliability for you know decades and so we've built some pretty reliable Services you may have heard of Things like Google search and maps and Gmail and Android and all these things.

So we've learned a bunch of things and here we're just trying to present present a few of them and a structured Way To Think through them. So who's this for? Hopefully this is for you.

If you're looking to build a reliable service in your software engineer or an SRE or or a cloud engineer or platform engineer or someone who's managing managing these teams this this is for you. So first we want to make sure that we understand our objectives what it is. We're trying to achieve reliability is what we call a non-functional requirement and you can have a little bit of it or a lot of it and what's important to know is is how much you want to invest in this and generally this comes down to cost and effort and it's a trade-off.

So it's important to Define ahead of time what it is you're trying to actually achieve so we use something called slos. Here's a bunch of tables that you can look into the the report to understand a little bit better. But essentially what it says is that you're able to use this concept of an SLO to decide just how available you want your system to be on the far right with the little green dotted line is that is the the nines that most people kind of are familiar with if we have you know one nine we have nine days of outage per year, but the other columns are helpful too because sometimes we don't have complete failure.

We have partial failure. 1% or even one percent failure forever and it's fine which sounds a little counterintuitive. So take a look at at this model to make sure that what you're actually investing in makes sense.

Another thing to consider is you should think about your system in terms of data planes and control planes. Sometimes we also talk about management or administrative planes as well. Those generally have a different expectation of availability.

Generally, your data plan is something that needs to have high availability and your control plane can have you know slightly less so think about what parts of your service are controlling what other parts of your service before you decide to like lay down one blanket SLO for everything. Because it might just save you some effort and some money and one rule of thumb that's important. Is that every extra 9 that you add to a system tends to cost about 10 times as much as the last one did so, you know, the money really sort of goes up into the right.

So be careful don't don't just start with, you know, wanting nine nines from the beginning because it's that's a Fool's errand. Okay, so next up we want to make sure we understand failure domains and how redundancy helps with us. So a failure domain is is any group of resources that can all fail together and cause that system to no longer be functional.

So an example of this is like one computer right like a host a machine or it might be Iraq or a row or cluster right in Cloud. We have like zones and regions. These are all failure domains.

So the underlying concept here is you want to make sure you push your changes into smaller failure domains when possible and we try not to like make changes across failure domain. So we don't want to step into puddles at the same time. So the picture on the top right shows that when we push a configuration to different servers a safe way to do that is take a copy of those configurations.

Sometimes we call those flags and independently push to those two servers so that way we're not altering two things at the exact same moment because they we might mess up they might both come down at the same time. That would be bad. So redundancy is the ability to Have failure to service in different failure domains and have them sort of like work for each other.

So if one goes down you can use the other one often we say, you know, if you have two copies of one thing like it costs twice as much but you're only really getting one out of it. And that's true. So often we suggest that people take have more than just like, you know, two for one kind of redundancy instead you have, you know, many copies of a thing and and the way to think about this is is we talk about if n is the number of copies you need to service your users that you want to have n plus two, right and this doesn't mean and times too.

It just means you want two more copies of the thing in case you have two failures at the same time. So you could have one planned maintenance one thing, you know, you're you're gonna take down and then at the same time you might have one unplanned maintenance at the same time. So that's kind of like a good rule of thumb in terms of planning for failure and you See what the curve on the right n plus 2 becomes cheaper and cheaper relatively in terms of utilization as you add more instances.

So this is just kind of a standard curve you can you can interpret yourself. Okay. So what about the scaling factors of the application itself?

Lots of times we use what we call algorithmic Big O notation. Sometimes this is called asymptotic complexity classes if you're into you know computer science and this is when we say things like oh of one of login and log in and squared things like that. So what we're saying here is your system has a whole probably follows one of these scaling factors as well.

And so if your system currently falls into like an N squared scaling Factor, one of the best things you can do is to fix that right is to is to push your system into something like end login or an Ole of and scaling Factor system because just that alone will have a tremendous impact on the scalability of your system. So trying to scale up and end squared system is way harder than trying to log trying to scale up a login system. For example So at a high level what are the ways we scale things up.

We call One Way vertical scaling and another way horizontal scaling and we'll actually get into this further in just a minute. So once we have the basics, we have to kind of know our dependencies. So what are our building blocks?

What are we working with? So the in the cloud like we mentioned we have regions and zones and multi-region. We call these scopes at a high level.

So it's important to understand that services and resources in the cloud tend to tie themselves to a scope so you can say like, you know, a VM tends to be live. It lives in a Zone which lives in a region which lives in like, you know, the global or multi-region space, but the the VM itself is scoped to a zone right but you might have some sort of like replicated data store that is regional, right? It lives in a region and it can survive the failure of any single zone in that region, right?

So this is the idea that that service lives in a region not in a Zone technically. So as you go lower like if you go into a Zone those Services tend to have lower availability. And as you go higher, they didn't have higher availability, but it's important to point that there are a lot more zones than there are regions, right?

So even though it has a lower availability there are more of them. So if you're able to spread your work across many zones, each of those has a lower availability, but as a total, they might have a far greater availability if you play it, right? Okay.

So one thing you have to consider as a developer as a architect potentially is you want to make sure that you align your failure domains. So you should be building what we call Stacks. We want to make sure that your Stacks are aligned with the services or they're aligned with other components in there in the stack and that the the services that they depend on are similarly scoped.

So if you don't do this, if you do what we call misalignment this can result in coordinated failure across domains, so this is bad, right? So this is the picture of the top right? So if we have one database that's in one Regional domain and we're using it from two different Regional domains.

If we lose that database we lose both the layers both the stacks and that's that's no good. What we use instead is a regional domain database and we have it replicated to a different regionally domain database and we lose the top one we can still use the bottom one and the users on the far left through the load balancer. We'll still be able to use the system.

So this is much better. Right? So the important thing here is like this is actually the same.

System done in two different ways, right? So it's actually the same stack but it's using what we call a different archetype, right? So it's it's a it's a different architecture.

It's following a different pattern. But all the code essentially is still the same. It's just that you're you're placing it in a different in a different format.

So if you look at the bottom right corner, there's a paper written by Calder and barenberg which is which is really good and it gives you a bunch of different archetypes that you can base your system on. Okay. So once you've kind of got the basic idea of what you want to build and what you're going to build it out of you should look at the the systems that you're depending on so cloud services provide slos and they kind of give you a sense of this is available for you know, two nines or three nines or four nines of the time.

What you're gonna do is you're gonna compose these Services into your system and it's important to think about do I need all of these systems to be up in order for my system to be up or do I need one of these systems to be up like am I using a replicated system of which I only need one or am I using independent systems of which I need all of them and if you're gonna do this this is important to to understand that the SLO math kind of works differently. I give another talk at a different conference that you can you can look up just called SLO math that describes exactly this and the point is is if you can use Union availability where we're just saying only, you know, at least one of these needs to be up you're gonna be have a much better time. If you need all of them to be up this is intersectional availability.

It's Charter so do your best to try to use Union availability instead of intersectional availability. And then another thing is getting back to the control and data planes. We want to make sure that you are able to have your data plane only depend on another data plane, right?

You don't want to have to have it depend on on multiple control planes. That would be a problem. Okay?

While you're at it, you're gonna pick some more services. So the top three are compute Network and storage. Here's just some some hints from the book for example in compute.

Make sure you use containers and make sure that they can start up and come down quickly. This is very very helpful when it comes to a flexible reliable system in networking. Honestly, try not to roll your own too much when it comes to networking use the providers CDN and load balancers and service meshes and things like that because networking is really tough to get right especially in a really distributed system in storage again take advantage of the provider.

So use things like object stores and no Sequel and multi-real databases instead of trying to you know, build your own just on on Raw VMS. This is just, you know, this this tends to work better using things like pub/sub services and Flume and and mapreduce and things like that are all so really good. Good ideas, and in terms of making a large scalable resilient system.

Okay, so now we're on to architecting our service. Here's kind of the three main points. So what you're going to do is you're gonna take your large system.

You're gonna decompose it into smaller components. Right and we're gonna call each of these a service the large thing. We might call it like a product or an app and each of the little things inside of it that compose it are called Services.

Now, we're gonna make sure that we're using well-defined interfaces to provide loose coupling between those services and generally we just refer to these as apis so these services are going to talk to each other through apis. These are abstract programming interfaces. And the point of this is we want to make sure that the aggregate system the whole thing the whole product or the whole app behaves and performs as desired.

So this this process is we sometimes we call this operations or something. That's it's scaling right? Like how do we make sure the whole thing is able to to grow over time and still be successful?

So here's some service Evolutions you may have seen in the past. So we have like a two or three tier service model. So you've seen the three tier sort of a historic, you know, very common pattern which was having the presentation layer the logic layer and the data layer.

I think I got that right to the right here is what we call SOA for a long time. So the service oriented architecture, these are good, right? This is you can see the evolution from two to three to SOA and now we're on the things like microservices.

So in microservices, we have much smaller Services. They have their own data stores, right and you can you can each of these can be owned by its own individual team. Um other things you can consider make sure you're aware of the difference between synchronous and asynchronous workloads.

So what's what's the difference? Well, the important part is that in a synchronous workload. You might need it for strongly or causally consistent systems, but it requires tight coupling of the server and the client which means that the client has to kind of sit and wait if it's waiting for something to to be accomplished by the server, whereas an asynchronous service the client can send a request and and go off and do something else.

Right? So the client is no longer blocked it can make it can make progress without waiting for the server. So any case where you know a server is is stuck doing something you you really don't want the client to be waiting around doing nothing during that time.

It's better for the client to be able to make some forward progress. One of the ways that we can get through. This is using something like a pub/subq.

That's just a method for getting system getting messages through a system as you can see. There's a few different Mechanisms we call these like at most ones at least once are exactly once executions. You can use all these for different reasons for getting your messages from from a server back to a client and avoiding a lot of headache along the way.

Another method that we suggest looking at is a batch computing. So having a general purpose batch processing system is really helpful, especially with large amounts of data. So often we say if you have a large amount of data and you want to be able to make a request against it are you want to have a user maker request against it and get some some output often.

It's too slow because the system is just the amount of data is just way too big think of think about Google Search, right like the whole internet. So if you made a query to to Google search and it had to go, you know crawl the entire internet and and do all the math and come back again with an answer. It wouldn't work.

It would take too long. So this is batch Computing is doing all that work ahead of time and building something like an index. So that way that you have this snapshot of it's like a like a phone book of the internet pre-built, right?

It was built using a batch system ahead of time. So that way when the requests comes in you don't have to look at the whole internet. You just look at the phone book and give the fast answer.

This can be much more efficient. Obviously when it comes to things like the side of the internet, but any other kind of large data data store It can also be much cheaper, right we can use things like preemptible compute which is types of compute that may or may not go away but during a batch processing. That's fine.

Like if you use if you lose a machine for a little while you can just retry and and it doesn't matter. It's also important to understand this is getting back to horizontal and vertical scaling that we mentioned before vertical scaling can be much more cost-effective if you know your cost structure. Each node does have a fixed cost and if you add more nodes, you're always adding those more fixed costs.

And so you might save a little bit if you're familiar exactly with how this is going to work horizontal scaling is actually much better for cases where you're going to scale very fast and maybe in a not very well understood way you're able to scale both the the traffic that you're serving the requests as well as the storage that you're you're putting onto onto disks somewhere. So this is important some methods that you can use for doing this we call sharding and replication to make sure that you have enough copies of the data across enough machines where you don't have things like hotspots, which we'll get to in a minute. And while you're at it if you're gonna be spreading stuff across a bunch of machines, you also need load balancing.

There's a few different models here. It's important to to point out that if you're going to be using using things like automated deployment and auto scaling you're going to need a dynamic load balancer, right? It's not something that you just kind of, you know, fill out a form once and you're done forever.

It's something that needs to have a feedback loop built into it. So there's some systems that are some load balancing algorithms like waterfall. There's also ones that we call like either query cost or capacity cost or some other ones that do a little bit more logic around like who should I send the next request to sometimes you have a homogeneous system where all the requests basically take the same amount of effort in that case waterfall works great.

It's really easy go for it. It's easy. But if you have some requests are very expensive and some are very cheap.

You might want to consider something like query costs. There's also another kind of a hack called utilization-based costs, which is just watch all the servers themselves. And if they are like highly utilized then send them less traffic this this is a simpler way.

Of calculating query cost and specifically calculating the capacity of Any Given service or server. Sorry of providing a service. Okay.

So what are some common failure modes that you want to avoid? There's a bunch of them. These are actually some principles that we use inside of Google with some examples in front of them.

So like change supervision doesn't mean a bunch of humans watching aboard it means basically monitoring. So we recommend that you alert based on slos which we talked about at the beginning but also having like synthetic end-to-end probes making sure that the whole end-to-end system works, even though you're not measuring the entire complexity of all your user classes at least having some examples of like, you know, hitting the front door and making sure it goes all the way down to the database and back up again, you know periodically doing that, you know every every minute or something like that. Progressive rollouts will show more in a second safe and tested mitigation.

So it's really important to be able to perform mitigations. One of one of my favorite ones hormone. The most useful ones is the ability to to almost always roll back.

I would say always roll back but you know, it's there can be exceptions the ability to say like a change was made stuff went bad and just say okay just roll it back. Push the button push the big red button without having to think about it. That is a really great ability to have if you have to sit and think and say, oh, well you will it be safe for this one to roll back like no we want to always be able to roll back that is extremely helpful capability for for any platform to provide.

The last one is defense and depth. So one of the easiest things here is to make sure that you at least you're you're deploying not straight from your laptop to production every single time, but you're you know taking some tests along the way right? So you're deployed to it like a Dev cluster in a staging and prod and each of those they're like copies of each other and tests are performed along the way so you deploy not by just like, you know sshing and YOLO and your way into production, but actually you you Can code into maybe like an IAC system and it mentions a 2fa and MFA.

So we want to make sure that the person that's checking code into the IAC is actually, you know, the person who is employed by us and not someone else. So it's a you can be very sure that the right person is checking in code to the right place in the right tests are being done along the way so all of these add up to the point where we're able to have a lot more confidence in the changes that we made. Here's gradual rollouts kind of in a graphical format.

So what you want to do is start small, right? So when you make a change to your system, you don't want to change all the production all at once often. We call this like a canary release.

So the first one might be at the top left corner here in region a and wave zero, so we're gonna say 10% of traffic is gonna get the new thing and if that one fails then 90% of the traffic doesn't get hurt and they're safe and it's fine so we can roll back region a and figure out what happened and move on. With this. This is a really effective way to especially have something that is a multi-tenant to be able to you know, you might put like your internal traffic in wave zero, right?

You're you're test systems might be in wave zero or something like that. And you know, you're most expensive customers are like the most sensitive customers might go away over and wave three. So this is a great way to spread change both across time and across space right and by space, I mean like different regions around the world.

One of the most important things in here is you'll see that we never touch the same region at the exact same time. So like region B and Regency aren't overlapping their, you know at different hours of of the same day. Okay, and then the one is avoiding cascading failure.

So this is this can be a really gnarly one there are cases where you know at the very top row well, you know one instance goes out and then traffic moves over to the other servers. Oh it took out another one move down took out another one and all the sudden all of our servers are dead. Right?

So sometimes we call this like a query of death or sometimes. It's just overload. And so this is a really important to just be aware that this can happen.

And one of the things you want to do is just have methods in place to mitigate this you can see at the bottom. We have a bunch of terms. You can you can look up cost modeling Doss protection load shedding having quotas and criticality blah.

One of the most important things you can do though is have a way to explicitly drop some portion of traffic. So you're basically going to have to say like, I'm sorry, but one quarter of our customers are just gonna have a bad day, right so that way the remaining, you know, surviving three quarters will continue to survive or to continue to use our service that can be better than taking down. The whole system so having a way to just drop traffic or excessive traffic in this case is super important.

Another thing that's related is something that we call thundering herds. So some of we call this like a wall of traffic. We'll just show up and this can come from all kinds of reasons.

Like for example, the World Cup going to halftime. This is a very kind of funny old story from Google is that we would see spikes at halftime when everyone would get up from their TV and go do something on the Internet. Another one is like it could be your own marketing.

Maybe you launched a new campaign campaign and it was very effective and we got all the customers to come and try to sign up at the same time. This this can be not great. Right?

This can be a problem. So some of the models that are some of the systems that we use to fight against this our exponential back off and Jitter. So this means that the requests that come in the front door if they fail they shouldn't just try back right away.

They should try back in an exponential Manner and they should add a little bit of random noise random time delay to that retry this this helps immensely and finally if you if that still isn't working that same Capability of being able to drop traffic is again important here. So being able to just shave the top of that curve off and say like that's it for today. You know, the wall has to be cut off at some point.

What we're going to do here often is have these methods in place to allow for our autoscaling to catch up because these walls are way too fast for any kind of Auto scaling or caching to really be effective. They all get swamped. And so what we're really trying to do is just buy ourselves some time.

We're trying to smooth out that curve so it's not a wall, but it's more of like a ramp. Okay. Another one is hotspots.

So we want to prevent hot spots. So this is this is when often we'll see something like if you Shard your data by, you know, the you know, if it's an app store you're starting it by like the app ID, but then what happens when an app gets really popular and they release a new version and everyone wants to go get the new version all in the same day because it just launched and they had a big marketing event. You can see how this relates to the Thundering Herd problem.

So one model the way we can use here specifically for getting data like from a back-end store system or a database or something like that. There's something that we call gating so gating is a system. It's described better in the book, but it allows you to batch up all of the requests that are similar to each other and instead of making, you know, a thousand similar requests.

You can make one request that is all the same shape and get one one response back and hand it out to 1000 people. Right? So if you can do this in a hierarchical manner you can take what was 30,000 QPS at the top and turn it into one QPS at the storage layer which sounds incredible it only works for some types of traffic, but it's an important ability if you can if you can use it.

Um, this is a big one. This could be its own talk it data Integrity is super important being able to Define what you want to achieve or what you think is achievable by your system is really important in terms of backups restores and replication. So you have to understand that, you know failure will happen.

So when it happens, what do you expect to do? Do you want to be able to have the ability to if you allow yourself some flexibility in terms of like it will take us a little while to recover and we might not recover everything that has changed this gives you some flexibility in your response. So we call these two lines RPO and RTO.

You can see the definition here. If you want both of those to be zero where you know, as soon as the failure happens, we lose nothing and you don't even notice that there was a failure that's really great if you can pull it off, but it's very expensive and it's very difficult to do so consider if you really need to do that or not. If you do need to there are ways to do it it in.

A lot of replication and you know load balancing essentially. Another thing you can do while you're at it make sure you're using probes to detect loss or even corruption in your system. So you should be measuring your system kind of as much as you can to detect any of this failure because you won't it won't often be super obvious.

If you do have any kind of data loss or data corruption. You don't want to hear from it here about it from your customers directly. That's kind of the worst case.

Okay, so given all that here's sort of the high level guidance. Design your system with an archetype in mind right set your extra your own expectations of the reliability and choose the right model. Look at that paper about archetypes.

Next up is you're going to build your architecture to model that that archetype you're gonna choose the right Services. You're going to map against the right failure domains. You're not going to step over failure domains, right?

Then you're gonna write your code. You're going to decouple the system using apis. You're going to choose some things to be synchronous some to be asynchronous.

You're gonna use some batch and you're gonna consider what it's going to take to scale the system up over time. And finally you're gonna make sure that you're holding yourself to the slo's then you're handling failure as well. Right?

You're gonna build mitigations into your system and you're going to practice practice. So we call this chaos testing or dirt testing make sure that you know how things fail and when they do you can recover from them because it's gonna happen. And I think that's it.

So, I'm Steve. Thanks very much for listening and big thanks to Phil and everyone else at Google that helped develop this material. Bye.