Efficient Resource Management Through Systems Design | DevOps Onramp 2023

here Hello, everyone and welcome. Today. We're going to look at Concepts and techniques for resource management, and we're going to look at this from the perspective of systems design.

And what that means is that we're going to see how you can integrate the instrumentation and observation. You need to understand your system's resource usage as a first class component of your services themselves. Briefly about me.

I'm slim virgi. I'm a site reliability engineer at Google where over the last two decades. I've enjoyed building distributed systems for compute consensus and storage and systems for efficient machine learning.

And the efficient machine learning is not about algorithms compilers or other tools. It's about how to build efficient production environments for machine learning, which is famously a very resource intensive part of computing. And I hope to bring these experiences to bear as we get started with understanding how our own systems and your systems use physical resources.

So the important part about understanding the way you use physical resources is that it improves the efficiency the efficiency of your compute systems the efficiency of The Limited physical resources that you have and it may even improve the efficiency of your organization as you'll understand where where you can best apply your time and effort. So in order to understand how you as a someone in devops or in software engineering or in site reliability engineering how you justify the money that you're team spends on Hardware? and Hardware is typically available as shared resources memory Network processors and Storage.

And when I say shared resources, that's because most of the systems that we're building these days are less likely to be on bare metal and more likely to be in the cloud where you're using elastic resources resources are fungible resources are charged as you use them. However, even bare metal systems or bespoke systems that you've built in a data center will be in some sense shared resources because they are tying up Capital expenses that's actual money that you could be spending or your team could be spending elsewhere. So these physical resources these compute resources memory Network processors storage are essential to the software and services that we build and developing and understanding of these resources becomes critical as our systems grow in part because of the monetary cost of building and maintaining the underlying Hardware.

So the building part is the capital expenses part the maintaining or operations is the operating expenses part. and A moment ago I said that many of us are working on cloud services. So even if you don't work directly with Hardware, well you do and we're going to see why.

Because no matter how virtualized your system is the software and services that you write rely on processors memory Network and disk. Yes, somewhere down the stack you will rely on a disc So let's dive into how you can understand what resources you are using and then justify this use to yourself and possibly also to people who are signing checks for the hardware. Now, why does your business want to put resources into what you're building?

We'll also use these Concepts to come up with Elementary guidance on how to make effective changes. And when to know the chain isn't worthwhile. We'll figure out how to understand priorities and apply them after all as systems evolve.

We want and need to manage their cost. We're going to use this service here, which is an image server. It's basically sending images of cats across the internet.

We'll use this as an example. This comes from a tutorial in the Google SRE website each of the components here the upload service the metadata service and the cat picture storage service. It's each is a different component and its own discrete service a microservices and other term.

That can scale independently. All right, so Management let's talk about why? Why do you want to understand the resources that your system uses?

Well other people in your organization might also be contending for resources Just As You Are. Your system or service will likely fall into one of the few buckets. It might be a product and will generate a return on investment.

So for example, you're building a photo upload app and the product will require subscription that subscription is part of how this generates return on that investment. It brings money back in that can balance the cost of the hardware and software that you invest in building the product. Another bucket is you're building something that provides a long-term payoff such as a load balancer photo storage system or other shared infrastructure.

Your system or feature may be experimental in this case. You'll want a shorter term plan with clear exit criteria. And finally your system may be required either by law or statute.

For example. This may be a system to filter for unlawful images. So with these buckets in mind and thinking about the comparative priorities, you'll want to anticipate what happens when your product or service undergoes a priority change.

That is you want to have a system for Reckoning about these changes in priority. It should be something that people across the organization can understand and agree upon. All right.

So now we have our obligatory image of money spilling uncontrollably from a container and this is a metaphor for how we need to be careful stewards of the cost of our resources. One way of understanding this responsibility is by looking at the trade-offs. A common measure of how much something costs to build in the devops and software engineering world is called a sweet hour or software engineer time.

And this is a rule of thumb that says how long will it take for a programmer to make a change? And then if that change results in a resource savings, we can measure that against the resource that is actually saved such as CPU hours Ram Network bandwidth in some cases. Let's say that takes a programmer five days or forty three hours at ten dollars an hour or $400 to make a change that results in a reduction in of 10% in the runtime of a particular piece of software.

So how do you quantify the savings? You know that the software runs more efficiently? But in order to quantify the savings you'll need to know the cost of CPU seconds in your environment.

And there are other metrics as well. A moment ago, we saw that cat image sharing system and that included multiple different microservices. Think about how your equation changes as your service grows in popularity.

Other factors to keep in mind when thinking about the economics of resource management. But that over time Hardware becomes more efficient part of this is Moore's law and the periodic doubling of processor efficiency. In order to take advantage of better Hardware there needs to be an ongoing investment.

And this investment includes such things as designing systems against the new hardware building and testing against new hardware ensuring that you're using the latest software that takes advantages of the hardware and the software might include systems such as build systems compilers and other parts of your tool chain. So that you are you are. Actually accessing new instructions or optimizations that are present in the hardware because if you're writing in a higher level language, you may not be as directly able to access the hardware improvements.

And this is an example of why working on cloud systems can be really good because Cloud providers are continually investing their capital in building out new hardware and making it available through virtualization to you the an engineer in devops. and another factor to bear in mind is it usually moving from one system to another when a hardware changes involved is what I think of as a big project which would cover multiple phases with Hardware changing in one phase Network in another control plane updates and yet another phase and that in itself will become a management Challenge and can also be represented by a cost in engineer hours. So in some cases these projects and different phases may be fungible and you might be able to spend some engineering hours optimizing programs.

So they run more efficiently or in fewer CPU hours. I'm going to take a brief break here and remind you to add questions to the widget on the side of your screen and we'll take a break in about eight minutes to go through any questions have come up and also towards the end of this session. There will be time for Q&A.

All right. Now let's not neglect design. I said that the outset of this talk that we are going to be looking at resource efficiency from the perspective of systems design.

and yeah, here we have a Well, it's an iteration on the design that we saw a few moments ago. And our example from a few slides previous has in addition to the upload service. We now also see a search service and a download service.

We've evolved the design remains similar, although we've expanded horizontally. we also have let's see if I can do this, right we also have This horizontal scaling where we've added multiple. instances or more copies of the same service and these edges indicate that that we're running multiple copies of the upload service search service metadata service and so on.

so our design is we intend it to allow us to gracefully grow in this horizontal Way by adding more services in parallel at each layer and also more copies of each of those Service as demand grows. So without systems that can grow or scale or iterate you'll wind up with software. That is too rigid to manage effectively.

The principle that we're illustrating on this slide is that you can add more copies of download service to search service because they're well understood discrete units. And they're well understood and we'll talk about this in more detail in a moment because we know they're it's profile in terms of resource usage. We can say adding another copy of the download service costs us 25 cents an hour to run.

And that download service is bounded and cost per instance. And I should also say that the principles for that sort of design with microservices and the scaling on multiple axes horizontal for multiple copies and vertical for it increasing the number of layers in an application stack come from the SRE workbook, which is published by O'Reilly a few years ago. You can read it for free online from the Google SRE website.

All right. So now let's talk about how you can get started with understanding the resource consumption for your software systems. And a good place to start is with high budget high energy items.

Where small reductions in these parts of systems will have high impact. And what we see in this illustration on the screen which comes from an article on on little big jobs that we published again on the Google history website a few years ago. And what we see here is that we have fixed costs represented in the red part of these different tasks.

And these tasks they're instances of the same system. So the fixed cost will include things such as the network interface is apis the message passing apis and that subsystem. It will may also include any control plane servers and services the variable costs.

Which are it's kind of another color here. here in the center Are things that? will change from one instance to another and the change is something that means maybe there's different amounts of ram consumed on different tasks because of the specific different images that they're serving and may have cached locally.

And then there's also Headroom the part at the top of the diagram and this Headroom may be just unused space. Or spaced for the variable part of the system to grow. So one of the advantages of combining these multiple smaller instances into a larger instance is that we save on the fixed costs.

That is the cost per task is health because we're running. One task where we were previously running two. the variable costs are here They're additive so we don't really save anything there.

And the Headroom also turns out to be additive. It may be with further optimization that we could let's see if I got the colors right maybe. that we could more effectively use that Headroom, but we've already realized some savings by taking the fixed costs and helping them.

And this is a pattern that I've seen used effectively. I use it effectively at Google when I worked on Google's workplace productivity Suite which includes docs apps and drawings. And this was one of the strategies we used to save a lot of ram across our systems.

And right here. This says that the systems in this illustration are suffering from low CPU utilization. So what we did is we packed the softer more efficiently and wound up with overall more efficient savings.

Okay. One thing however to bear in mind that fewer instances or replicas of a task usually means a less reliable system. So even though our illustration here shows going from two to one tasks in reality.

We might be going from 200 to 100 tasks where our image download server previously at 200 tasks. We pack them onto 100 servers with Savings in the fixed costs and slightly more efficient use of the resources overall and we still maintain a certain level of reliability. Okay now.

Let's take a look at what these systems are doing. Well happily, there's a lot in the systems that's automatic. There's instrumentation that is built into many Cloud providers and it's essentially available for free.

We'll and there are also more in-depth ways to get resource management information through your systems control plane. And application Frameworks the framework that you use for building your application on either cloud provider or in your application development environment. Usually let you know how much CPU memory and network.

Your system is using. You can export this data and aggregate it and then build dashboards from this and then Aggregate and use that data to understand overall system performance. So I'm going to go back.

here and each of these tasks is ideally sending information about how much CPU it's using the network Ingress and egress and the amount of memory and this may even be broken down into buckets over the last Minute five minutes 10 minutes 15 minutes and so on so that you're able to build a nice. dashboard using your your monitoring and operations console with this information. All right.

So here we have some dashboards and all of a sudden we've got lots of data. How do we interpret this? How do we make this easy for ourselves?

All right, so I'm going to leave you hanging there for a moment and we're going to take a look at Q&A while we think about how do we make this easy for ourselves? There's one question in the widget. I encourage you to add questions.

The question we have it says one of the problems I've found is related to Legacy implementations. How should effective Resource Management be done for monolithic or Legacy architecture and implementation. I love this question.

It's a tough thing to answer because it almost always comes down to it needs to be done. In a way that's very specific to the Legacy system that you have. But generally speaking the approach that I've taken is to build a sidecar service.

So it's something that runs on the same piece of Hardware as the well As the Legacy service itself, perhaps not as a privilege user but it reads the it reads system performance from that monolithic or Legacy piece of software and then puts it into a format that can be shared or exported to your dashboarding software. And this kind of a Bridge Approach allows you it's it allows you to Standardized the information across all of your systems or as many as you can and this in fact relates to the second poll question that we have which is about how much overhead does monitoring take. So this will involve some overhead because you're running a different piece of software.

It's usually a small custom built server and This Server will read things say if you're Legacy system is a running on Linux. It might read information from the kernel about the process that you want to Monitor and Take that information from the monolith and put it into a data structure that can then be either pushed or pulled to your dashboarding software. So that's one approach to dealing with Legacy pieces of software is that you write a small sidecar that does the conversion?

The trick is then you also need to modify that convert. I mean you need to monitor that conversion software as well. and but that if you do it effectively can be done as part of your overall strategy to to monitoring and I stumbled a bit there because I got a pop-up with the second poll, which is now active.

But let's take a look at what people said for the first poll. Uhm. It looks like overwhelmingly the answer to how do you build a capacity plan was A and B, which is I guess or I guess I guess and I add instrumentation.

And yes, the correct answer is D which is a and b you guess and you add instrumentation. Initially, you will be guessing you will be guided by your experience and your intuition. But as you iterate and as your service runs in production, you'll bring the guesses more into line with what the instrumentation says.

So initially your guests might be too high or might be too low and you'll converge upon a much more accurate capacity plan over time. And then the next time you build a system you'll have that experience. to bring to the new system that you're building and to assess its capacity and they will begin to understand.

Oh, so serving out a photo cost about this much RAM per request and of that Ram that we're using 3% goes to the monitoring and instrumentation overhead another 1% goes to the network stack and there's a bunch that goes to maintaining connections with our storage system and the rest is for the application and caching on the server itself. com that I put that in there as a gag. com.

We're really my goal here was to say that there is some guesswork and there's a lot of instrumentation involved in this and Over enough time and as you work with other experienced devops people. The amount of guessing will approach zero you might still have some guesswork in there if you're using a new framework or you're trying out a new message passing strategy. But your experience over time your collective experience will come together and really give you very good initial capacity plans.

Okay, so now we're going to come back to what we see on the screen here. And definitely please keep questions coming. We'll have more time at the end to talk about them.

And what we see on the screen here are actual graphs from a service that I ran at Google where we showed the number of requests over time for CPU and memory and this is for an infrastructure service. And this gave us a baseline for the services healthy operation. and This is a kind of graph known as a use graph use all in caps and I try to be clever about that by putting it here on the slide and what this is it represents utilization saturation and errors and it gives you insight into the hardware resources used by the system.

The more importantly gives you a very straightforward correlation between the requests that your service receives and the resources used to handle those requests. So We can see. The CPU is here.

And then the memory usage is on the right hand graph at the bottom. And this establishing a healthy Baseline for the system is really important. A lot of infrastructure systems will look like this a lot of applications that sit on top of infrastructure systems will have a more diurnal graph and I'm sorry, I don't have an image of one of those here.

But it'll look more like an oscillation where over the course of the traditional workday usage will go up and then come quitting time usage dips down. So it's important to invest time in building dashboards that you can use to communicate the Baseline for the important resources for your system. It's also important to keep the graphs up to date and to prune them when they're no longer useful or relevant.

All right, so From our poll. We know that you may use your experience to guide you with a little sprinkle of guessing in there. But what is a capacity plan the capacity plan will describe how much of the physical resources that we mentioned earlier memory Network processor durable storage each instance of the service needs in order to run.

And one of the assumptions that we're making with our service-based architecture. Is that instances are horizontally scalable? So each instance is the same as any other and they're instances are fungible.

So requests can go to any instance of a particular service. And in addition to the per instance resources, there may be implicit or inherited resources that you need to account for. And these might include such other systems as load balancers proxies compute resources that are necessary for asynchronous or batch processing.

And if you have systems that require login, you'll need to account for resources for authentication authorization. That is can this person log into the system. Does this person have access to see this particular cat photo?

And of course there are fixed costs as well because your ecosystem has these inherited resources as well as monitoring and logging themselves. So the capacity plan needs to represent all of these things and your capacity plan should be written down. It might be a spreadsheet spreadsheets are great.

It might be a yaml file that can be ingested by tools that you write. It needs to provide a durable record of your system's resources. In some cases it may connect to a container manager such as kubernetes or some other platform to establish guardrails for how many resources the system is allowed to consume.

And that's one of the great things about having a text-based configuration file for capacity plan is you can stuff it into your container manager and then the container mattresses. All right, I can spin up endless amounts of cat serving image servers until I hit this limit on our resources. and those limits are here in terms of the number of replicas That we allow the system to create.

and as I was saying you don't want to allow your system to endlessly feed your Java server more RAM, that's a problem that we had on some of our workplace applications. Nor do you want kubernetes to create endless instances of your front end server? So in addition to any guardrails that your capacity plan provides you should also have a big red button and this big red button.

We call them generic mitigations. It's a way to just hit stop and inhibit the further growth of your system until you can diagnose what was causing this very rapid growth and address that root cause but big red button is something to have and use and it means that no matter how well developed Your Capacity plan is no matter how deep and mature your instrumentation is you have a backstop that allows you to to stop the systems growth when your capacity plan has turned out to be less inaccurate and your instrumentation hasn't given you sufficient warning of the system's growth or even if it has given you sufficient warning and you're a big red button allows you to stop growth before it gets out of hand. And if you search if you use a search engine to look for the phrase generic mitigations, it will probably find a lot of good stuff written about it.

There's also an article again on the Google SRE website about the importance of generic mitigations and some examples of how to develop and use these another important part of automatic system management and resource management is load balancers. load balancers are phenomenal load balancers allow you to direct traffic among multiple instances of the same service load balancers can be responsive and respond and react to different weighting criteria. In your systems.

So your systems can tell the load balancers. I can accept so many requests of this type and only some three requests of this other type. Whereas another instance might say I can take 300 requests of any type you can get.

And part of knowing how many requests of each type a particular task or instance can handle. Is a result of instrumentation the instrumentation says it has led to an understanding that it takes so many CPU seconds and so much RAM to serve a response to a particular type of request and for each type of request that your system has. it can then quantify this provide that as metadata in the systems control plane and the load balancers will talk to that control plane and and and react accordingly.

and even as your system grows, you might need multiple layers of load balancing and what that means is that here again in our image serving system. We've got a fairly traditional HTTP load balancer. we and this is something that might be very familiar to you.

You could be using a a squid proxy or a piece of Hardware like an F5 Appliance to effectively balance load across multiple instances of the front end, which is right here. However, you even internally within your application environment. You may want to have load balancers that are able to distribute subrequests between different layers of your service this provides protection.

For the services behind each layer of the load balancer so that for example, the upload service can't unintentionally launch a denial of service attack on your own metadata service. It also provides. Good efficient use of the resources within the your internal ecosystem by spreading Loadout effectively.

And in theory Distributing it equally across all instances of the metadata service of the storage server. The load balancer also provides a benefit for the upload service because the upload service can send. multiple identical subrequests to the metadata service take whichever response comes back first and discard or Short Circuit the other requests and this at first blush might sound like a terrible use of resources because you're essentially generating multiple identical requests.

But what you're doing is you're saying this is a really critical piece of information and for my user journey. I want to get this information back to the user as quickly as possible. So your balancing the user facing objectives with some internal cost and the internal cost.

You'll definitely pay for one full request and response. But you may also pay only for a part of the other. sub requests that you generate because you can discard or short circuit does requests when they're in progress depending on the type of message passing framework you're using I don't know if you can hear there are very noisy Sirens outside.

I hope it's not getting through to you, but it made it almost unbearably noisy inside this room. But when we put all these together, we have pieces of software that need to work together. And these are software ecosystems or just systems.

We called each of the pieces of software service or microservice. We also need a way for people in our organization to agree on priorities and these priorities help us understand the different benefits of software and systems across the entire organization. What you're working on needs to be placed in context with what other teams that the company may be working on even if ultimately it's all part of the same product.

There are different costs associated with back-end processing with synchronous real-time user facing requests and the batch processing may be things like generating thumbnails for cat photos that have been uploaded recently or deduping photos where we have multiple copies of the same exact photograph. But having a language and an agreed upon set of priorities also helps an organization understand how it wants to use its resources and having this policy and implementing it in code is a big step in in agreement on those policies also allows you to understand how to shift resources when the systems behave unexpectedly. Unexpectedly often means when a system grows a wildly in popularity.

And finally, the element of Design Systems need to grow and scale effectively. Otherwise, they're too rigid to manage effectively. And this is one of the characteristics of what we call Legacy or monolithic systems where you have a really beautifully designed or beautifully put together system, but it's all just so you can't make another one of it without invoking some very expensive design and build process.

So it might be a bespoke or custom it might be a custom built piece of Hardware. It might be a very expensive piece of Hardware the long lead time. And what we're what I'm encouraging is for servers and services that can use either off the shelf or cloud-based components so that they can scale.

At the touch of a button or with the ingestion of a fresh up-to-date yaml file. Then what's next for you? Well start by developing a model you might have a spreadsheet to start with or small Python program.

Write down the constants that you can identify the cost of a CPU second the cost of a software engineering hour. Then add in the variables. How many instances of each particular piece of software do you need in order to run your system to its expected service level objective?

Build dashboards with the use method for utilization saturation and errors use these dashboards to see how much CPU your system your service or system consumes. With these dashboards, you can establish a baseline for a service and then track how different experiments or new features change your services Behavior. You can also see what happens when you add a load balancer when you add a caching layer.

You can track and evaluate these experiments. Then finally look to the data to understand your system take this data to new generations of the system and you can see how the data iteratively helps guide improvements in your system and its efficiency. all right, so that's My feel for today, I welcome questions.

We've got time for Q&A and I'm going to pop over to that tab. Now. We've also got On the screen you can find me on Twitter at Salim or at Google SRE.

google. google has a lot of tremendously good resources. pretty much everything I referred to is available on the website the generic mitigations the system design the book on system design and the article on how to combine jobs from small into big ones in order to be more efficient Okay.

So now there's a question from there is a couple of people are very enthusiastic about load balance. And yes load balancing is awesome. And then there is a question that says load balancers are really great.

Should we consider latency as a metric while choosing or deciding on layers of load balancers? Yes. Absolutely.

Your load balancer should add close to zero latency for each request that passes through it and acceptable additional latency in a load balancer is in the single digit might milliseconds. And most of my career I've been working with custom-built load balancers, but even before that. When I worked with Hardware load balancers, these load balancers add very little latency to the system.

And this is done mostly because the software the load balancers keep a map in memory of which back ends or that is which targets are available and they're estimated capacity. More current and sophisticated load balancers maintain a an active two-way communication with the backend targets so that they have real-time information on the availability of the back ends to provide service for different types of requests. And so for each method or request that's represented by an incoming message.

The load balancer has an almost instantaneous decision to make about where to Route the request. So yes, absolutely consider latency as a metric, but regardless of the latency the load balancers do add some RAM and CPU and network cost to your system. A and I'm working on another version of this talk that includes latency.

And so I look forward to sharing that with you all at some point.