Patterns and Practices for Building Resilient Applications at Cloud Native Now 2024

Hello friends. Hope you're all well. Today we are going to talk about the patterns and practices for building resilient application, but just before we do that, I would introduce myself, Opitz Sha and the, the senior partner solution architect in AWS.

Um, I've been working for AWS for more than three years now, helping our global system integrators to help customer migrate their workloads from on-premise to cloud. And then obviously, resilient is one topic which always comes down, um, on the cloud. So today we are going to talk about all the different patterns and practices which you should consider when building the resilient applications on the AWS cloud.

So why resiliency? That's the first question which gets asked. So as you know that the cost of downtime, um, even a Fortune 500, um, applications, if we look around thousand applications, downtimes cost it's billions.

It's every year we have been spending billions. With respect to the downtimes, this is some general figures which we have been listed down. Average cost of per hour of downtime, 470 4K cost per hour.

For critical application failures, it's significantly more so when critical applications, which is impacting the customer that cause significant and above all the loss of trust of the customer. And when it hits because of the infrastructure, it is currently forecasted as a hundred K dollars per r. Yeah, so some significant numbers of downtime and hence why resiliency is quite important, uh, onto this uh, uh, aspect.

So yeah. So what is resilience? So what does the mental model looks like?

So basically failure is inevitable. Yeah. Failure will happen.

So what we need to be working toward is how quickly we can respond and recover from the failure. Yeah, high availability, disaster recovery, both are those kind of, uh, uh, mental model in which we can achieve the, uh, how quickly we can recover from the failure high ability. Make sure that if a service fails in, say for example in one our availability zone of AWS or in one data center on-premise, we are quickly able to recover into another data center, or the application in the same data center can be recovered onto a different virtual machine.

Yeah. Similarly, disaster recover. Most of the time we have seen on-premise that it's from one data center failing over into another data center or on the cloud from one easy to another AC or one region to another region.

We are going to talk about what factors needs to be considered into the disaster recovery at, to the upper point and what what is we need, uh, as part of the continuous improvement. So what we have often seen is that we recover the service and we forget about that. The new model of continuous improvement is there to leave for all of the high ability and resilience things as well.

So as we fix the defect, we need to make sure that we have the right operational model in place so that the issue doesn't happen again. Yeah. And it's all modeled through operations as a code.

So where does it sit into the shared responsibility model for the AWS and customer? So as you can see at the bottom, hardware, compute, storage database, networking regions, observability zones, edge locations, all of those are the responsibility of the AWS for the resiliency and where the customer is running the workloads on top of those hardware and services, which has been provided by AWS is the customer's responsibility. Yeah.

So for example, how does the application behave when application fields from one AZ to another AZ from one VM to another vm, what does the change management, which needs to be looking like when the failure happens, what kind of monitoring of the application needs to be placed in place so that the, uh, the correction action has been taken, whether automatically or by involving some people into the chain so that we have the right, um, uh, recovery model for the resiliency of the application, resiliency of the cloud. So a culture built around resilience. So we have the service ownership model.

Who owns the service? What is the operational readiness, which looks like for a service before it goes live? We do, we have the safe and continuous deployment of the applications onto the cloud as well as the infrastructure and what processes we have for the correction of error.

Yep. So as I said, issue is inevitable. Issues will happen.

How quickly we can recover from those root cause and prevent it to reoccur those quite important aspects which we need to be factoring into. So building that culture that where failure is accepted and how we are improving the service is quite important, uh, in the organization. Categories of failures.

So we have seen failures can happen because of the code deployments, code configuration. Somebody did the configuration change into the production to fix some issue, but then when the new code got deployed, the configuration was overwritten and hence the failure happened. Again, core infrastructure, it can be failure of the host on which multiple VMs are running.

It can be the data in the state, the da, the data got corrupted and hence the database went down and hence we were not able to complete the transactions. The customer was not able to place the orders and so on dependencies. So, um, it's very often we have seen that one application is dependent on another application.

Another application is depending onto the third party application. And if one application goes down, the whole application for the CUS and customer is not getting resolved and hence it fails. Um, we have started testing towards the, uh, highly unlikely scenario.

So what happens if a war happens? We need to lift the data. What happens if there is a flood into a particular data center?

Are we able to recover from all of those scenarios? Uh, but those also needs to be factored into as part of the, uh, failures. So AWS well architecture framework for, uh, operational resiliency is in place, which helps, which guides you to understand how we can build and deploy faster, how we can lower or mitigate risk, how we can make the informed issue and what are the best practices about that.

So that's something which is quite important. If you haven't seen that, please go through that framework. It helps you to build across those aspects, design principles.

So, uh, it's very important to say that how can we, what kind of issues when it happens, we recover from failure. Yeah. Um, there was a company I was working around and they had the scenario where they can do everything automated, but then because there was a lot of payments which was involved, they needed their customers or the third parties to be involved before they take the decision to say that we are going to be making the cutover from the DR perspective.

So how, how do you recover from the failure is quite not just important from the technical perspective, but also from the process perspective and recovery procedures. So have you tested the recovery procedure? Have you got the right people in the frame to know that how you are going to be, uh, recovering from those processes and so on?

Yep. Um, how you can make the change. Is there something which is a change, which is required?

Who needs to approve that change? It's quite important, uh, in terms of those aspects. Um, are you able to perform the operations as code?

Um, uh, it is always preferred that you don't have anything which is not, which is manually getting done In the new world, you should have everything from monitoring to alerting your all the changes to be done as code. Yeah. You can refine the operational procedures as frequently as you want.

That helps you to make sure that you have fixed it once and it is fixed forever. Yeah. And you learn from all the operational failures.

So if a team has encountered saying, oh, what happens when one VM goes down? I need this kind of a monitoring of the CPU memory and I need to alert that, then the other teams can just leverage those failures, which has been learned around. So it's quite important to make it and run it as operations, as code, what we are going to talk about.

So we are going to talk about how do we anticipate the, uh, uh, issues, how do we monitor it, how we are going to learn it, and how we are going to respond it. That's basically the continuous resilience mental model, which we need to be start thinking about. Yep.

Um, as you can see in, uh, our Amazon, we, we, we, we learn a lot from our prime days. Um, we had significant amount of CPU utilization that increased 325% in 2018 prime day. And in 2024 it is just going high.

Yeah. Uh, 39% of we reduce the cost. So it's not just about how we can do the things, but how we can use the tools to make it more efficient and then help to save the cost as well.

So, um, uh, Amazon Code Guru is a service which can be built into your, uh, pipelines. Uh, in order to make sure that you have, you have the right code reviews in place, you have optimized the any expensive lines of code, and you have got the performance and cost cost improvements, which has been running into the production environment. And once it's running into the, you don't need to be an ML expert in order to run the code guru.

In fact, you don't need to know any aspect of the ML machine language machine learning languages in order to use the Amazon code guru. And that's the beauty of it. You build, you measure Yeah.

Some patterns, uh, back off and retry. So, uh, if, if there is a failure, you don't immediately give up. You have a retry after two seconds, a circuit breaker where the circuit can be open and closed and we will go, we will go through that into a bit of a detail.

Graceful degradation, that's another part of the thing where system is running, but system is quite slow. Yeah. So how do we recover that throttling?

Um, so our system is overloaded because all the systems are hitting my backend system. The same backend system is getting overloaded, and hence not just the non-important transactions from the customer is impact impacted, all the important transactions are impacted as well, and load sharing. How do we make sure that we are able to share the load equally for the important services and the non-important ones?

So microservices, um, as you can see, the customers in is looking to book the hotel, uh, before they can book the hotel. The flight needs to be there, the payment needs to have made happen. There is a reward service, there is a trip and recommendation services.

All these formulate our travel plan for the customer. And when a failure happens, network failure at the customer end, we don't know anything about it saying, we still say that, Hey, you know, uh, our systems are looking great, but it might be that the customer is not able to connect. So we should monitor things like what is the average transactions we are getting at that time of the, uh, day, and if that time of the day, if the transaction are falling, what is the reason?

It might not just be internal, it might be the internet has gone down. It might be the DNS resolvers are not working. So we need to be keeping an eyes not just internally, but externally from the customer's perspective as well.

And I was talking about the retry. So you try a service once, doesn't work, wait for some time and try again. Uh, that's the kind of a retries with backup service throttling.

So in this case, the reward service, which was getting called from multiple, uh, vendors, if you take it in that way, then they were all hitting the reward service and, uh, the reward service was not working because it had an issue. So what you do, you throttle that service. You say how much transactions are going to go through, and I need to control that.

Uh, I can, I, it's not something which is immediately, which is required to be told to the customer that how much reward they have earned. But we can do it over the period of an hour as well. It doesn't need to be real time.

AWS step function comes into the rescue, as you can see, that you can have multiple lambda functions, uh, where, uh, process reward is an Lambda function. If it gets successful, great. Proceed to the next step.

5 seconds and then if it's failed, then you mark down as a failure. So step function allows you to iterate through the failures and wait for a certain amount when you are using Lambda, uh, and other patterns in order to make sure that you can process the request asynchronously for the customer's transaction to be successful. Synchronous call is another problem.

If, as you can see the recommendation service is slow, the customer is impacted because the hot hotel service is going to run slow. So what should we do? Recommendation services, high latency timing out customer is not happy with the transaction and might leave the website and might not use it.

So application timeout is one of the common theme. Um, what we need to be looking around is we should put the so breaker pattern. So customer comes in, go to the trip service.

If they found that the service is failure, open up the circuit. Yeah. Um, and try it after three seconds.

If the circuit is closed, now that means the recommendation service is responding back. You can send across all the services, all the, uh, calls to the recommendation services. Again else, you just book the trip and return back to the customer to say, Hey, you know, uh, we are unable to book the trip right now because we are not able to serve you the recommendation.

Or at the same time, if recommendation is an optional service, you can just say, okay, customer knows what flight he wants to book, what hotel he wants to book, and I just go and accept the request from the customer without showing the recommendation. So circuit breakers, it times out periodical retry as we were talking about, uh, which is there. Uh, if it's a, a success, then we return back the success after the periodical retry when it has recovered.

And this is the mental model of the how the circuit breaker once works basically. So lambda function makes a call to the step function. Is the circuit status, is it all open?

Is it closed? Is it updated the circuit? If it's running fine, then go to the success step and keep the circuit closed.

If it's not running fine, then change the circuit when you're executing the lambda timeout, update the circuit status to open in the DynamoDB and then continue the uh, process till the time it works. Set cellular architecture, single region. So, uh, many a time we have seen that there is a same copy of the entire domain.

So you take around the EC2 instances, RDS or the applications which are running on to the EC2 instances are need to be available onto the multiple availability zone. Yeah. Uh, and and why we do that is that if one cell, if one group of AZ services which are running in one AZ goes out, you can quickly recover into the another az.

Yeah. Uh, and that's something which happens. Failure can be happening because of the, uh, the blast radius.

Uh, it could be that, uh, you are trying to reduce down how much the impact will be if that one is is going down. Yeah. Or one is has been hacked or there is a DDoS attack, uh, onto a specific VMs.

So you reduce down those kind of, uh, uh, possibilities as well by controlling and deploying those things again. So, um, what we do blue green deployments, uh, you might have heard about, it's very common into the industry that you try to deploy the new feature. Should I deploy it into all the, um, uh, so the orange ones are the one which is the chain changing servers, and should I deploy all of them at one go or should I deploy it in a rolling pattern so that if one is successful, the second deployment has been gone through and does it looks okay.

After that, I'll continue to deploy onto the another, uh, another application. So was, uh, before, uh, it it, it creates any more issues. Yeah.

So you validate at the same time you are deploying the application and you are rolling it out, not just all in one go, but you're rolling out making sure there is no impact to the customer. This was kind of a thing which was quite, uh, a change. When the cloud came in saying previously we were always running onto the servers and on the servers someone will make the change in the night and morning new admin will come in too, and they're doing the deployment of the new applications.

And that way we were, uh, updating the same infrastructure, uh, by deploying the code on tweet. And what happens is when that some issue happens onto that VM and we try to build that VM from the snapshot, it is change. It's no longer working.

And then lot of hours, which was going on to the downtime, here's where immutable infrastructure, infrastructure as a code come into picture and say, Hey, you know, you don't need to, uh, do the things with respect to update the things onto the fly. But you always have the infrastructure which is immutable. That means you create new EC2 instances or you create, you run new lambda functions along in the parallel and then you take out the other ones.

Yeah. You don't need to update any part of the infrastructure. That means you need to start managing as a code.

Uh, as I was saying that you can have the redirection of your, um, the, the switch between the new feature and old feature. And this is all supported by the A LB uh, application load balancers to say that how much percentage of my volume I want to redirect to the old application and how much I want to redirect to the new version of the application before it can be, um, rolled out into all the application servers. Moving on strategies for disaster recovery.

So from left to right, um, the simplest to the complex one because of, uh, how the real time application needs to happen, what is the cost impact and so on. The disaster strategies can vary. Now, for the premium services, which is your tier one services, it's always either warm standby or active active because you want them to be recovered quite quickly to the time where it is only the HR system or, uh, noncritical systems.

Uh, you, you can retrieve in the 12 hours or 24 hours, but you want to make sure that it is, um, cost effective. Then you use the backup and restore. Yeah.

So RTO and RPO recovery point objective, which is how long, um, uh, I can have the data which can be lost. So should I take a backup every hour? Should I take the backup every four hours?

That will determine how quickly you need to restore the application. Yeah. And recovery time objective, how much time I have to recover my application without the data loss.

That is RPO. Yeah. Or with the loss basically.

So obviously the higher the RPO and RTO, the higher the cost and the more we are moving towards the right of this slide. So active active gives you the lowest RPO and lowest RTO. It's near zero basically to the point where the backup store, that is a recovery point objective.

You might see that from S3 Glacier storage. If you need to recover the data, it might take six hours, eight hours depending on the size of the data. And then you might see that for six to eight hours you are not able to run your application.

And it might be six to eight R still data as well. Or if you're taking the snapshot every R then it'll be one R of RPO data, the date, the point in which the data is lost. Resiliency hub.

So this is where, um, most of our customer came to us and said that, you know, it's great we, we have the, we know the RT and RPO, but we don't know how long it'll take for us to recover. And hence we developed this application called resiliency hub, which will help you to inject the fault, which will help you to say what is the R-P-N-R-T-O for that particular application is looking like. And it helps you to validate your application resiliency.

So have a look into that fantastic service and it helps you for a lot of use cases, AC failure, EC2 instance failure and so on, which you can test it through. Um, as I was saying, it's uh, what should we doing in the assess recommendations and operational recommendations and so on, which is a part of the key capabilities of the resiliency hub. Moving on to the monitoring most important aspect, we need to detect the failure, investigate the failure and respond the failure, um, measure the customer experience.

We always starts from customer, uh, not from the systems because when you are too ingrained into the technical details to be captured, you don't know what the customer experience will be looking like. Yeah. So always measure from the customer experience and then detect and risk want.

Yeah. Uh, your health metrics should I should be, am I failing? Is the customer is not able to make the transaction?

Is am I seeing the last number of payments going through? Yeah. Those are the kind of health metrics of the system, not something like what is the uh, disc utilization at 2:00 PM in the afternoon?

Is it going to be similar at 2:00 PM tomorrow? Yep. So that's also is a metric, I'm not saying that's not important metric, but always try to focus from the customer perspective 'cause that tell you the real picture as to what will be the metrics which needs to be monitored in order to quickly identify what the customer will be facing on to.

Yep. Essential metrics. So customer experience, metric impact assessment metrics and operational health metrics.

Yeah, it's quite important to implement as those are the essential metrics. Synthetic monitoring, um, where possible try to implement saying, Hey, customer is running onto the website or mobile app. What is that we are seeing right now?

Is it two millisecond? Is the average response time for a call with the DNS lookup? And then one second is what this transaction takes.

But is that a, is that something which is slowing it down now because we are getting less number of transactions but the customer is complaining that the app is running quite slow so we need to start putting some synthetic monitoring in order to monitor the website or the API endpoint. Yeah. And the client to server side basically don't just monitor on the server side corner.

Those days when we were just monitoring our infrastructure and we were saying that, Hey, you know, my infrastructure is looking great, we have all connected world now. So real user monitoring, uh, something running onto the browser, which can give you the uh, uh, monitoring with respect to the web application and what the server, you can correlate those resources with the server side and you can get the alert if there's not an optimized user experience. If the page loading is taking time, you should be able to know on the Chrome because of the new version which has been released is taking quite long when the Firefox is working fine.

Yep. And try to reduce the mean time to recovery. MTTR, it's quite important to, uh, make sure that we are always going towards the, uh, we have the measurement around how long we can have a particular page to load.

Yeah, we can, we cannot have something which is not tested or something, which is just made up on the fly monitoring in the cloud. So as you would have seen from the AWS CloudWatch, uh, we have metrics, logs and traces, all part of important, uh, uh, pillars of the observability. We need to make sure that metrics are there, logs are there, traces are there, which can be captured on two.

And these are all the different observability suit. So we have a lot of AWS services, uh, CloudWatch in the domain, but we do have, uh, customers using either third parties or they're using managed Grafana open source, Prometheus open, um, the open source services for the log aggregations, x-ray for the monitoring of the, uh, transactions and so on. DevOps guru.

So, uh, as I was saying that it's quite easy to use, just plug a D and you will start saying, if something is not benchmark, something is not working as normal as it would've expected it. So it captures all the metrics and you fit it into the DevOps guru. Uh, even with the deployments, it'll start capturing the change and if something is not normal, then it'll start responding on to it.

So in the response side, so what does it looks like? So you might have something where an event happened, um, a failure yet it's a known failure and while the fix has been worked by the development team, you want to create some process that when this issue happens, when there is a 60% of the memory utilization happens on the Java, then I should restart my application so that there is no impact to the customer. So you can have those time-based event, uh, or some log-based event.

Uh, so with the rules so that you can execute that function as lambda function or a step function to restart the application S3 bucket. So if you leave the public bucket, you can have the configuration AWS config capture it that someone has created the bucket, which is public, and I want to enforce it to be a private because buckets are not allowed, uh, because of the security constraint. Same goes with SSH.

If someone creates the security group, which is quite open, you can again have the remediation through the config, capturing the information and create an event. And through that event run the lambda function to close down those public SSH access ports. So system manager, um, if you have heard about it, that's basically a lot of, uh, things it does for us, including the patching of the servers, the incident management, the change management.

You can create your own run books and can execute those runbooks and you can have the response plans into the system manager. And there is an incident trash board as well. So you can track your issues defects, um, any ops items which needs to be integrated.

And it can also integrate with Jira and ServiceNow for the change management point, prepare response plan. That's quite important. Monitor and detect issue and respond and resolve.

Yeah. Uh, it's very important to have those things in place, uh, in order to make sure that you are geared up for the resiliency of the system arc. It provides you with this ownership.

So what happens if one A goes down, you can still redirect it to the another two A. Um, our CEO ex CEO, uh, he was always saying, um, people work with the good intentions, but the good intentions never work. You need to have good mechanisms to make things happen.

And that's where we have the correction of errors in AWS. We follow it to our heart. We need to make sure that we have the right correction of error mechanism in order to resolve the root cause, but also make sure that we, we are making sure that it is getting fixed at the root cause.

Yeah. It's so that it doesn't happen again. And that's one of the reason it's been quite successful.

What should be there as part of the COE? Um, and the five whys, which is very important. Ask any issues when it happened as to why five times to get to the root cause.

That's one of the very important aspect. Yeah. Um, QoS engineering, um, which is quite important.

Um, the process of making sure that you create the scenarios to disrupt the application and see how the system is responding. Yeah. Uh, what happens if the CPU utilization is quite high?

How does my application, uh, behave? What if the number of users increases? What if one server goes down?

Those are kind of things which is quite important aspects. So steady state system, you have a hypothesis, um, saying, okay, my system will be behaving well if it's a all two servers out of three, that's a hypothesis. We need to run the experiment.

We need to verify that it does as what we were expecting and then improve. It might be that the three servers are required to run all the time and not just two servers in order to make sure there is no impact to the customer. And it is continuous.

The reason being, as the number of users increases, you might see the increase in the number of transactions. And hence you need to make sure that your hypothesis, which was at the start of the project, is still standing true. So for that, we have the FIS fault injector simulator, easy service, and it helps quite a lot in terms of making sure that you're able to replicate the real world in adults, uh, test regularly, game days.

Uh, we do quite often game days are quite important. Uh, exercise your procedures, make sure there is no impact to the users. And similar, the exceptional event.

Yeah. So, uh, it's not just technical as I was saying, you need to make sure that you know which people to reach out to. Uh, someone needs to make clinician.

If, if it's a failover, uh, you need to create the DR and you need to move over to another region or another. A, what is the process which needs to be followed? Yeah.

Who are the people who are going to run that process and who are the people who are going to make sure that the systems are coming up? Rightly, is it we do we need to perform any internal test, external test before we sit down, that it has been working as expected and thus the ops people knew those, those, all of those things. So we do have the operational game days as well.

So as we were saying, resilience in the cloud, anticipate, monitor, learn and respond. Quite important aspect of the resiliency in terms of achieving the continuous resilience. With that, thanks a lot for your time today.

It was pleasure to talk with you. Any questions you had, it was, I'll reply back, which you have put into the chat, but please do give your feedback of the session, how did you felt about any, any aspect of the fit. So scan the barcode and feel free to reply back onto any of the aspects which you need the answers on.

Thank you once again. Signing it off.