Francis Cordon – What is Causal AI for DevOps and Why Does IT Need It? – DevOps Experience 2024

Hello, my name is Francis Cordon, chief Customer Officer for Costly. Today we're here to talk about coastal AI and why it is needed for resilient cloud applications. As we go through this presentation, feel free to ask your questions in the chat and they'll be taken care of.

So we need to understand the problem that we have at hand. I actually believe the problem of application reliability and building environments that are resilient is a timeless problem. However, it is undeniable that businesses are changing rapidly and thus application management has also to change.

Environments are more dynamic. The scale and complexity of the applications is just growing and operating Cloud Native tech stacks has absolutely sur robust human capacity. So here we have our friend, let's call him Mr.

Pip, the little bird sitting on no bird sign. And we're going to have a very interesting this conversation about why correlation is not concession. And so first, let's talk about humans.

When you see this bird on an no bird sign, we want to believe the bird is just being contrarian. It wants to be on an no bird sign just to be contrarian, right? And we know that that's not true.

It's pure correlation. There's no ion here. But this proves a very nuanced point and that is that humans are wired to think in terms of causation.

We are wired and as we grow as babies and little kids, we think of why, why? That's how we operate. Now let's talk about machines and let's talk about correlation versus causation for machines.

If we were to feed a gigantic computer and large language model data from 2000 to 2009 on mozzarella consumption, and I raised my hand as a lover of cheese of all sorts and how many PhDs were awarded for civil engineering in those years, nine years, the machine would have to conclude something's going on because there is clear correlation. And yet most people, a 10-year-old will tell you, obviously there's no causation here. Just because we've had correlation doesn't mean one thing caused the other.

So what's really happening, the examples we just covered are very interesting if you think about them because one, they look, you know, funny, humorous, but on the other hand they show that humans and machines operate very differently. And there are examples that correlation is not causation that can even stump the human. For example, we all think of colds associated to cold and there is a correlation of more people getting colds in the winter when it's colder, but there's no causation or at least medical science tells us there's nothing there.

Inherent in cold to create more calls, you can um, you can research on this topic is actually really fascinating. So I will uh, quote this book, the book of Why by Julia Pearl. Highly recommended very eye-opening in terms of causality and how deep a topic this actually is.

And this quote specifically states that machines lack of understanding of causal relations is perhaps the biggest roadblock to giving them human level intelligence. And even though we're all in all of L and what we can do with machines and certain types of ai, which is a very generic term today, the reality is that there is a big gap in terms of being able to operate with causality and to think in terms of causality. And yet it happens to be as we have seen, one of the most important signs of being human, one of the wi ways we are wired to think and to operate.

Furthermore, I'd like to refer to this study that is a collaboration with several entities like University of Michigan, Hong Kong, meta ai, uh, recent and of course feel free to scan the QR code to get to the source, uh, paper. And the conclusion is that there is really a shortcoming, at least as of yet with LMS in terms of causal inference scales. In other words, machines don't do causal inference very well.

And uh, when put in a test, they show almost like a random performance. It's not really the forte of technology to think with causality and to do causal inference. At the same time, garner is telling us that causal AI is a key enabler of the next wave of ai.

And why is this important? Because we're trying to move AI towards creating decision automation, autonomy, and I love this term common sense without causality. There's going to be the big gap between AI and us, but that also limits how we can use this technology as a copilot in our life as an assistant.

So that causal AI on the rise, uh, and key to the topic at hand. So now let's get more specific. What is causal AI for it?

Or we like to call it causal reasoning. How can we set some practices that are really, really tangible? Every one of you listening can put this into practice today to help you both from an application reliability standpoint and building resilient environments.

So our first suggestion of this causal reasoning is that we need to do the work of capturing human knowledge in structure and abstract data models. In other words, there is knowledge about cause and effect relationship in technical stacks in environments that we need to capture. And we'll give an example later so that this is very clear.

And then we need the ability to apply these generic models into the specific environment at hand. So we need to things, we need to be able to understand causal models and what is the behavior of entity types from a root cause and symptom or effect standpoint. And we need to be able to apply this specifically to our environments with all the complexity of the dependencies that we have.

Now, there is a temptation to always blame the resources, the infrastructure. How is my CPU, how's my bandwidth? How's my memory?

Because it is easily observable and because it's easy to ize. And there is this trend to think that applications are always dependent on these resources, but the reality is sometimes is the other way around. Sometimes application behavior is creating problems on the other tiers, including the infrastructure tier.

It can go in any direction and as environments become more and more complex with more and more dependencies, many services that depend on each other with dependencies on infrastructure resources to these correlations and these um, uh, the correlation of what causes what is actually not obvious, that's why we need to think with causation and not just correlation. So having high CPU on a server does not imply that that is causing the problem on the application tier. So what can we do to see this in a practical term?

All of you following this application may already have an environment that you can apply these to. Um, and so I would suggest an exercise if you do not, I will give access to an exercise that you can do at home to see this firsthand, but essentially less work, work with microservices. We're gonna use Kubernetes for this example, but this is of course, of course technology agnostic and we're going to attempt to see a process by which we could capture the causal relationship between root causes and the manifestations, which are the symptoms.

You can scan that QR code, do this at home. It's a nice environment. We at costly created this just for people to be able to play with it and to uh, draw conclusions and learn.

It's called chaos mania and it's essentially a microservices application with multiple services. It includes a front end card services, shipping services, and several scenarios in which various things can break or become slow and they propagate. So this allows us to see how they propagate, what are the symptoms and what, and we'll have to infer what is the root cause for those symptoms.

Now let's set the stage. The first thing we need to do very important, uh, also for everyone following this presentation to remember is that we are going to consider everything you can see a symptom. So if you have an observability, a monitoring solution, application performance management, if you're looking at things like your CPU or your latency, all of those things are things you can observe, you can measure, you can have metrics, you can have alerts.

Those are all symptoms. Then we are going to understand what is the infer root cause from those symptoms. Now let's think about it with an example.

If you are coughing, I can see that you are coughing. If you go to the doctor, it is observable it, you are coughing. The inferred root cause if you have other symptoms too, is that you have a cold.

What I can see is that you're coughing, that you have a headache, that you have aches. What I infer is that you have a cold remediation is only possible when you understand the root cause. And this is the famous example in which you, you are taking a painkiller and you're not actually tackling the root cause, you are tackling the headache, but the headache was a symptom.

The root cause is is the cold. So what are some of the things we're going to attempt to do? We need to capture knowledge.

And again, this is not just data. This as we have seen, is not just feeding data to an LLM system because today those systems are not able to establish causality and we're going to try an attempt to infer the root cause based on the seed. So here's a practical example.

You can follow along if you have an environment or do this at home with our Ks mainly a boutique application. And so things that I can see, what are some things that I can see? I can see CPUA memory on my notes or CPU on my Postgres database or high memory on me on my Redis cache.

I can see a slow queries, I can see the queue size growing. I can see impact of the DB queries that do the shipping or I can even see sophisticated things if I have the right monitoring, like high mut weight for lock-in mechanisms when you have multi-threaded or slow transactions or a service that is simply not available. And I have a set of root causes.

These are not observable things, these are inferred conditions that cause the manifestations. And we can start mapping this and I'll provide a couple of examples following this, but this is our, our our the work that we need to do to start mapping this. When you have a problem with your cluster size, you may have high CPU with your knot size, you may have high memory or high CPU or even by propagation impact your post grades database high CPU or your cache memory.

So I'm going to build this for the sake of time, but essentially through observation and by measuring our symptoms and inferring the root cause in controlled environments, I can build this database of knowledge of root cause and effect behavior. So let's see how we do this. In an example, if I have a scenario in which I'm receiving alerts, let's start with the alerts.

Something as simple as I'm getting two alerts, high memory utilization for a node and high CPU utilization for a node, but I have my monitoring and I can see more things. These are now manifestations symptoms and for example the high memory ization on the Redis container is certainly what's creating an alert about high memory and the same with the CPU. But what can cause high memory on my cache, it could be the growing Q size and the consumption of the message here that is too slow can be the cause of that.

And in turn the fact that I'm processing transactions slowly for my shipping service can be the cause of that. And at the bottom you can see the same logic for the uh, CPU and even another branch. If your memory continues going high, what's going to happen that you have an out of memory kil and thus your cache will be unavailable and your cart will be unavailable, meaning you will have errors.

What happens is the user cannot even use the application or the service cannot even browse if it's um, browser based. Up to this point we were talking about observable things and dependencies to alerts, 10 symptoms or manifestations. All of those you can see with your systems.

But by understanding my environment, I know there's one root cause that I have the application with poorly optimized code and that's it accesses data too frequently. Now if we read this in reverse, it makes a lot more sense. What happened was that I had poor optimized code in my application accessing data too frequently and this is what's creating causing slow transaction processing, which in turn creates both slow queries and slow readiness message consumption, which ends up creating the high memory and high CPU that gives us those alerts and even possibly the fact that users cannot browse.

So what I've done here is understand the relationship between one inferior root cause 10 manifestations and two alerts. And I capture that. Now I know what can happen when poor optimize code behaves on my readies, on my database, on my uh, various components of my application and I can do the same and for time I would build it for the sake of time with another problem.

What's very interesting about this problem is that the root cause is completely different, but many, many of the symptoms and one of the alerts are identical to the previous ca case. However, the overall set is different. So I have an application that is not handling tx locking.

So that is when you have multithreading and you have access to files data and you have to create a lock condition. So the data is kept stable and consistent, but if you don't do that well, you create transaction processing. Too much lock-in threats are waiting on threats.

That in turn creates that the product catalog obligation has a high mu wait time that's observable. So what's interesting is when I look at the set of symptoms, there is a lot of similarities with the previous case, but it is not the same. Now I've been able to capture two cases with different finger, different set of fingerprint at different profile, even those similar.

And I can infer through looking at those symptoms whether I have the extensive mutt lock-in problem or root cause or I have the root cause that I had before in which I had data being accessed to frequently slightly different profiles with symptoms complete different root cause. So what have we learned? Let's do a, a set of very, very practical key takeaways.

I want to provide takeaways that you can put into practice today with whatever software you have. As long as you have an application that has multiple components and this is, uh, interesting to you. The first thing is you should have a center of excellence or a group of DevOps practice or however you want to title it, but start capturing this knowledge, this human knowledge.

Start understanding the difference between symptoms as observable things and inferring its root cause and create a set of group of symptoms that correspond to what root cause. So as you build that library, you'll be able to understand your root causes. Now something is very interesting here is that I think many people default to meantime to resolution as the main KPI of success, meaning I'm doing this well if I can troubleshoot faster.

But in reality we're missing a very important, um, metric of success, which is the sheer number of incidents. If you remember my two examples in the first one, we had two alerts and 10 symptoms, but one root cause to be even if it took me a while to fix that, I am dealing with one incident, not 10 and not the two alerts, the reduction of incidents and how fast I can fix it now that I know the true root cause. Those two combined are really the benefits of doing this well and it's worth doing this process even if it takes us a little bit of time.

And we apply this knowledge proactively to also build resilient environments to you and your center of excellence, your DevOps, your engineers start thinking costs with this causality in mind and understanding these relationships. Now, when we build now when we add new features, now when we add new components, we're also taking this into account to build resiliently. So problems don't happen to begin with, which is the vision and is where we should be going aspirationally.

However, to be frank, it is quite a bit of work. I think it's worth that everybody can try this and do it. Uh, otherwise we're really blindfold and it's really difficult to do this work.

But even if you try to do it the right way, it will take quite a bit of time and most of your environments will be beyond human scale. So another possibility, if you want to experience a software that can help you doing this automatically, then you can try costly if you're interested and you have um, URL there to try it. Thank you very much.

I very much enjoyed talking to you all about correlation is not causation and how to apply the principles of causal inference and reasoning to obligation, reliability, and building resilient environments. I am Francis Cordon with costly and thank you so much.