Cloud-Native Security Challenges with Xebia’s Filip Chyla | SecOps Vision 2024

Welcome to Cloud Native Security Challenges with the Hospital 10 list. My name is Philippe Hilla. I'm a cybersecurity shepherd.

At Xia, I help organizations, uh, answer the question, how do we do cybersecurity? You can find my contact details on this slide. The QR code will take you to my LinkedIn profile.

If you have any follow-up questions or if you wanna, uh, ask something that I might have missed or add something to the presentation. Let's start with a quick agenda. Uh, for this, for today, we'll have a quick look about at Cloud Native and what it means.

Uh, in the context of this presentation, I would like to establish some basic vocabulary and a common long language and context. Uh, after that, we'll take a deep dive into the, uh, part of the top 10, uh, or ask cloud native security list. So a quick disclaimer.

I will mention different tools, uh, make sure that they're applicable for your use cases. Not everything will work, uh, uh, as described within your context. That's extremely important, so you apply your own context.

Uh, to this, uh, to this, to the solutions that I provide, I, I do not endorse any specific tool. I do mention, uh, open source tools. I do mention some commercial tools.

I do mention tools that are built into, uh, the, some of the cloud providers. Uh, the lists are, by all means, non exhaustive. New tools come out.

All, all tools get depreciated. So, uh, please do your own research and make sure you check again with your, within your context if it's applicable for you. So, uh, what is cloud native?

So there's multiple definitions, um, and, uh, yeah, asking different people will, uh, tell, will give different people will give you different definitions of what it means. So, for the purposes of this presentation, I like this definition. So loosely coupled systems that are resilient, manageable, and observable.

Combined with robust automation, they allow engineers to make high impact changes frequently and predictably with minimal toil. So that's the more, that's the most important part of why people and organizations move to the cloud, to be able to stay flexible, to be able to develop fast, to be able to make high impact changes, and to be able to profit from the ability to grow rapidly and scale on demand. But with that comes, uh, some challenges.

The, uh, landscape grows extremely fast. We have multiple cloud native applications released every other week. Uh, so, uh, I think I believe the land, the CNCF is currently managing over 300 different projects on different at d, which are different levels of maturity.

io, you can see and track the changes and see what's actually developing in, uh, in cloud native applications. And the goal for security is to enable the business and to help them actually use all these applications. Uh, but with great power comes great responsibility.

So when we're talking about cloud native security, we're not just talking about cloud security or web application security, we're actually talking about a combination between them. So it's inevitable that with our need for speed, we need to be able take on more responsibilities as a team. So we do not have to wait for the platform or for security or someone else to do things for us, or enable things or deploy things.

So we want to give the developers a lot of power, but again, that comes with a great dose of responsibility. A bit more on the background. Uh, here is an example, uh, diagram of a monolithic, uh, application.

This is the one way of developing application where we package everything nicely into one, uh, big chunk, uh, big chunk of code in a monolithic architecture. And we deploy it usually on one single server where all the capability and all the abilities, uh, are built in to the application. And all the functions communicate internally within the application.

So that makes it a bit hard to track what's happening, where unless you have really well-defined logs and every single function effectively logs itself. So for the purposes of security operations, uh, you might have an additional challenge convincing, uh, developers adding certain logs to your, uh, to the application so you can actually extract data that's valuable for, uh, for security operations. So the evolution of underlying infrastructure allowed for the evolution of our applications, you start managing parts of the platform in order to deliver the service efficiently.

Nowadays, the application is built from many different components in a microservice architecture, and each one can be written in a different language and stored in a separate code-based repository. They can also use different backend service. These microservices are installed on containers, managing orchestration platforms such as Kubernetes and are deployed in the cloud, uh, which can be public cloud, like A-W-S-G-C-P, Microsoft Azure.

You can also run Kubernetes on-Prem yourself. And for the purposes of this presentation, Kubernetes will be the main orchestrator I will refer to. Uh, this approach changed not only the application, but also the risks and the way we, they are managed.

So, data can be passed between multiple services, be before it ends up in a dangerous function with the addition of the event bus, uh, where all the information flows through. Uh, it might process the information and process the sy the calls in such a way where it passes a, uh, piece of data that will actually, for instance, exploit, uh, a service without us even realizing it. Because the event bus is a higher privilege service that can simply pass the data.

So we might have to do additional work, uh, on the front of the microservice, be it on the IPI gateway or on the way users input data into our services. But cloud native is much more than the application itself. So it is actually the way we build it, the way we deploy it, the architecture in our case, uh, in this, in the case of this presentation, the microservices, the way we package the application, again, for the purposes here, we use, I refer to containers, our way of working also.

So this is a common overview of the components involved in a cloud native application. Cloud native is also a whole journey. So, uh, I I would like to invite you to visit, uh, the cloud native Trail map from the Cloudnative Foundation to get an idea of what it actually entails, what components you might be using, especially from a security perspective.

The addition of all these additional bits and pieces generates more and more data that, uh, we need to keep track. And especially for security operations. All these data sets are extremely important.

And understanding how these data sets can feed us the information that we need is extremely important. So the OSS Cloud native, uh, security top 10 is a, uh, one of the newer project of the OSS Foundation. It started in, uh, 2021.

It is very much alive. Uh, we're still working on expanding the project. We're looking for contributions.

And, uh, yeah, it's the, it's main goal is to provide some common challenges that the community has, uh, figured out and has seen out in the wild. So, uh, we can better understand what actually working with cloud native applications mean. So I would like to go through part of the top 10 list and probe, uh, uh, for some techniques and ideas on how to address each challenge and, uh, share, share my, uh, knowledge and my choice for, uh, actionable items.

So, starting with number one. Uh, so in insecure cloud container or orchestration configuration. So, uh, an application is not a cloud native application with one major thing, the environment, the application is deployed on.

So many breaches in the last few years happened, you happened due to misconfiguration. So public packets over permissive security groups and misconfiguration on the, uh, control plane of the Kubernetes clusters, uh, permissions that are not properly set up, uh, pipelines that have not been properly secured and logged down. So for that, we have, uh, we have had actually ways of securing it in the form of, uh, simply having proper baselines in place.

We can address the challenges of, uh, insecure cloud container orchestration configuration by applying proper baselines. Organizations like the Center for Internet Security has been building, uh, security baselines on a, uh, as a community, uh, where people come together, figure out the best action points, the best practices, and put them down, uh, for everybody to see, and for everybody to use and adjust as necessary. So there are multiple tools for our cloud environments we can use, uh, to be able to actually scan the, uh, code that we, uh, write.

In our case, we're gonna be looking at infrastructure as code, uh, tools like checkoff, uh, Aqua Trivi and cu. They will help us, uh, do during the development phase to scan our application code and the infrastructure code or any possible deviations from the standard of our choosing, be it AWS, be it Google Cloud, be it Azure. Uh, it'll provide us with the information, uh, that it'll provide us with information that will give us the sense of security, and we'll be able to validate if the infrastructure meets our expectations.

Uh, the next, uh, problem on the list the community has recognized is improper authentication authorization. With the, uh, i, with the identity becoming the new perimeter, I expect this, uh, this, uh, problem to actually rise, uh, on the list, uh, in the next, in the coming year or two. So, uh, this starts from authentication and authorization from the infrastructure to the application code itself.

So anywhere when you need to pass credentials, uh, or login, uh, is based falls under this, uh, challenge. So the application doesn't perform or impro improperly performs authentication or authorizations checks on attempts to access a resource or perform an action, be it an authenticated IPI on the microservices where you're exposed too much or we don't, or we, uh, do security bios, security where we try to hide certain endpoints, uh, rather than actually, uh, authenticate and authorize access to those, uh, services. So, some, uh, over permissive cloud IAM roles, which is quite common when, especially for, uh, human operators, where the role over time just grows and grows and grows.

And we'll use track of the actual permissions, uh, the user has. Uh, if we run our own Kubernetes, uh, cluster, uh, quite commonly, uh, the lack of orchestrator node trust roll, uh, means that anyone can add a random note to our cluster without us, uh, noticing and without any additional authentication. So for this problem, uh, we have the principle of least privilege.

Uh, this has been with us, uh, for long years. Uh, it's extremely effective. The challenge of improper authentication authorization ha is pretty, uh, hard to solve as the permissions.

As I mentioned earlier, the permissions grow organically over time, and we really have to keep a track of about what the, uh, roles, uh, we build for cloud or for our applications or for our service accounts do. So the cloud service providers, uh, have, uh, some tools for that, which is the Google Cloud policy analyzer. Uh, AWS has the IM Access analyzer, and Azure has, uh, discover in insights for the roles.

So these tools will help us actually figure out what scopes, uh, the roles we build have, uh, and are they, do they exceed, uh, those, those scopes. Uh, one other approach is to generate a customer role while the application is within development in your development environments. And then based on the actual calls, the application or a user makes to, uh, the cloud environment replicate a production, uh, grade role based on only the necessary calls.

So, uh, the next on the, uh, cloud native, uh, application security is the deployment pipeline. So quite often, yeah, this is, uh, this falls under, uh, this falls in between the couch, uh, cushions, uh, basically as a kind of, uh, yeah, we just do it. Uh, but, uh, this is and very crucial system.

And, uh, yeah, things like the pipeline does not, not apply proper integrity checks, uh, uh, or containing inherent and security flows that might be used as malicious access. We had attacks, uh, but, uh, similar. We had attacks in the past, for instance, SolarWind.

Uh, we had attacks, uh, on the code of development system, basically, uh, the NotPetya, uh, if somebody's familiar with the Maersk, uh, one of the biggest, uh, uh, uh, ransomware attacks in the world on Maersk, that one was actually caused by a, uh, pipe CICD pipeline being not secure, and actors being able to inject, uh, malicious code into an application, which then got distributed all over the world. So, uh, for the purposes of this challenge, we're looking at insufficient authentication on the pipelines use of untrusted images and overly permissive, uh, access to, uh, registry artifacts. So these are kind of the things you can look at from a operational perspective and signal, uh, if, uh, something, if one of these, uh, things, if one of these systems gets, uh, exploited.

So securing up the cloudnative applications just during the development process is not an enough. We need to really do it, uh, for the whole life cycle. So really starting from the planning phase all the way to running and operating the application.

So, uh, to one, one way to address, uh, this, uh, problem is to have proper base image control. So we can, we really need to know what are we running it, where are we running it? And by having a base image that we trust and we're able to able to verify, we'll help immensely, uh, with our security posture, and, uh, gives you the ability to actually, uh, on the operational side view and detect if something weird is happening, uh, uh, within your cloud native application.

So, uh, in this case, we're looking at, uh, base image repository with, uh, the list, uh, privilege access, uh, to cover our previous point of, uh, authentication authorization. And, uh, we can sign those containers. We can have templates for pipelines.

Uh, we can use applications like, uh, open the open policy agents to ensure that only proper images can be deployed to our clusters and to our Kubernetes nodes. So, uh, all of the major cloud service providers give us the ability to, uh, store our artifacts within their clouds with additional protections. So we can really have a, uh, end-to-end pipeline where the application is being built, uh, and stored within the cloud service provider without actually having to leave the, uh, cloud environments, uh, which adds that layer of certainty that the code has not been adversely affected, uh, by, uh, malicious actors.

So for Google Cloud, we have the Google Container Registry for, uh, uh, uh, Azure. We have the Azure Container Registry, and for, uh, AWS we have the, uh, elastic Container Registry. So, off to the next one.

Um, yeah, over permissive or insecure network policies. Uh, so because the of the way the networking is done is the cloud, we quite often overlook what the, uh, what our systems can access, and how farther access actually reaches. So single, uh, cloud native app.

Since cloud native applications are built on loosely coupled microservices and often rely on, uh, different cloud services, it is, uh, critical to apply a strict network policy and go as far as, uh, our, uh, a bit overused term of zero trust. Uh, but we need to establish effectively a white list of where our services should be going, uh, towards, and what they should be actually accessing. Because if we try to apply a blacklist, uh, it's next to impossible, because the possibilities for the attacker to reach out to go somewhere to change their access points are effectively infinite.

But if we limit our applications to, okay, we know these two services communicate with these other two services, then we can really create a, uh, network based on, uh, a source that we know that we understand. And for operational purposes, it makes it also much simpler to detect deviations, because we will be able to immediately see, this is not what we expect from our service. This is not what the service is supposed to be doing.

So based on just creating simply a white list of expected and known actions of a service, we can create a very simple detection list in our, uh, in our seams, in our source. So for this case, things like over permissive pot to put, to put to communication internal microservices exposed to public internet, we still quite often see, uh, Kubernetes nodes directly exposing, uh, their services to the internet. So, uh, network traffic to unknown or measure malicious domains not blocked, that's also quite often seen.

So, uh, for as a way to address this challenge, uh, I'd like to also try simply egress control. So how much, uh, how much do you need actually full and open egress for your applications? In most, uh, cases, the services we built and provide, uh, rely on the fact that a user will reach out to us.

So, uh, just by having a situation where the application does not need to reach out to the internet in the first place, we already built a really, uh, good wall around our services, where the cost of attack for the attacker is extremely high. And by disabling their ability to reach out to the internet randomly, uh, we make it extremely difficult for them to be able to perform, uh, malicious actions. And the detect the detection becomes also much, much easier, because we immediately know that if it's not on a white list that we have prepared, we have built, it is a signal something is going wrong.

So, uh, to make it a bit easier for, uh, services, uh, for, for services like Kubernetes, for orchestrators like Kubernetes to be to able to control the traffic, there's two projects. Uh, one is called Calico, the other is cilium. They enable us, uh, they enable us to build lists and ac uh, network control access lists to properly control the egress of our applications.

Um, the next one, uh, for today is, uh, inadequate compute resources, resource quota limits. So this is a relatively, um, new challenge, especially because it is specifically related to the cloud. On-prem, we, uh, had a similar challenge.

This was usually caused, uh, related more to a simply a denial of service where, uh, if we overspend the resources that we had, our services simply stop working. The issue here with cloud native applications is that if we enable scaling, uh, very few cloud providers, uh, give us the ability to really fine tune the scaling. So we might be able to say, this service, uh, has, can go only up to X amount of nodes, basically, but that additionally with the cost of not being actually able to access the service, if a denial of service attack like this happens, we still have to pay for the additional capacity.

So this is a very different angle. Some attacker stay where this is, this might result in a purely, uh, attack, designed to not only exhaust your compute capacity in the cloud, but also, uh, affect your, uh, bottom line. So malicious actors can attempt, uh, to exploit computer resource resources for a un unintended use.

Applying proper computer resources quota is the first line of defense against such attacks. So we have attacks that, uh, will spin up, uh, containers that will mine cryptocurrencies for attackers, because, uh, there are no resource, uh, binds on the container themselves. We have, uh, over permissive, uh, request quotas for APIs, which means, uh, attackers can, uh, simply make our services scale because we do not control how much they can request.

Or even with serverless functions, especially going one layer up. Uh, yeah, those are, uh, also, uh, quite commonly not set up properly, where the attackers can really go, uh, full, uh, ham and really, uh, expand our finance, uh, expand our monetary resources by, uh, by requesting so much that our application cannot no longer keep up, and we spend more than we actually earn because the application doesn't really do much because of the attack. Um, yeah, a relatively simple solution for it.

Uh, so we do have to make sure we have these limits, uh, basically requests and quotas. Uh, and, uh, in the context of Kubernetes, uh, these three are present there. So a limit, base limit, uh, uh, sets a limit for a specific container.

Basically, uh, the minimum and maximum requests, uh, are on a, uh, per container basis. So how much they can actually, the container can request in one go and quotas can be set up for the whole, uh, namespace within a Kubernetes. Uh, so, uh, you can say, this team has this much CPU, this much ram, and that's basically what all they have to do.

Uh, so, but it's not only that in the cloud, there's multiple layers for our limits. So we have our load balances, we have our web application firewalls, we have our ingress controllers, our service meshes, so, uh, they can, uh, scale as well, and they, we can use them to control, uh, the amount of resources consumed by application. So, uh, what's important, it is to set up, uh, alert thresholds for excessive outer scaling.

So if we have a very large, very, uh, very large spike, uh, that's a very, uh, good indicator that something, uh, really weird is coming based on the indicators that we set up. And it's of great value for security operations team, uh, to actually view that information and be able to integrate it with, uh, with the sim and correlated with other events that might be happening. So you can immediately get a good overview of, uh, yeah, there might be something wrong.

Is it sufficient? Does it cross our threshold where we need to start a, uh, for instance, a security incident? The next one on the list and, uh, uh, the final one for today in general is ineffective logging and monitoring.

So, uh, this challenge came up because, uh, of the way we have created our cloud environments, and we've been adding layers on top of layers, on top of layers of software on hardware. And at a certain point, we lost, uh, insight in the into actually what's happening, uh, if, uh, anyone has ever, uh, had to configure SELinux or App Armor. Uh, I've been there.

So this is, we lost that capability once we moved to the, uh, hypervisors. And once we start running visual machines at scale and even went further, uh, going through, uh, even went further as far as the, uh, as far as the layer, as far as the security layers are concerned. So, uh, for this attackers, uh, can rely on the lack of monitoring and timely response to achieve their goals without being detected.

This is quite often, uh, the case nowadays where, where the attackers roam around for a month or two before we're even able to, uh, see what is happening. So this becomes extremely critical in cloud native applications, uh, which on compo, because they're composed of, uh, different layers and dozens of microservices. So there is a lot of noise, and it's, uh, sometimes very hard to get the information that we need, uh, reliably.

So, uh, for the purposes of, uh, this problem, uh, we'll be looking at addressing, uh, lack of container, our host process monitoring, uh, LA lack of network communications monitoring between the microservices, because in the current, uh, uh, day and age, uh, yeah, we, we've lost the ability which we had, uh, in the past to be able to tap directly into hardware and actually duplicate the traffic. Uh, now it's much, much harder to do. And then lack of monitoring of the, uh, orchestration, uh, configuration because, uh, once we push, uh, configuration to all clusters, it can spread to, uh, all the other nodes.

And we might be, we might not be able to see, uh, if a malicious node is attached or if a malicious body is attached to our clusters. So a way of addressing, uh, this problem is the concept of observability. So, uh, the goal of this is, uh, to provide us with the insights into what's actually happening with, uh, within our systems, uh, what's happening within our clusters.

So the, uh, the concept of observability is relatively new as far as cloud native applications are concerned. Uh, one underlying technology that's, uh, very important that has been essential in giving us back the insights and visibilities, EVPF. Uh, this technology allows us to hook directly into the, uh, kernel of the operating of the host system and actually monitor all the containers that are running.

It doesn't matter if it's one container, if it's a thousand containers, as long as you look at the underlying host, we have all the visibility that you need so you can see exactly which system calls I made. And if you can see the system calls, you know exactly what's happening on the network level, because that has to go through the kernel to request resources. You'll know which, uh, executables are actually running, uh, are files being opened, our files being read, our files, our files being written effectively.

Anything and everything that happens on a, on a machine or within a container can be monitored. And, uh, you can generate a lot of data, and that gives you the ability to actually figure out what do you, uh, want to see. Because if you can establish, again, a white list of what a container is doing, so you know, you're running an application, you're running a binary that does free system calls, which are very specific to the application and nothing else.

You can also use this technology and the tools I'll mention in a second to only allow those free system calls. So anything else that the attacker does will be outright block, or you can at least monitor it if you, you cannot block. So, uh, uh, tools like, uh, Falco from Cystic, uh, Tetra, uh, from ISO valent threat mapper, and, uh, Pixy give us this ability to actually view, uh, what the systems, uh, and, and the orchestrators and what our cloud native, uh, applications are doing.

Uh, the other bonus is, uh, your, uh, platform operations or site reliability engineers or DevOps teams, they might actually be already using some of those tools, and they might actually have this data for you. So you might not even have to explicitly deploy it, but you might just need to go to them and just ask them, okay, send us those logs, or enable this type of log, or enable that type of log. So this is also a very useful operational tools for just simply running services and having very good insights in what's happening within our, uh, within our a applications.

So a very good chance that some of your teams might actually be already doing this. And with that, uh, I'd like to, uh, leave us basically with the, uh, with this quote. So we cannot solve our problems with the same thinking we used when we created them.

So I think this sends a, uh, really good, uh, message where, where we, the problems we encounter right now are not extremely novel, uh, but because of the way we try to approach them, uh, we're kind of running them into constantly and again and again, again. So, uh, the problem of, uh, authentication authorization, we had it, uh, 10, 20, 30 years ago as well. Uh, so this is nothing that we're still kind of struggling to, to figure it out, or the, uh, problem with visibility to within our, uh, systems.

It's a similar story where we can come up with, uh, addressing the problem now rather than trying to, uh, figure out, okay, how might it actually look in five or 10 years? Maybe there are ways of, uh, giving us the ability to, uh, not necessarily get rid of the problem or, uh, but lower the risk sufficiently where we feel comfortable and we have sufficient, uh, capabilities to address it. Uh, and with that, I'd like to thank you very much, uh, for attending, and hope to chat to you online.

Thank you.