Live Hacking! Practical security examples for AWS EKS/Fargate using Falco

I don't know, I was working before. Actually, hang on. I think it may be going now.

I see it. The progress bar. Yeah, I see light.

OK, I think we are ready to go. Kris, it's all yours. Ok, great.

Um, hi, everyone, and welcome back. Cool. So it looks like we're going live now.

Thanks for everyone. Oh, cool. I can see my screen now.

Thanks for joining everyone and for hanging out and being patient with us. I'm really excited about doing the live demo for everyone. And I guess because it's live we're getting some exciting challenges.

But this did give me an opportunity to to ask folks a few questions, both on Twitter and live here in the chat in 6connex to see if folks are interested in learning about anything in particular. So it looks like we have Chris Izola here said, "Hi, Kris. If you'd like to do a very quick overview of Falco, if that wasn't in your plans".

That is definitely in the agenda for today. And I will give a couple of high level overviews of Falco and a couple of different relevant environments. And if you have questions, Chris, or anyone, you can ask him to chat.

And I want this to be as is relaxing and fun and is engaging and entertaining as possible. So the question here from Urkesh, "why full screen is not working". I am not sure.

But we can do a quick test to make sure that my screen share is working. So we should be able to see each top on my computer here and you should be able to see that we are here and the Falco EKS directory on my Unix computer. And I'm going to actually just zoom in just a little bit, give folks a little bit easier view of the screen here.

Might be a little bit easier on the eyes. Cool. So real quick and then I'll get into the presentation.

That looks like we have another question from Walid. Sure. "What advice to new corner to the container security field.

" Container security field. I think if I was to start researching container security today, I don't know if you saw Liz's presentation. She does a really good overview of container security in general.

She is a coauthor of one of the container security books. So that's a good place to just get a breadth first approach to security. My personal approach to learning anything, whether it's security or a new school skill or anything, is I'm very much hands-on.

So I would say try to break them, run a container, get something running in an environment somewhere, and then try to hack it and then research different exploits, different ways people might try to take take advantage of it or expose some sort of unintended behavior and and start thinking like the the thing that you are trying to protect yourself against. Some people call those hackers. They could just be people trying to do coin mining.

But yeah, I think hands-on approach has always been my preferred approach. And then, yeah, it looks like we have folks helping each other out with full screen. So let's get started and we're going to talk a little bit about container security and Amazon.

And just for today, I brought my AWS cup of coffee or what's left of it because I've been pounding coffee all morning. So anyway, we're going to talk a lot about containers. We're going to talk about Kubernetes.

That is we're going to talk about EKS, ECS. And we're going to look at Falco running in all of these environments and kind of talk about the nuances between them. And I'll be able to share a little bit about my history and my involvement with a lot of this.

It's it's kind of surreal to be coming in this late in my career and still talking about one of the first things I worked on when I started working in Cloud Native. But we'll get in more in a moment, so let's get started. So I like to do all of my presentations on the command line, so I have a repository here that's got all of my slides and the slides are is low quality is possible.

There's probably four or five kilobytes of information here. And that's because I don't want to distract from what we're actually doing, which is hopefully all I'll be showing you concretely ways you can start to use and understand these different systems for your own use case. And we're going to start off with a little history lesson.

So we're going to open up our slides. So here is our very high quality slide here. So the name of this talk is called Practical Security.

Falco with EKS Fargate and a little bit about me. I was a coauthor. It a really great guy named Justin Garrison of a book called Cloud Native Infrastructure.

Over the past, I think it's been five years now, five years now. I've been a maintainer, contributor, a collaborator, whatever you want to call me, of a ton of different open source projects, including Falco, including Kubernetes, including Linux. And the one thing that has sort of stayed with me throughout all of this is I envision myself as a hacker.

And yesterday I use the word hack and I think a lot of people hear the word hack and they think expose or exploit. And that's not what I'm trying to say. I am trying to say that I will continually poke at something to try to figure it out and to, you know, see what it does, observe it and try to do things in different ways that folks might not have done before.

And usually while doing that, I learn a lot about the tools and I learn a lot about security by just exploring and experimenting with things. So that's that's really what I mean when I say hackers. Imagine me hacking away at a tree, trying to chop it down or something.

OK, so let's start off by defining the problems. What problem are we trying to solve and what are we going to be learning about today? So there is this question that I hear all the time.

How do we secure Kurbernetes on Fargate or even we could even simplify that a little bit more and just say, how do we secure Kubernetes? And that is, honestly, that's the million dollar question, right? That is a very easy question to ask in an extremely difficult question to answer.

So what I'm going to do is I'm going to talk about one of what I consider the three approaches to a holistic security solution, which is intrusion detection or surveillance or just security detection in general. Anomaly detection is another way of saying it. And we're going to look at an open source tool called Falco and how that can give you some tools and controls for gaining visibility into your system and how we can use these existing systems like Fargate to help secure our containerized workloads.

And and then we're actually going to do it. And then we're going to hack it and then we're gonna see to see what Falco has to say. And we have a real target cluster with EKS up and running.

And we can we can go explore with that as well here at the end. So. Let's get started with a little history lesson.

So let's talk about Kubernetes on AWS. I wish we were in a real auditorium right now. I can see here it says current participants is forty five.

So hi, forty five people out there. Whoever you are, I'm glad you're here. And I wish I could just ask show of hands who here is is running Kubernetes on AWS.

And that would give me a little bit of an idea of who we're talking to here. But I'm going to assume that you're you're either currently running Kubernetes on AWS, you're interested in it or you're just here to learn and you feel like there's going to be some interesting takeaways from this, regardless of of your current implementation, in which case all of this is going to be relevant. And thank you for raising your hand while.

I appreciate it. So once upon a time, there is a shell script at the company's GitHub repository called Kube-op, and that script later turned into a full featured go program called Kops, which was one of the first Kubernetes subprojects that I got involved in. Later, another project called Kubeadm came out, which solved a similar problem to Kops but was a little bit different.

Kops was more of a holistic solution and went as far as creating and managing infrastructure, whereas Kubeadm drew the line at Kubernetes and basically said we're only going to be responsible after an operating system is working somewhere, whereas cops said we will be responsible for creating this operating system in Galion virtual machine from thin air and getting your Kubernetes set up that way. So naturally, Kops was probably the the good first step because it was it was more inclusive. It did more.

But if you if you look at a scope of responsibilities, Kubeadm had been adhered to the do one thing and do it well, which was to set up Kubeadm has come a very long way. And it's my preferred way of sending up Kubernetes is to this day. I created a project called Kubicorn that was the first implementation of using Kubeadm had been while also doing what Kops did, which was to create the underlying infrastructure.

And this all happened over the past five years. And somewhere along the way, Amazon came out and said, hey, we're going to offer a full fledged quality service that we're going to call EKS, Elastic Kubernetes Services. And by this point, there's probably a dozen other Kubernetes installers out there running on us that Amazon had the luxury of investigating before they put together case and they put some engineering resources on it.

And now you can go to the the Amazon console and you see something like this, which is very public over here on the left, and you're actually able to click on clusters and poof, there you go. You have to create a new Kubernetes cluster and you can even select which version of Kubernetes he would like to run, which if you were using Kubernetes in 2017, this was a twinkle in the eye of all of us. And I think a lot of us would not have had jobs if this UI would have existed.

So anyway, let's talk about some of the things we learned while we were setting up Kubernetes in Senegal, because in the early days, the first one was we always ended up with a concept of auto scale groups. Right. We had a group of servers and that group of servers would often and often cases would need to change.

Furthermore, we knew that we needed to run. Container's on two instances and there's a lot of nuance that can go with that, which is the isolation between the container and the host. What role does Kubernetes play in the how do we ensure that are continuously going to be running happy?

And then from a security perspective, how do we make sure that these containers are are isolated and that we're not going to introduce a new sort of container escape concern or security concern while we're setting this up and then like unleashed Pandora's box of how do we manage certain information? How does the network look like? What holes are we poking in the firewall?

How do we structure the security group between the different ASGs? We open up everything. Do we only open up what we need?

And then we started to discover all of these Kubernetes components have these network connections that nobody really knew about. So there was a lot of complexity here. And we started to kind of put together the list of what does it actually look like to get Kubernetes running on Amazon.

OK, so let's fast forward and let's look at some of the takeaways here. So the implementation ultimately turned into two well-known concepts which we called instance groups with in the case of Kops or we called server pools in the case of Kub icorn. And these were groups of machines, just like an on a scale group that you see in eight of us to this day.

This is relevant because we were mutating infrastructure. We would have won on a scale group or two out of scale groups or different auto scale groups and different availability zones for those folks that like to run a highly available group. That is.

And we noticed that we would have to change the amount of infrastructure based on our use cases. Now, this is starting to allude to what I consider like the final problem that we needed to solve. And Kubernetes, which was how do we only have infrastructure for what we need?

It's really easy to go to Amazon and say, I need one hundred servers. It's really hard to go to Amazon and say, I don't know how many servers I need, but I would like for you to figure it out based on what I'm actually using. And that way I'm only getting charged for what I'm actually using.

Now, that is the million dollar question. And we started to look at ways of changing these sets of servers based on arbitrary metrics. And I'm sure you can imagine, you know, load CPU memory usage.

There's a lot of different things that we could we could try to adjust these infrastructure components with. But ultimately, there was a cleaner way of doing this. And this is where we get into the whole serverless discussion that we see today.

So kind of in summary, we had groups and we knew we needed to change the amount of servers in the groups, but that wasn't always an easy task. So let's look at other problems we encountered while we were setting up Kubernetes in each of us number one nodes, our servers, they have static dependencies, just like any server that you manage. Kubernetes is effectively a software package that needs to be upgraded.

If you haven't gone through and tried to upgrade Kubernetes some version into version and plus one in your career, I strongly suggest you spend the rest of the day today trying to do that. While keeping your Kubernetes cluster online is a fantastic exercise. You learn a ton of you learn a lot about yourself.

You'll learn a lot about that is you learn a lot about Linux. It's a good exercise to do, but it's not easy to do is the point. And you also have dependencies, right?

Like the Kubelik, the fundamental work horse behind Kubernetes runs as a System D unit, which means now we're we're dealing with Linux concerns. And if you look at CNI, those are going to have different dependencies as well. A lot of CNI implementations used IP tables as the underlying mechanism in which we do the routing and the network configuration.

Some of the newer ones, like Psyllium use EPF, which is what Falco uses, which we're going to talk about a little bit. But ultimately, this is there's a lot that you have to keep in mind here. Ok, so now let's talk about Kubernetes scheduling.

So this is this decision of how do we actually create an application running in this environment? So in traditional AWS, you would have your Kubernetes cluster up and running and it would have some amount of nodes that really wouldn't change. And if it did change, it wasn't very often and you would now have to schedule a workload.

So let's say we're scheduling a simple Web service and this there's a dashboard, a UI you can go and access and you can have multiple copies of this pod running on a single node. Ultimately, Kubernetes is the thing that goes and schedules these pods for you, right? It makes the decisions and it's the implementation of the Kubernetes schedule that's going to decide where and how these containers run.

So we're starting to look at exploring a world in which we can create infrastructure as needed directly. And that's exactly what you see with Fargate. And it has if you schedule a pod with Fargate, it's going to then create the underlying infrastructure for you as needed.

And if you kill the pod at the container level, that infrastructure is destroyed. And there's so many beautiful guarantees about that in your only paying for it for as long as it exists. And we'll look at that when I do the live demo in a moment.

But we have that now because Fargate solved the problem of first running containers and then we were able to translate that to what Kubernetes expects and actually use Stargate in the same way that we would traditionally use the Kublit to schedule that using the container runtime like Docker. Again, I had mentioned there was this two way communication. So if you've ever exacting to a Kubernetes container or if you've ever looked at the logs before, we're going to do both of these in a moment.

The Kubernetes control plane has to know about the nodes and the nodes, have to know about the control plane. In other words, how do we go out there and how do we access them? And that communication needs to happen in Amazon, which here we are again, talking about security groups and how we're going to expose these two different sets of nodes to each other.

So already you're starting to see the complexity and how this is all set up. The good news is that ECS takes away a lot of this for you and sets it up in a prescribed way that they suggest you should be running all of this. Ok, so let's explore ECS and Fargate.

Now keep in mind ECS is similar but different to EKS. And in Fargate now is mixed in with all of these. So I want to spend it just a few moments talking about the differences and make sure that we have a really crisp understanding of what these services are, what their responsibilities are and how they all fit together.

So that would we try to do some malicious things here in a moment? We actually understand what we're doing and why we're doing it. So ECS, Elastic Container Services, is a way in which you can run a single container or set of container images.

In fact, we pull it up here in the dashboard. You can see I have my nova-hacks cluster running an ECS and you can see here I have to Falco tasks and running. I have Falco trace as safe and vulnerable server and I can come in here and do tasks and you can see these things are running.

And all this is, is this is just a container image that I built and pushed to my container registry that's now running. So I look at ECS like I look at a Kubernetes pod, right. Like where the service is.

The deployment in the task is the pod. And you can talk about how many pods you want in your deployment or how many tasks you want in your service. And it just gives you that same abstraction.

But ultimately, ECS is a way for you to run a container image. And that's exactly what the Kublet does, right? If if you look at my coauthor, Justin Garrison, he was able to rewrite the Kublet implementation and bash using Docker run because it just does a doctor run for your container is behind the scenes.

And and that's effectively what ECS is doing now. Granted, it's much more complex than that and it takes security concerns into consideration as it does it. But ultimately, that's what you can think of it is doing, is it's just giving you an API and interface in which you're going to run a single container.

So far, gate is the mechanism that does the scheduling. Fargate is a task execution system. It's it's the piece of software that you can give it a set of things to do.

And I look at it like Jenkins' like it goes in and kind of just does stuff for you in a in a clean way and it gives you a set of controls in which you can control things. And the first implementation of Fargate was to schedule containers, and that's exactly what the Kublet and Kubernetes does. So now that we have this underlying component of ECS and Fargate able to schedule containers, we can then wire that up to the Kubelet and we can actually run a Native Kubernetes installation.

And instead of creating a container running on one of these nodes that we created up with tools like Kops in Kubeadm many Kubicorn, we're actually going to create a Fargate instance as needed. But we don't have to care about how that works. We can just schedule things like we would traditionally in Kubernetes which is pretty cool.

So in summary, everybody just wanted the same thing. We just set it in the. Ways, in other words, everybody wanted a way to define a set of containers and wanted a way to represent a single container in Kubernetes as we call that deployment in pods, and we call that services and tasks.

Now, it's just a matter of translating those two things to speak the same language and now they're compatible. So that was really the engineering exercise and the beauty of setting this up in the way that they that they did. Ok, so now let's talk about it, yes, which is different than you see ECS focuses on the container, EKS is a broader Kubernetes installation.

So let's look at some major differences between EKS and what I what I think of when I think of Kubernetes on Amazon. This is going to be like a Kops installation. Or if you remember, when we were at Apptio, we did the heftiest QuickStart or one of my cubicle installations or even like a home grown Kubernetes install on us.

There's some major differences here. The first one is if you if you look at how EKS and Fargate schedule pods, they're effectively two instances on the back end. But you don't really have visibility into them, which is both a good thing.

And for some network administrators like myself can be a bit scary because any time there's there's something that's happening that I don't have access to and something goes wrong, I didn't have to trust that the things that I don't have access to are working as expected. And this is my own folly. But somebody who's worked at a major cloud provider in their career knows that it's not always the case.

And so. So, yeah, so we don't have visibility, which is both a good thing and a bad thing for users, depending on the type of user you are. And I would say the majority of people out there are not like me.

The next one is we have private subnets. So if you if you actually go and you look at how to set up a subnet in eight of us with networking, it's it's crazy. And network isolation is one of the best ways to secure a system.

I hate to say it, but if I want to truly lock down a system, I start with the network. It is the first place I go to and I should shut it down. I shut down the network.

And if you can't access the network, then right away you're just making everything else is substantially, substantially harder for an attacker to to do anything malicious in the private subnet in Toovey or VPC basically says that you have to be an instance that can within that subnet in order to reach out to each other and you have no access to the subnet outside of it. So one of the first features I coded into Kubernetes Kops was private topology, which the only way to access these Kubernetes nodes was through like a Bastion server, because there are these nodes had a resolvable public IP address. They basically were an isolated set of nodes running in an imaginary data center.

And easy to and that's what we see in EKS and Fargate today, which is very, very powerful, this one. Exactly. And this is really cool.

One of the biggest things about ECS is you can't exactly. To the container. I let me rephrase that.

It's hard to exact into a container and it's not what you think it is. So if you come here, if you notice when I set up my demo for ECS, I have an SSH container. And if you think about it, that's actually kind of weird.

Why would I SSH into a container. Right? Usually your SSH agent into a server and that's because you don't really have this fundamental access that you would have otherwise in in a natural CONTAINERIZE environment like like you would see with Kubernetes.

However, in this case, it's quite easy to use Kubernetes negative that and you can get into one of these pods naturally. And we're going to do that here in a moment. And then, of course, we have native access to trace.

If you if you come in and you watch any of my presentations on ECS and how we secured it, one of the problems we had with Falco's was getting a driver working. How do we go and how do we install some layer of software that's going to allow us to start putting together the system called meta information about all of these the system calls that are happening. And in order to do that, we needed trace.

And that's what we're seeing here. And I can show this in a moment if folks want to see it, whereas we didn't have to go and turn on key trace for the implementation I'm about to show you. We had native access to P-trace as it was.

So you can already see that the the container runtime is a little bit more open with EKS and it is with the ECS, so you're trading off some security there. Ok, so after that we can talk about scuttling a pod in communities. So traditionally when you schedule a pot, like I said, the Kubelet says, you know, the Kublet is given a message from the API server that says you are responsible for executing this pod.

And the Kubelet does that. It starts a pot and that usually looks very similar to a docker run like execution. Now we have an API call that goes to the Fargate workhorse and Fargate will go and actually schedule a forget container and then that will respond back to the rest of the Kubernetes system as if it's just a normal pod like we would have running on one of these hosts traditionally.

So we're starting to see that serverless is really a way of synthesizing what the Kubelet expects a regular server to do, and we can offload that. A service that can change from underneath of us and we're kind of protected from that would be a good way of saying, OK, so let's talk about Falco, the project I work on. So that's Kubernetes in a nutshell.

Now that you understand all that we talk about how Falco works on all of that. So what Falco does is it accesses system calls. So any time you you execute a command on a system, whether you're checking your Twitter page or you're running Kubernetes at ease or you're breaking into a bank, you're more than likely going to be using system calls.

And that's how your application is interfacing with the underlying kernel. So what Falco does is it reads the system, calls in the information that you're passing to the system, calls and it rebuilds the state of your system at runtime. And then we just we see if anything unexpected is happening.

Right. Like now, we have the stream of data. And if we notice something anomalous or something unusual, we can trigger an alert and say that this this is not what we would expect to be happening in this environment.

And because we're doing data that the system call there, we're taking away all of these attacks vectors where you're you're actually spoofing or tricking the system. And I think it's doing something safe when really you're doing something malicious. The most simple example of this is replacing a binary, gaining access to a system, taking it less and then replacing it with less.

That also sends all of your secret information out to a third party server somewhere. So any time somebody types, it looks like they're just listing files, but really they're doing something that they shouldn't be doing. There is a lifetime of attack vectors like this that we see out there.

And Falco, because it works at such a low level, we'll be able to detect if one of these system calls is being used in a malicious way, regardless of how it's being used. And then we are alert. We have an entire learning mechanism that can be calling a web hook grapeseed.

It can be going to cloud watch native us, which we're going to see at the demo in a moment. And then ultimately all of that comes through a driver. So a driver is the thing that in a secure way consumes the system called information traditionally.

And Falco, we had used a kernel module that wasn't always a viable option. For instance, would they do not allow you to load a kernel module for good reason? And then then we developed EBPF Pro that gave the same information, but using a different way of consuming the system calls.

And now for ECS Fargate, we use the P trace implementation that we call P dig. And P dig uses P trace to parse the process, start gathering system, call information about the process as it is executing it, and then reporting that back to the Falco system in the same way the other two drivers report back as well. Question here, why does Falco support audit logs from ECS Cluster?

Yes, we can consume audit logs and we can assert rules against them. It will require some configuration on your end to get all of that set up. So good question.

And if you want to find out more, just just messaged me afterwards and I'm happy to put you in the right direction. org, I wrote this blog called Choosing the Falco Driver, and this talks about the nuances of the kernel module, how to compile it. It goes into the pros and the cons of using a kernel module, compares that to the probe and of course, compares that to the P dig implementation.

And you can see here, it'll even show you, you know, here we suggest you use ECS Fargate and EKS Fargate. We suggest you P dig. But, you know, if you want to use negative or not negative, native Kubernetes we can we can get away with using the kernel module.

So this will help you kind of make sense of the driver and the different ways you can get Falco setups. org, and you'll see that blog there. OK, so let's summarize this and then we're going to get into the the the demo here in the last 10 minutes.

OK, so ultimately we need to run a container on a Linux house that is terrifying. Any time you run anything on a Linux host, it's terrifying, but it's not impossible. It just comes with a lot of concerns.

So that's the problem we're solving. And we're doing it using these tools to make it easier for us as as administrators and as operators. So let's talk about Kubernetes.

We can build our own Kubernetes and that comes with a set of drawbacks, but it also gives you some control and some visibility you wouldn't have otherwise. Or we can use EKS and Fargate and we get this really, really nice interface that the the folks at AWS have put together for us and prescribed for us. And we're only paying for what what we use as we need it.

As these these containers are, the infrastructure for them is created as needed. The three pillars of security in this type of system, isolation, prevention and detection, isolation. And this is going to be isolating the container workload from the rest of the system, things like namespace sharing both in communities and at the Linux level, as well as network isolation.

Remember, we talked about private subadults earlier and how on that we're able to control what can and can't access these different pods as they're running in excess. And that's funny or ironic, I would say, because ultimately the goal is to connect applications. We want things to talk to each other and we're making it hard for that.

So that's a big concern for us. Prevention. So this is a set of locks or controls preventing known attack vectors, everything from our back to SU Linux and to Seccom to Policy Enforcement, anything that you as a user would do to prevent something from happening.

This is going to fall under prevention and both of these are necessary for a holistic security approach. So make sure you're checking the box for isolation, make sure you're checking the box for prevention. And then that goes into the third pillar, which we call detection, which is where Falco comes into the picture here.

So anomaly detection or surveillance? Remember, we talked about how Falco uses system calls to detect if something unexpected happens. So we're going to be looking at that concretely during the demo here in a in a moment.

And we're going to be showing it both on bare metal. So we're going to kind of start in its most raw form and then we're going to, like, slowly build up in the different systems and all the way up to EKS with Fargate and show if Falco can do. And we'll kind of repeat the same set of exercises so you can really see what it's doing.

And we're going to talk about how it's doing, what it's doing behind the scenes. OK, so yeah. So other than that, we have we have the demo.

We're going to start off in bare metal, which is my bi Linux computer that is right here in front of me. And then we're going to we're going to move up to the cloud and we're going to we're going to show a lot of this and then we're gonna go to ECS and I'm going to show you a single use case of deploying a container and what that looks like directly. And then we'll glue everything together with, like the final the final piece, which is looking at how EKS can use node groups and Fargate profiles to put all of this together with Falco and create a holistic solution that not only is secure, but you're paying for it as you need it, which is really rad.

And like I said at the beginning of all of this, like if I put this five years ago, I would not have a job right now. So it's really cool to see how all of this comes together. So, yeah, we're going to show Falco and Kubernetes using X and we can talk about the different driver implementations to.

Right. Like, do we want to use the kernel module or do we want to use pedic? What does all that look like?

So we're going to have a lot of fun there as well. OK, so that is the history lesson in kind of the lecture session, you know, component of our of our discussion. And now let's get into the demo, which everybody is here for.

So I do have chat pulled up so folks have questions as I'm doing this, please, please ask, I am here, I can read it and I am happy to to answer any questions that folks may have. So earlier in the chat, somebody asked about Falco. Let's talk about Falco.

Let's look at it. So what we do is here on my computer, I have this developed directory, which is where I do all my development work, and here I have Falco. com.

Falco security Falco. So that's going to look like this. 26, which I just released recently.

So that's what we're going to be building right now. So we're going to build directly and I think this is all written in C++. I'm going to compile Falco and compile the kernel module and we're going to run it on my local system and see what this thing can do.

So the first thing we want to do is we can do make Falco and I have 16 cores, so let's let's use them. So we're going to compile Falco, which a lot of this is already done. So I'm kind of giving you like the cooking channel like and then we put it in the oven and like, it's poof, it's already done.

But you can see we're already at 94 percent, so we're going pretty fast. It doesn't always compile this fast is what I'm alluding to. Great.

26, which is the most recent version of Falco, and that that compiles. So now we can build the driver. And the driver is just going to be the the kernel module so we can we can list modules in my Linux system by doing this mod and we can grep for telco.

And you can see that we have the Falco module loaded and we can remove that. ko. Run that as root.

And now we have a driver, and so if we list slash dev on my system and we grep for Falco and it's getting warm in here, so one second and if we grep for four, Falco. You can see I have 16 cores on my system and and we've created a Linux device for each one of these. So this is a device driver, right?

Making sense. And each of these is it's representing one of the CPUs on my system. And we're gaining information about what's happening at the kernel level via each one of these files.

In fact, if I remove this driver. Falco list, again, I don't have any device files there anymore, so let's insert the driver again. And here you can see we now have all of our Falco device drivers there.

So that's how we're gaining information about the kernel. So we can execute Falco by basically doing its userspace/falco/falco. This is the binary to just compile and you can see here we want to run it as root and Falco's is running.

It's detecting every system call in my system right now. And if I do something malicious, Falco will alert us about it. And that's all governed by a set of rules that you can configure.

And there's a home chart for all of this that makes it really easy to install this in Kubernetes. And this is the fundamental technology and how how it works at a low level. So you can see here I ran a control C and I sent a SIGINT and it exited.

And you can see here that Falco cleanly exited and it's no longer running on my system. So let's talk about what Falco looks for by default. So to do that, what we're going to do is we're going to run our recently compiled Falco.

And it looks like they're going to let me go a little bit longer and my demo, which is exciting. So I'll keep going until we're done here. So we're going to run Falco and I'm going to open up a new terminal.

OK, so let's full screen this. And you can see here in the back, we have Falco running behind this, and what I'm going to do now is let's let's actually make this a little bit easier on the eyes. There we go.

And let's turn up our scaling. There we go. So now I am going to cat/etc/shadow and I'm just going to put that into such slash nothing.

And I'm going to run that route, and if we go back to Falco here, you can see we've got this lovely alert, warning sensitive file open for reading by non trusted program. And then you see we have these key value pairs. Users equal to root program is equal to cat, a cat lets you shadow.

And we get all this lovely information from the kernel. And and that's how we're able to trigger an alert. And it's like, OK, so that looks like Linux security.

Let's go up there. Let's talk container security. OK, so Falco is still running and let's go back and look.

Let's do some scary things in a container now. So we're going to run it interconnects and we'll do bean batch. This is the same engine container that you can you can run today if you want to go run it.

I'm going to download it here on my local system. It's downloading and now you can see where route at. And I'm going to do the same exercise, cat/etc/shadow.

Right. This is in a container and I can exit out of this container and now I'm back at Emily, which is the name of my host. And if we go back here, you can see now we have two other alerts.

We have noticed you shall respond in a container with an attached terminal and we have sensitive file open for reading again. So you can see that we're able to detect when people do things that we don't expect them to do. And in this case, I use regular Linux commands.

But if anyone would have used the kernel to open shadow in. And any way, we would have gotten an alert for it. It doesn't have to just be the Cat program.

Right? Like I could of you, I could have written another cat program. I could go write something and go or in Python and call it Cat.

So it looks and feels like Cat but it's not really a true version. That ship with your system and this is you see this all the time and how to exploit a system in different attack vectors because we're going through the kernel, we're able to send alerts and we do all kinds of great stuff in Falco if you if you go to log. You can come here and you can go to the documentation and you can even see these are all just some of the examples of things that you can detect with Falco.

So privilege, escalation. If you saw my fast and keynoted, I was able to escalate into a cabinet is hosting us using privilege, escalation with Kubernetes, Kops, a tool that I worked on because I knew how it worked and I was able to take advantage of it. Any time somebody does a namespace change using something like Saturnus, known directories at news or bad news for Espin, creating similar links, these are all things that we would consider suspicious.

And we have a set of rules in place that will alert you if any of these things happen. OK, so that's Falco in a nutshell, what it does. And now let's look at and running it with ECS and it has.

OK, so let's talk ECS. This one is a little bit a little bit tricky. And this is going to be relevant for you to really understand what's happening in EKS and how it's different.

So in this example, we have a ECS cluster up and running. It's called Nova-Hacks. And here you can see task definitions.

I have nova-hacks and you can see I have Falco Trace, Falco Trace Server and Vulnerable Web Server. And in the same way that you would create a deployment and Kubernetes, you can create a service and that service has a task and a task definition and that effectively is running the container for us somewhere in the cloud that we don't have to care about. So here is where things get exciting so we can go to Falco Trace, which this is just like something I put together in my free time.

You should not run this in production and this is just solely in place to show you what we can and can't do with a lot of these tools. And this is just what I use for me for my demonstration here. And you can see here we talk about how to run it with any container, how to use Kubernetes.

That is how to use SSH. And it goes into even setting this up with ECS, which is what we're showing now. And you can see how you have to come in and you have to define the Linux parameter and you have to add the P trace.

Right here. You have to add these P trace capability into the ECS, JSON, because that's how the driver works. So let's let's back up and let's think about this right here on my Linux host.

We have a kernel module in place for Falco. That's how we get the system call information. Now, we can't do that.

So we have the probe, which is dynamically loaded. So instead of typing this mod, welcome to the program itself, will actually execute a system call and load the program dynamically. And by that I mean it has the program basically loads the driver that it needs as it needs it.

And then in this third example, we use this P trace and we use this tool called P Digg to execute everything. So going back into ECS here, we have Falco Trace Association, Falco Trace Vulnerable server, and if you look at the Docker files for how we built these things. Here you can see we have SSH, and we have vulnerable server.

If you come in the SSH and you look at the actual docker file. P dig, dash a for all, an SSH and P dig is just a small... It's also written in C++, and this is what uses P trace to gather metrics about which system calls are being executed.

So any program that is launched with P dig will be tracked and secured with Falco. So all we do in this case is we were on P dig and we execute an initial script. So anything that you would put into the Shell script is now being watched with Falco.

So first off, Falco is watching itself. And second of all, we can then run the SSH as a demo and then you're up here, right? Line 18.

That doesn't exist would be your application. And so so that's kind of how this whole thing happened. So then we can go in and we can actually SSH into into this.

We'll grab its public IP, copy it here and we can SSH using route because it's a security demo and we're intentionally showing you things that are terrifying. Yes, enter our password and you can see I am SSH-ed into a container running in ECS. Think about what I'm saying, SSH, not exact into a container, running an ECS using Fargate.

Within this container, if we do ps ox, you can see that we have Falco running and you can see we have P dig minus A. And if we list slash dev, we don't see anything Falco because there's no kernel module running and we're not mounting slash dev from the host. You can see that this whole thing is happening within the context of the of the container.

And P trace is the avenue in which we're communicating with the hosts that this is running on. And by design Fargate just schedules, all of this in an isolated unknown sort of magical black box, easy to instanced for us somewhere, but we're still able to secure the system with Falco. And here's how it works.

If we come in, we can refresh our logs. And these are cloud watch logs, right? So what ECS does is, if you come in and you actually echo something, the standard out, it's going to show up here in cloud watch.

So let's go back to our container and let's cat... remember cat/etc/shadow? The same thing we've been doing this whole time.

We're going to get some information and you can come here, refresh our ECS logs. And you can see right here warning signs to file open for non trusted program cat/etc/shadow. That was me.

I just did that. And now you can see how natively that's working in AWS using ECS. So what did I do?

I created a container image. I executed an application. I secured it with Falco and I ran it in ECS.

That took years of work. So first off, that's cool. Let's take it a step further now and let's talk about EKS now.

And how EKS is going to use the same primitive and basically schedule containers for us in the Kubernetes that way. So this is this is that this is the final leg last few minutes here. So I hope you're paying attention.

So let's exit out of the ECS single container and let's talk Kubernetes. Better get notes. Here's where things get exciting.

So I'm going to authenticate with AWS. And you can see here we have three nodes running. We have this IP, which is an actual EC2 node.

So if we come here to my AWS page, you can see running instances. This is my personal AWS account, so you'll see my blogs and everything. But we have this this instance here called Nova EKS cluster and you can look down here.

It's 192-168-10-244. 192-168-10-244 right there. And then we have these two other nodes that I don't have visibility into and these are called Fargate.

Farmgate dash IP and not just IP like you see here. These are managed for us by the Fargate scheduler for 192-168-10-244. Check this out.

So we have three nodes running in Kubernetes right now. So let's do kubectl, run, ngin x, image, ngin x. Which this is basically like doing a docker run but within Kubernetes.

So it's going to take some time to run. But I want you to see what EKS is doing for us on the back end. So this is a pod engine x was created.

We can kubectl, get pods. On to. And you can see we have nginix and it's pending because what's happening right now, as I'm as I'm showing you this, what's happening is we're creating a Fargate container.

Fargate is creating a container for us in an EKS like environment for Kubernetes. But remember, it's different than ECS because it's got different concerns. We talked about those earlier.

We can exec into it and it's got a security group concept. There's a lot of subtle differences between ECS and what we're seeing here so we can get pods again. And if it goes according to plan, there should be scheduling any moment now.

It's still pending. So why are we waiting for that? Let's get a couple of other pods kind of in the queue so we can look at them in a moment and then we will wrap up and be done.

So what we want to do is we want to schedule I have a Falco trace hack container that contains all kinds of goodies that we can play with. So what we're going to do is we're going to do kubectl run. We're going to call this hack.

The image is Kris Nova Falco Trace Hack latest. We're going to expose Port 80 and that should do it. Exposed, and then we're going to do a dash dash court.

There you go. So that's going to create a service for us as well. So that's going to run the Falco trace hack, which again, you can get that.

I mean, that GitHub, if folks want to link to it, I can drop it in the chat. And you can see we have a service and a pod have been created. The Falco hack container is just it's got Falco, it's got P dig, it's got IMAP.

It's got all kinds of goodies that make it fun to explore Kubernetes, that is. And we're going to show you how we can do some scary things in how Falco can alert us in the same way that it did in ECS. So let's get pods again.

And if all goes well. To do. Yeah, our next party is running, so let's copy this and we can kubectl, exec, it, nginx..

bin, bash. And this is the big difference between ECS and EKS in Fargate. Right.

I'm executing into a pod now. You remember earlier we had to SSH. Not anymore.

So here we are. I'm in my pod in the default namespace. And here I can cat/etc/shadow again.

And Falco is not running to secure this nginx pod. So I've effectively, I'm unmonitored at this point. I can go in and I can do whatever I want to do.

And there's nothing that's watching me at the system level to ensure that I'm actually following the rules, so to speak. So if we exit out of that pod and we get nodes again, this is where things get exciting. You can see what Fargate has done for us on the back end.

Look how many nodes we have now. Notice we have five nodes because it's creating infrastructure as we need it. That's the whole point.

I'm only paying for servers that I'm using. It's really cool. And so now we have a new a new node for Falco and a new node for nginex.

And so now if we get pods again, we'll be able to see that the Falco pod is running. And then I'll do one final cat/etc/shadow and then I'll wrap up for the day here. So let's exec in the hack.

This way. And I exposed port 80, so there's I should have a dashboard built into this hack so we can actually access this dashboard if we want, but we definitely don't have to. OK, so here we go.

We're exec into the hack and if we ps aux, you can see that we don't have anything running and we can do a Falco minus you. Falco is running and then we can do that as a daemon and run that in the background. And we'll wait to make...

Clear some room here, there we go. And now we can P dig minus a root. What are we going to do here?

I'll just do P dig minus a... sleep, five hundred. Actually going to say, let's do this.

Let's do a bash. So now I'm running back with P dig. So it's running P trace to consume every system called executing Falco's, checking the device on this container, using P trace.

And if I do something malicious like that... cat/etc/shadow. log.

Haha, right here. Warning sensitive file open for reading by non trusted program user route. And this was all done using Kubernetes.

And so we... let's high level what we just did. Kubernetes is running in Amazon in a private subnet using ECS and Fargate, I created a pod running Falco, I executed a program in that pod using Falco and P dig.

We're able to hack the the system namespace is using P trace within that pod and into it and show you concretely how to connect all of this together. And and now I would be able to craft my infrastructure in such a way that I could secure all of my pods like this and I would be able to use Kubernetes native login and Amazon native login to actually to detect if somebody did something at the kernel level I did not want them to do. And I will only be paying for what I mean.

I feel like this is like a moment of celebration for Kubernetes over the past five years, this is the final frontier that we've been waiting for in DevOps for forever and we're finally here. So anyway, that's that's kind of the long and short of it. If folks have questions, let me know I'm here.

The Falco project is on the Kubernetes slack. We're happy to to help. We're happy to answer questions.

And if there's anything I can do, let me know. I'm here assisting the company I work for. We have enterprise technology built on top of the technology we see here.

So if you like this and you want to learn more about a future of working at this stage or talking about some of the tools we build, feel free to let me know and otherwise check out Falco. It's a really open source project. And hats off to the EKS team and the head of the US team for putting this all together.

For us, this has been this has been great. Walid says, "Hooray". So, yeah, I, of course know that.

Let me know if I can help in any way. And on that note, I'm going to let the the folks here I have like my back and know that we're done. So, thanks for joining.

And we'll see what they have the same. Thanks, everyone.