Matt Johnson, Prisma Cloud | DevOps Experience 2022

Hey everyone pleasure to be presenting here to you all at devops experience. My name is Matt Johnson, and I'd like to talk to you today about SecOps and the edge or supply chain security practices for Edge deployments. So a little about me, please do reach out with comments questions for the discussions.

We have a all things infrastructure as code security slack down there and I know I'm gonna be talking to you about security. So I promise you this QR code on the screen is just my contact info but you know take that with a pinch of salt. I'm Matt Johnson.

I work for Palo Alto as a desktop Dev set up Advocates. I've been playing with containers since before they were called containers before Docker was Docker and always kind of leaning on the security angle and in my spare time. I've got a growing collection of single box Scotch.

I'm learning to be a Private Pilot not that those two things go together, and I'm based in not so Sunny Manchester in England. But anyway enough about me, let's talk about Edge and deaf secops. So first of all, I'm going to go through a couple of things that aren't necessarily related to Edge but are important for the wider context.

So first of all infrastructure is code security. If you're not familiar with it the idea of preventing misconfigurations, when we are deploying to places that we can codify our infrastructure like against Cloud providers apis, for example, and the reason this is important is if we look at the OS 2021 top 10, which used to just be all about cve's it used to be, you know, these development mistakes SQL injection and you know, writing your own crypto functions instead of using a validated crypto Library, we can actually see that misconfigurations in infrastructure as code are now just as common for you know, top 10 list like this as the reason for a successful attack or a successful intrusion as the actual code basis themselves things like broken access. control can easily apply to infrastructure configuration as much as the application itself same with identification authorization security logging and monitoring and it literally spells it out there in a05 security misconfiguration and you could argue that ao6 probably could be infrastructure as well depending on whether the dev teams do what don't manage the underlying a West and things like that, but we'll get into that the point being infrastructure misconfigurations are just as important as cve's so that's kind of the first thing we're gonna dive into especially when we come to Edge deployments and why some of these things are really important And so Shameless plug we have a open source tool called checkoff.

It is designed to scan for IAC misconfigurations here. You can see it looking at some terraform for AWS and going. Hey, you probably don't want to have that S3 bucket public and you probably want to enable authentication on your RDS cluster for example can scan kubernetes and other things pretty much every IAC framework you can think of but that's the kind of thing.

If you're not familiar with IAC security that we're talking about when we're looking at a misconfiguration. It's not a cve it's not a vulnerability per se it's maybe someone writing a piece of telephone for the first time that doesn't know you need these extra lines and parameters because the defaults as interpreted by that terraform might be insecure. And we can also do kind of things like automatically put this information into pull requests again better than having to run extra commands as part of your rights and code hit commit, you know, get push actions.

So here we see an annotated terraform cloud with automatically checking and failing because of misconfigurations and also the familiar GitHub actions with automatic annotations for a particular resource again an S3 bucket with a public read ACL highlighting that that's probably not a good thing on a bucket. And then we can go even further than this rather than waiting till the pull request for those again not familiar with the kind of West instructors code security has been going over the last year or so. We can also get to the point where it tells you directly in your IDE when a developer is writing be it kubernetes manifest Helm chart piece of terraform Etc.

And so that you can find those issues and know that they're going to cause problems further down your CI/CD pipeline before you even save that file or make that check-in so super useful. Again, keep in mind infrastructure was code security. Hopefully that was enough of a primer as we continue.

The second thing I want to talk about is supply chain security. And at this point you're probably thinking what supply chain what does this have to do with Edge security, but you know, let's set some contacts then hopefully you'll see it all come together quite nicely. You have probably seen something like this slide from thousands or at least tens of you know, marketing or sales pitches the idea that you know, we build up our application deployments.

Our whole stack isn't just our own code. It's built up of you know, the dependencies we pull in through pip or npm or you know, any of the package managers for reusable pieces of code for whatever language you're using go modules, you know, pick your favorite and I'm not biased and the whole idea of reusable code is great. There's so much that goes into a modern application.

It's estimated that between 17 and 90% of an application will be someone else's modules. You are using as a dependency or a dependency of that dependency or dependency of a dependency of dependency, you know anyone that's tried to download any kind of dependency through npm or generally know what trust it install the whole world onto your developers machine. So probably familiar with that but the point is reusable software is bringing components in that we're not necessarily sure on the safety of and therefore we've had you know, Composition analysis added to this list of security things we need to care about to make sure that we can have a look at what versions of What dependencies or applications are needing and making sure that we don't have a vulnerability in a version we're using as a dependency either directly or as a transient dependency of a dependency of a dependency.

Again, why am I talking about this when I'm talking about Edge and infrastructure in particularly at this point and the thing with infrastructure is code when we're deploying into Cloud providers or kubernetes or with terraform or cloud formation or full stack doesn't actually look like this instead. It looks something like this because just like it's really useful to reuse code for your applications and that same concept of modular code being reused is also alive and well in the world of infrastructure as code you might may have heard of artifactor where you can find thousands of Helm charts or customize kubernetes manifest which can be used and customized to your need or just use directly. So you don't have to write your own kubernetes manifest for deploying a third party piece of software or some open source tool and similar with the hashicult terraform registry where you will Modules for creating vpcs rather than having to create write your own terraform for you know, eight to attend different interrelated components looks like security groups and root tables on all of that good stuff.

So in this scenario, you know, we are still reusing code and it's not a bad thing, you know code reuse is good more people use that piece of code more people secure that piece of code Etc. However, where this falls down we find from a security perspective is the readme the getting started guide and it goes back to that security versus usability problem, which is as old as time or at least as old as security I guess and we actually did some research last year where we took every home chart on artifactobio and ran it through our instructure as code policies and found that with the defaults more than half of those Helm charts were insecure and insecure with some pretty big issues such as a lot of root container use a lot of raw network access capabilities not explicitly banned from those kubernetes manifests. And so Again, could you modify the parameters given by the author of that houndshot to make it secure?

Yes, but do a lot of people just find a solution and run that home chart with the defaults. Yes. So again, like we have to care about our application dependencies.

We also have to care what we are sucking in in terms of our infrastructure as code. And then there's other things that we need to care about which can give even the nicest infrastructure as code security a not so clean bill of health because it's all well and good having our secure kubernetes templates or secure terraform templates. But how is it built?

We're not probably going to deploy that ourselves. We're not going to be running Cube CTL commands across our infrastructure. We're going to have some alternation and some CI/CD pipelines doing all that work for us, testing bundling building or packages and pushing that to our production or Dare Board Test environments.

And you know pick your favorite that might be done through GitHub Actions gitlab telephone Cloud, maybe Jenkins. You're probably familiar with with one of those at least and This gives us actually another attack Vector. My colleague Steve did a really really good presentation at the cloud native security con last week in Detroit.

You can watch it on YouTube at the link below and it's about how easy it is. Actually again, if you were kind of using example GitHub action manifests to pawn someone's public see, I repo so tldr for example if you're using any Use a provided input in your CI pipelines like hey modifying the issue title, like looking at the branch looking at the tag, which can be used as applied. It's very possible to then, you know exploit those user inputs to effectively run your own commands and end up running your own code in someone else's CI/CD pipeline, which is obviously not what you're going to want to do.

And that's just a couple of examples of things, you know, we always talk about defense in depth or security as a journey. Not a death, they donation all these kind of silly security mantras. But in reality the the modern software Supply and delivery pipelines also complex that here are just a handful of the things you do need to think about whether you're thinking about them or not forward deploying modern software, you know, we've just talked about the templates.

We've just talked about insecure CI/CD pipelines, but there's a load of other things there as well. And in Cloud deployments, we are getting pretty good at ticking most of these boxes when we have a cloud provider. We know that you know from a from a security vendor perspective when we have a cloud customer that is running in a cloud provider.

There are solutions in kind of more towards the runtime area. We have apis that we can get to to validate that hasn't been any drift to make sure we're not seeing any manual modifications compared to what was defined in our instructors code templates. We can scan the Ian, you know AWS using myself so we can scan the IAM logs and the I am apis to make sure we're closing down any overly permissive access in that environment as well as all the things we can do with tools like check off to prevent insecure templates and insecure CI/CD pipelines and things like that.

And it's with those standardized apis and known kind of destinations. We know we know the Amazon apis. We know an Ami image is an Ami image.

It's a unique and we can scan it and things like that that I am going to provide this image to move us on to the edge topic with a massive pinch of caveats. I am going to say see the stars there. I promise I'm gonna caveat the hell out of this.

I am gonna say that edge runtime. So these two kind of sections here deploy and run or a lot more Niche than Cloud because you know the cloud apis the cloud. Products and solutions that you have in each cloud provider the terraform and the IAC are well known well documented because they have hundreds of thousands.

If not millions of users Edge generally speaking again caveats aside is more Niche and that means that it is harder to do the bits of security that we like to do in these phases of the of the pipeline. And again so many caveats I would love to do an entire like conference on what is Edge. Anyway, I know that people are gonna be saying no.

Well, my Edge is just you know, for example ECS anywhere. It's the Amazon control plane running my containers in my own Hardware. It's my Edge is GitHub action, but using my own personal Runners, it's you know, again Cloud management control plane.

I don't have to worry about it. There's lots of different definitions for Edge anything that basically isn't in a cloud provider and isn't in your traditional Enterprise seems to be classed as Edge by someone or somewhere, but let me just give a few examples and one size is not going to fit everyone here. So please take with a pinch of soul and kind of apply the bits that do apply to you rather than the bits that don't but you know an edge device often will be low power iot kind of rural base station.

um access to maybe temporary or daytime only or fluctuating power delivery mobile solar applications and bedded systems, you know, either low power in terms of the power delivery in terms of physical power to run them is limited or low power as in very low compute power because of embedded systems, for example And they might be low bandwidth super remote locations really low bandwidth but long distance radio links, for example, or really expensive uplinks like satellite which have to maybe be scheduled and can only be used at certain times. So, you know, not permanently online. We also then have you know, potentially Edge deployments which are unmanaged in terms of you know, there isn't a control plane running and provided by a cloud provider to check that everything is healthy that are latest operating system has been updated that or even just Warners that there are problems that need looking at it might be very much a role your own set of off the shelf Hardware which has been used for some kind of edge deployment and again, and then it's systems which aren't as easily upgradable and may lack the compute power to run kind of a full-time management layer on top of them.

And then again with that role your own or with that kind of low power embedded systems and we might not have an API like kubernetes or ECS in Amazon for a VM, you know, it might not be that kind of device where we have the luxury to have a deployment API which is well understood and we can just kind of throw containers with throw, you know fat VM images add a Remote device to upgrade it. It might be a much more complex or much more manual automation. You know, I know that's a bit of an oxymoron, but hopefully you get what I mean that And so for you know security related things.

Depending on which of these factors affect you these can come with some interesting challenges. So, you know, I've seen small iot devices which are you know, happily Internet connected but don't have enough power to run an SSL or a TLS Library. So how do we do secure even if we wanted to do over the air updates and metrics?

How do we do that? Securely? Equally with low bandwidth.

If you can't be online all the time, then real-time alerting goes out of the window. How do we understand any indication of compromises on our application but also with an edge deployment, you know, it's very possible that that application is the only thing running on that device but that comprises a full stack you have an operating systems think about you have all those other areas of potential compromise to think about on those systems again with unmanaged systems, you know kind of taking an opt background look at this like we have done with kind of infrastructure and infrastructures code misconfigurations who updates the OS and how if you have low power or low bandwidth, you know, and if you don't have a application lifecycle API, you don't have a kubernetes endpoint or something like that push applications to then you can't really use IAC anyway. So how do you do infrastructures code security if you instructors?

in the first place and so back to our diagram of kind of things. You should be thinking about when you are securing your application lifecycle and infrastructure life cycle, as you can see a lot of these issues relate to that last section kind of the the further we get towards runtime the deployment the running, you know, how do we get to things that a cloud provider kind of take makes us take for granted but we may or may not have in a edge deployment or you may have again take the ones that matter to you. I don't know where a vegetablements which are basically like Cloud providers in Telco data centers.

So obviously very different you can kind of just treat your deployment as a cloud and use something that can deal with the full stack of these like Prisma Cloud to to secure it. So what can we do, you know not all doom and gloom? the kind of Defense in depth You know Mantra that we're talking about earlier.

It's not really in depth is it it's more that they're just are so many components in this diagram the you know, one type of security one type of tool one type of check doesn't fix them all so, you know, there's still plenty we can do here even if the kind of last sections aren't unknown and so, you know my kind of take on this and what we've been doing with kind of our own like more remote deployments is kind of double down on knowing your environment earlier in the process as much as possible because you can use that data in a much lower power lower CPU intensive way later on if you do the work build, So for example, I've stolen this slide from another of my talks, which is a lot more about cloud provider based deployments and infrastructure as code security. And even there I am saying just because you have no issues with your instructors code you write most secure instructors code ever, you know, the Amazon gurus with all the certs are gonna look at your Amazon VPC and go that's nice. I like that just because you've done all that doesn't mean you're not gonna have security issues.

You know, you need to check that the packages you're deploying in your ec2 instances and your Landers as dependencies in your Docker containers. You obviously gonna still need to check that. There's no CVS in there.

You're going to need to continue to check that. There's no new series on the things that were clean when you deploy them as well, and we need to protect our CI/CD pipelines from abuse and take over protect our source control repos from a recent take over. Check we're using the images with think we're using signing or binary signing our images to make sure that someone hasn't abused or image registry or taken over one of our Amis or our build process and obviously table Stakes like making sure we're not spamming secrets to public repos and things like that.

And so a lot of those as you can see we can do whether we have an IAC Cloud endpoint at the end of our Edge Journey or not. So the first one I would recommend and I'm going to put some further reading below for if these kind of tick certain boxes and apply to your version of Edge and know your images and if they've changed, you know, use things like the ability like Cyclone DX and s-bomb the software bill of materials to know exactly what's in your image be it your application deployment image if you're rolling your own Edge, or you're not part of a managed service or even if you are or something like Amazon ECS where they still mandate that you have an operating system and then install their stuff on top generate an s-bomb for that OS know exactly what versions of what libraries and your dependencies and your runtimes know exactly what is in your system talked about on as as you deploy because then you can go back and not only keep running that SBOM completely offline from your Edge against recent cities to see whether you are still secure or you need to push. New Image but you can also keep validating those signatures against your Edge in potentially low power way like a scheduled job to see whether any libraries have changed and if you don't have over the air updates and libraries have changed or files of change, you know instant indication of compromise there maybe not as real-time as you would with kind of cloud provider apis, but still a good step to have that known signed referenced esport.

And here's a screenshot from the Prisma Cloud platform where you can actually see we break that out into more of a supply chain graph. So you can see if you do have infrastructures code. If you do have Docker files, if you do have application dependencies, you can kind of see how they all relate to each other and almost kind of visualize potential attack path to help prioritize.

Maybe if you were just starting to think about securing an existing application stack and it might help you work out, you know, where you are highest priority items to tackle or And automate as much of your knowledge tasks within secured CI. Like I said CI can be an attack Vector itself. So be careful check out Steve's presentation, but also like anything that you are pushing to your Edge.

It will be building. It will need testing. It will probably need to know us like automate the packaging and building of all of that in CI so you can run.

You know. As many security tools and as many tests as make sense against that you can generate those S bombs automatically, you can sign those binaries automatically that you're building. There's some really cool ways of running like Linux kernels in sign binary only mode.

So if you're generating everything that needs to be on a low power Edge, you can make sure that unless it's one of your assign binaries. It won't run all of that can be it sounds complex. But when you're automating all the signing in CI all the testing in CI and it's actually not that much work because you're not doing any more work you're doing it once rather than every time you deploy.

So yeah, ultimately as much of your non-edge tasks anything before you hit that new image onto your Edge do that with CI and make sure you're securing your CI and securing your GitHub actions. Again, we can with open source tools like check up. We can check for issues in your GitHub actions manifest and get lab manifests that might allow that CIA to be compromised.

And then obviously scan for the basics on the build again tools available to put in your CI compute CI costs. Nothing realistically, especially for public repos with things like GitHub actions. Make sure you're not using Docker containers or images with a load of cpes.

And obviously if you do have an API driven runtime, if you're looking up to run like k3s or some kind of kubernetes endpoint at your Edge, then do use IAC security tools. Make sure you're not making common mistakes. Make sure you are, you know doing everything that the CIS for example kubernetes guidelines recommend.

You should be doing for that deployment. And that gets you a lot of the way there again, like as I said edges Niche I stand by that like there's more chance of me hitting something. I don't recognize on an edge compute deployment than in a cloud provider, but we can kind of defend ourselves against a lot of that and give ourselves the data to verify or Edge deployments as regularly as we need to by kind of treating the first few sections as if they are a cloud deployment and taking the same supply chain security precautions that we would take in those environments.

And that is it from me. We've got a write-up of kind of those General steps for software security supply chain checklist at the link there by all means do reach out to me. I'm at metahertz on Twitter.

And as I said, we discuss all things in structures code Security on our slack channel, so thank you for listening and hope you enjoy the rest of the virtual conference. Just