Stephen Chin, JFrog | DevOps Experience 2022

welcome everyone to my presentation on the iot landscape and I think there is No more fitting place to be talking about devops and iot. Then right from an airport you can. See a plane out there.

Which is parked outside the gate and this is this is the ultimate convergence of in-person and virtual conferences. So I was just giving a presentation here in Atlanta flying back and this is giving us the the mix of our hybrid and Virtual Worlds converging together. so What I'm going to talk about today.

is all of the iot devices which are taking over our entire landscape according to Gartner and statista by the year 2030 we're going to have more than 30 billion iot devices which are impacting our life which are everywhere. You probably already have some iot devices which you're using which you are taking advantage of which you're possibly even developing or or deploying software updates to And I think that as a consumer, it's great that we have all of these different iot devices that are convenience. But also that were able to push software updates and then people have come.

To realize that they need the latest software. It's always deployed and always active on their devices. And when we think about software updates, this is kind of the thought process that most end users.

Think about when a new updates available. So first they decide do they want it do they want to actually do this update? That might make a decision if it's riskier or not if they trust it if they can verify it.

Doing it verifications is very costly and this this process is really slowing down our ability to do quick software updates. Because as consumers we want the latest features, but the last thing you want is to have your device inoperable or to have it reboot and take a long time to do an update. And this is causing the bottleneck of human intervention where us being in the middle us being Gatekeepers of updating devices of pushing new software updates to iot stack is slowing down the ability for companies and for developers and software teams to continuously update devices.

So the way of solving this problem is having a robust continuous integration integration pipeline. Where you have different stages where you are making sure that your Cushing things through and integration testing system testing staging production. You have quality gates at each of these different areas to make sure that the right tests the right verification.

It's the right acceptance testings performed. And this means that for the end user when they actually get an update pushed to them. Then they don't have to worry about whether or it's risky or if they trusted they can short-circuit this whole process.

And immediately update to the latest version of the application and if you if you look at common applications which which you use like you do, you know what version of for example the the LinkedIn app or the Twitter app, or do you know, do you know what version of websites that you're using? And usually the answer is no. Because we're moving to a cloud-based SAS model people are expecting software to be updated all the time.

And this extends Beyond just applications to iot devices. So we expect our mobile phones. We expect our cars we expect our industrial appliances to also constantly be updating and also constantly getting security fixes and patches which make them.

better devices So speaking of industrial applications, let's stop for a sec to talk about and think about the next generation of iot which is industrial Internet of Things. and when you look at all of the different devices in in manufacturing in power plants and a bunch of different industrial applications, they're seeing the the advantage of having a modern iot architecture in the robustness the flexibility and efficiency that they get from using iot devices They're getting power savings and efficiency from having better Solutions and better technology. Getting better communication between devices where they can do use both wired wireless networks or even 5G and Edge networks to deploy.

Meeting safety standards by having better standardization and certification requirements which they can enforce with iot devices. Getting better security as well. So having better cybersecurity and security and we'll talk a lot about security and how this falls into.

but iot and iiot and then predictive maintenance figuring out when things are going to fail and then being able to update devices before they fail and making sure they have the highest amount of runtime. And when you look at the how their architecting the the networks to support these sort of implementations. There's a bunch of different components which they're using to support all of this.

So they have industrial control systems. supervisory control and data acquisition units human machine interfaces distributed Control Systems programmable Automotive controllers program logic controllers intelligent, electronic devices sensors and remote thermal units and a whole bunch of other sensors coming from a variety of different manufacturers all of which need to be interconnected secured and upgraded and when you look at the the network topology of how you bring all of these disparate systems together, you need a large Network converges together your your wired internet your wireless internet and makes a secure connection between all of these different devices. So that's that spans all the way from your Enterprise Network which has all of your your servers your DMZ gives you connectivity to Services, which Supply email identity management.

Call management Etc. down to your plant servers Which would typically behind me behind it industrial DMZ, and those would be all of your VPN portals your remote Gateway servers your application mirrors and different devices. You need to have.

on device at the edge Then site operations which allow you to control your factory do networking and storage arrays all the switches and distribution, which you need to supply your total communication Network for the device. And then different zones for sensors for controllers and for supervising different devices. Which support a variety of different interconnections based on the device whether it needs a direct switch or it needs a wireless connection, or maybe it's a 5G device and kind of converging all of these different devices into a network which is secure.

And can support different use cases like we talked about for different industrial uses. So when you bring all of this together in this huge converged Network. What happens is you end up with a lot of devices coming from a relatively small number of vendors?

So these are the top 10 vendors which are supplying some of the critical devices to iot companies. And the being able to secure these devices being able to update these devices is critical. to maintaining secure infrastructure and when you look at iot as a general area It used to be that oil spills were the most the most critical thing which stopped their industry and now security vulnerabilities are becoming the new oil spills.

Because when you have a critical outage in a power plant in an industrial setting that can affect and that can take down critical infrastructure and resources. Which the entire population depends upon? so here are some examples of ICS cyber attacks, which happened since 2010 and the the number in the severity of different attacks has just been increasing year over year.

There are a bunch of different nation-stators which have been targeting critical infrastructure of companies like dams and power grids. There's also a bunch of General exploits which affect devices across the board. Like the Windows 7 EOL update and a whole bunch of other security updates of older systems which expose them to ransomware expose them to different attacks.

And these are critical systems which everyone depends upon and when you you hit a situation where a targeted attacker can take down. A power plants or a nuclear power plant or an energy system. Or a dam.

This is critical infrastructure, which is being affected. Which is affecting us globally. And I think that just recently this is with the news today.

So The Russians have continued or their aggression towards Ukraine and they've been specifically targeting their power grid with missiles and and physical attacks. which is really unfortunate but this is something which has been going on for a really long time and let's talk for a bit about the 2015 Ukraine power grid attack. Where it wasn't a physical attack with missiles.

It was instead a Cyber attack where they got into the power grid and they took down some critical infrastructure. And basically the the way which they executed this attack. It was a trace back to a hacking organization in Russia.

Is they got in Via fishing emails and attacks on? people's behavior Once they got into the network, then they were able to spread out and get credentials. So inside the VPN, they were able to steal different credentials for critical servers.

Find that Network hosts and do Discovery deploy malicious, but build malicious software and then deploy it. Um into different systems, they ended up getting into the UPS's doing firmware updates. Uploading malware to a bunch of servers in order to to kill it.

And then eventually causing a large power outage in Ukraine. And all this is done without a physical attack. It's a completely Cyber attack on critical infrastructure and it shows the the danger to not having well secured systems where the the right people management of tokens and credentials are in place the right server management of the the network and security of systems and also making sure that you don't have zero day days or exploits which hackers can use to then compromise and get into critical infrastructure.

another example of a of a similar sort of Cyber attack was on the Saudi Arabian petrochemical. They penetrated the iot network got into an engineering Workstation. using social engineering we're able to download some critical files, which they modified and put exploits in and then the exploits gave them full access and control of devices and controllers which allowed them to shut down the chemical plant.

So again these sort of combination of cyber attacks and social engineering attacks allow dedicated attackers to get into critical infrastructure and cause quite a lot of damage. So in in thinking about how we mitigate these sort of challenges in our industry. Really the the biggest change which has happened recently.

is the devops team the devops developer has become the center of the security organization and The reason for this is all of the deployments all the updates all of the software device controls pass through. You as the devops team where you're have all the software releases. You have all the dependencies of different open source libraries.

And you can control what goes out. and when you think about your role in devops really your your All of these roles put together. So you're you're enabling developers your facilitating this appointment devices.

You have all the binaries all of the deployment artifacts. So you're single source of Truth. You're becoming a security expert because we all have to with new security exploits and software updates.

Whenever there's a production issue or a patch or release that affects the entire organization in particularly devops team. And you're helping to remediate and fix security vulnerabilities. So I think that coming to the devops experience conference and learning about iot is a critical part of how we secure the software supply chain.

And when you look at the the types of software supply chain attacks, which are happening. There's really two categories of different exploits. Which are happening.

So one is attackers which are explaining existing issues. And those are some of the attacks we talked about in the industrial Internet of Things. You're taking CVS or zero days.

They're finding secrets and configuration issues and they're using these as ways of. Getting into systems and exploiting them. But perhaps the more dangerous sort of attack is when attackers are now injecting.

malicious code be a public and private repos or they're getting in Via ideas and ci/cd systems to actually inject the malicious code which they can then use and the Saudi Arabia attack leverage some some of these techniques in order to break into their into the Petra chemical plant. And I think what really brought this home for all of us was the the recent attack of solarwinds. Where they broke into a system in the supply chain got into the ci/cd server modify the binaries before they were signed and they used this as a way to then attack Downstream organizations.

And these sort of supply chain attacks are increasing in frequency. They're becoming a new critical piece of infrastructure, which needs to be secured. And it's affecting the whole industry just like the recent log for Shell incidents.

Where? We were all patching systems last January when this came out because there were so many critical systems so many affected systems that we had to update. And this is just really the tip of the iceberg for what we need to do to secure.

The entire software space but in particular iot devices and the iot ecosystem. And all of the software we build is constructed from open source software over. 80 85% of software which is contained in a typical Enterprise application is composed of Open Source.

And the open source software comes from one or more of these systems you're downloading software and trusting. Npm, you're getting things from PI. You're getting things from rubygems or Maven Central Etc.

And one way of thinking about any of these systems is when you are pulling code from a external repo. It's the same thing as plugging in a USB stick off of the pavement. So you're exposing your critical software systems to a relatively untrusted resource, and I'll explain did a great attack leveraging and PM as a supply chain attack to get into Enterprise systems and the mechanism for doing this is called the dependency confusion attack.

Was taking advantage of the fact that vulnerable package managers. Instead of going out and pulling the library from the internal repository. They'll go out and they'll check for example npm Pipi and external repositories to see if there's a newer version.

And if you are pulling the dependency from an external source and going around your internal Repository. This is giving you giving an attacker the potential of publishing a malicious Library which gets pulled into your system as a seemingly updated version of an internal Library. Another problem with this also is when you make requests out to npm.

It's in the clear text for what libraries you're requesting. So an attacker doesn't even need to know what your internal libraries are because they can just sniff the network traffic and figure out exactly which internal libraries you have based upon the requests being requested off npm where there's no version available. So this is Again, like a huge security issue for companies.

It's something which you need to patch or be aware of so that you're not pulling in vulnerable libraries one way of doing this is putting a package manager which understands the difference between external repositories internal repositories and shields you from pulling in libraries from public repositories where it's clearly an internal corporate library. And if you don't do this, then you can be attacked Alex Pearson made a total of a hundred thirty thousand dollars author dependency Confucian attack. Simply by doing bug bounties on different companies.

So fortunately he wasn't using it for malicious purposes. He was just using it as a as a way of collecting bounties on companies, but he did catch. Apple Facebook and a whole bunch of other companies with vulnerable systems to this particular exploit which he proved.

And he was able to collect legitimate bug bounties off of them simultaneously. So it's perhaps one of the bigger bug bounties which has happened recently. And another example of a vulnerable package manager is or vulnerable Central Library is what happens?

With the Azure libraries, so the jfrog security team found instances. Where the Azure libraries were being targeted. for typosquatting and if you left the namespace off, so for example, the Azure core tracing package.

They uploaded the exact same package without the Azure prefix. And if you left off the prefix, then you would pull down the malicious Library. Instead of pulling down the legitimate Library.

So again, this is a attack on in this case everyone using Microsoft Azure as a cloud service. Depending upon you know developers to to mistype or to shorten things which often your ID or you just do at a convenience. And there were 218 packages affected at Azure at Azure test at Azure tools at cataling.

And they were very clever in how they did this because they published with different user IDs so that you couldn't tell that these were related and they had also created scripts to automatically generate and do the quickly create all these different user IDs package them. Now our security team reported this they were all taken down and it seemingly they didn't get a chance to actually use this for any malicious use But you'd be aware that these sort of attacks are happening and other very prevalent. So one project which we launched recently jfrog to address the security open source, supply chain security is called, Persia.

It's now a continuous delivery Foundation CD Foundation project. It's vent or neutral and it's entirely open source, and the goal is to make it. Easy and seamless for a software developers to securely publish their libraries without being victims of type of squatting of someone maliciously changing the library solarwinds or somebody who is trying to trying to subvert the open source supply chain.

Now the project has Greek Origins, like any good Cloud native project needs and Persia comes for is actually an ancient Greek system for communicating over mountain tops. They would use 10 torches and two arrays of five. Kind of use this on the Greek alphabet.

So then communicate letters over long distances and we want to accomplish the same thing for open source software to Reliably and securely publish libraries across the entire open source ecosystem and make this a secure reliable and open. Source way of receiving libraries and dependencies from open source developers and we think this is a critical link that's missing today and a critical piece of infrastructure that will help to improve software supply chain. now there's three parts to how this technology works the First part is having a secure and reliable distribution infrastructure.

So we have A peer-to-peer based Network the backbone of this is the authoritative package registry is which are run by the different companies which are supporting Persia. Then there's a clients that you can install on your cicd server and your developer machines, which let you pull packages. from Persia from the secure network It's a drop in replacement for your typical cicd or for your typical Central repositories and it gives you a more secure mechanism for getting exactly the same code.

The second part is doing verification to build. So we do multi-node verification to build across the authoritative nodes where you're not relying upon one company is or one individuals build. you're getting it built on multiple different corporate servers simultaneously and then they're being verified against each other and if the verification fails the Package doesn't get published to the network.

If the verification succeeds on all the servers over the verification that it will publish it to the network and what this does is this make sure that if somebody wanted to attack the network. And pass off a malicious library or try to inject something into the network. It would require them to simultaneously subvert all of the different servers which are doing these to build verification.

Which makes it a much harder attack surface than getting into a single corporate Network and if you remember from the iaot attacks. There are a lot of social attacks which can be used to get access to corporate networks to the critical infrastructure. Saigon builds even from large companies like Google Microsoft Amazon or others you basically have a single point of failure.

If a dedicated attacker can it get access to their internal Network? In the final aspect of this is the immutable transparency Ledger. So the this is the backbone of the network.

This is how you verify the libraries you're getting it publishes all the information about the published artifacts who's verified them who signed them and can be used to create software build materials down the line so that you can do the full attestation and provenance. Of where your artifacts came from? So, um, we announced percya joining The Continuous delivery Foundation the companies which are currently involved in our jfrog Docker deploy Hub Huawei feature way and Oracle who are all supporting this project.

Were also open and looking for new folks to join the project join the network and help us to secure the open source supply chain, and we think there's a huge opportunity here. To make the entire open source ecosystem a better place. Okay with that, I want to thank everybody for joining this presentation about hacking and securing.

The internet of things and what we're going to have next is we're going to have a panel. With some of the different experts who really understand the security space and have some great practical knowledge about what's happening in the ecosystem and one of the folks who is joining us amidst serper. He actually discovered the not petcha exploit, which is one of the iaot exploits, which I had earlier on one of my slides.

And it's exciting to see and hear from some of the the researchers and the folks who were actually working on the state of day also be joining the panel. So thanks everyone for joining this presentation and a couple minutes. You'll be flipped over and you'll be able to see the instrument of things panel, which we're going to put on.

So, thank you.