Docker Hub Insecurity – Cloud Native Now Podcast EP12

Hello, and welcome to the latest edition of the Cloud Native Now podcast. I'm your host, Mike Bazar, and we're with Paul Nati once again from Theum Group, and we're talking about the latest and greatest events in the world of Cloud Native. And there was a shocker last week, JFR posted a report saying that there are millions, quote millions of these repositories sitting on Docker hub that contain not malicious code, but rather malicious content.

And I guess what it was is that folks started using Docker hub to share quote unquote marketing materials that bad guys figured this out and started putting up everything from eBooks with malicious links in it to well ads for games that took you to sites that downloaded malware onto your developer environment, or at least took your credentials one way or the other. Paul, are you shocked by any of this? I mean, it just seems like there's millions of these things and the potential for, uh, the number of developers to have their their credentials compromised is just staggering.

Yeah, no, Mike, well, well, let me, let me first off say thanks for having me again. Uh, this is great, uh, being on your, this, this show and, and the episodes here. It's, it's really, really helps expose these, these challenges.

You know, I, I have to say, like, you know, don't these hackers and security threat people, don't, they have something better to do with their time? I mean, it's just kind of crazy and it's like they just really gotta get out there and just kind of make, you know, make people's lives miserable. You know, I, I I think one of the things that, um, uh, that, you know, when I was reading this, I was, I was kind of brought back to the OSS, uh, the, the summit, uh, right, that just happened recently.

And we talked about the backdoor vulnerabilities that happened, you know, with that. And, you know, that's, you know, I'm, I'm, it had me thinking like 1984 Matthew bro, war games, like, haven't we learned anything? Like back doors are not secrets like, you know, around that.

But when I look at the J Rog, how it reveals, uh, you know, Docker hubs, uh, compromising the, the, the, you know, the, the issues in the, in the hub and the repositories, it just amplifies the fact that code has to be checked and tested earlier in the process, right? And, and it amplifies the fact that, um, you know, putting out, uh, to basically putting out code faster, you have to slow down to go faster, right? Um, you know, what we, what we see in, in trending data, right?

What we see in our reporting and our trending data is I, so in 2022, I ran a, a study that showed, uh, testing in the, in the CI/CD pipeline, and it showed that only 29% of organizations were doing, um, continuous testing, which is, it blew my mind. So all of 2022, in 2020 going into 23, I'm interviewing, talking to people about this. Why is this, why, what's going on?

And, and the, the response was basically, well, because they do sprint reviews so rapidly that if they catch a problem, uh, they can push it out within a, a rapid, uh, you know, statement or, you know, within two weeks or so. And, and my response to that is, well, you're really letting your end users be in the Guinea pigs of the testing. And that opens up all sorts of challenges, right?

So what we saw in 2023, that number one, from 29% to 66%, uh, which I really, I hope I had a little bit of influence over that, but I, I really think the industry realized, you know, testing needs to be moved closer to the developers needs to be closer, shifting left, and making sure that that testing of those, those repositories, the data that comes out of it really has to be, uh, valid. How much personal responsibility should developers be assuming for all of this? Because to your point, I mean, not only are they potentially downloading malicious containers, but they're also clicking on things that they probably should know better not to click on.

Yeah, I mean, you know, it's a great question. Uh, accountability needs to be held at the organization for sure, right? I mean, we, we can't just ab absolve people for, Hey, I downloaded something I wanted to get, you know, push code out faster so I push, you know, use this, uh, this common, a common set of code base from, you know, from a repository that doesn't give an excuse to just say, okay, well, we introduced vulnerabilities, right?

That has to be tested, right? And earlier this week, I had a, a, a presentation with Alan and, and, and we did this a session with Pint, and they were talking about, you know, the API integration, the API security calls and such. And, you know, and in part of this discussion, it was great dialogue.

'cause we were going back and forth about, you know, all the pieces that have to be, uh, that go into, into play. And this conversation, this comment came up, Mike, and the comment is, you know, what you just said, it's like when, when organizations KPIs are to push code out faster, it doesn't give you the excuse to say, okay, well we push it out because we just grabbed something off the shelf. It has to be tested.

So it's not just testing things that work in connectivity, but it's also about ex uh, exposing those vulnerabilities that happen. And those, those testing tools that are in place a lot of times are in place to just, you know, uh, are available to just use to, um, you know, for free to try it out. And then you can kind of say, oh, is this gonna work for my process is gonna, is gonna work within the, uh, the, the flow that we have.

So I I, the accountability is on the organization to test it. It seems like there's a relatively simple fix here in the sense that I should just constantly rotate the credentials of the developers. 'cause that's what gets stolen most often in this type of attack, and yet we don't.

So what is the issue with kind of just, we need a more rigorous approach just to simple credential management? Yeah. Credentials is, uh, is definitely an area that, uh, you know, obviously protection of your, of your, uh, your authentication is, is critical, right?

That's very critical to, but like, when they're pulling data, when developers are pulling data and, and, and code sets from specific areas, they have to make sure that that data, that that code set is actually going to be what is expected, right? And, and, and testing that code set before implementing into production is, is key. So when we look at, you know, what, what, what's going on with, with J Rog and, and how they kind of looked, you know, put this, uh, exposed, the fact that there's these thousands of repositories that were, you know, impacted.

It really doesn't, as I'm going back to it, it doesn't, it's not just a matter of authentication, it's a matter of checking what you're actually pulling down as well. Do you think in general, I mean, you were talking earlier about the trade off between speed and security. Some folks will argue that you don't have to make that trade off, because theoretically, at least that, um, if I'm going faster, I am working with smaller, uh, code segment changes, and then if there is an issue, I can narrow the fix to that small amount of code versus trying to, uh, fix a massive amount of monolithic code.

So they argue that the whole cloud native process is a little more secure, the faster you go. But this assumes that we are actually running the tests in real time. So, you know, are we still just exceeding the ability of the security tools, the key pace?

Yeah, it's a good point. Um, microservices and having, uh, smaller attack surfaces certainly, uh, reduces the vulnerability, right? Because you have the air, a narrower focus.

And then, um, being able to do a rapid updates to that microservice, uh, really allows for that, you know, very kind of strategic, uh, you know, uh, tactical kind of just go in and, and pull what you need to what you do. And it doesn't impact the entire mono, like, as a monolithic application would be impacted. However, that doesn't, that still doesn't mean that it's okay, right?

Because I think that if you look at it, any surface level attack or any, any vulnerability that can that comes in is, is basically exposing that company to a number of things. Reputation risk, monetary risk, customer impact data, all this stuff, right? And, and, and fines, compliance, et cetera.

You could go, the list goes on and on. So even though the surface attack space or area is smaller with the microservice and cloud native approach, it's still, it's still there, right? And it just means that, you know, and that goes back to what I was saying in our earlier study where we had 29% doing testing that that wasn't acceptable.

Because 29% is still saying that, um, they're just letting code be tested by the end users. And that's not a good thing to do. I guess my issue with the whole idea of, um, going faster and still being secure is it assumes a level of maturity around DevOps workflows that I just don't think is there.

I think most organizations or, you know, they use DevOps no doubt, but they're not at the level of a Google or Netflix or somebody like that, that has that kind of cadence and the ability to manage it. So, um, maybe, you know, you need to decide as an organization to what degree can you speed up and be secure, or do you need to slow down and there's no shame in slowing down a little bit. Right?

Absolutely, Mike. And you know, when I look at it, uh, you're, you're spot on. And this is something I've been kind of touting for, for quite some time.

Uh, you know, when we look at organizations' goals, right? We've seen that organizations are trying to push code out on an hourly basis. Uh, 24% of respondents indicate they wanna push their code out on a, of that cloud native survey.

They wanna push their code out on an hourly basis, yet only 8% are able to do so. But the question hits spot on to what you're saying. The question is, does that 8% really, uh, are they doing this effectively?

Are they doing this properly? Right? And, and because DevOps is sitting there going, no, and, and no offense to DevOps folks, but they're pushing the code out the door because they have a business, KPI, it says, get it out the door, right?

Um, but the reality of it is they may, you may have somebody flagging it, uh, you know, in the developer side or the QA side saying, Hey, there's a vulnerability, there's a risk, there's a challenge. But DevOps going, we need to push your code out the door. 'cause that's a business KPI, so moving and shifting left to having the testing done and having DevOps tied to platform engineering and engineering closer to developers, that means you have more exposure to the application as it's being developed.

It can be properly developed and pushed out. But I also think organizations need to, uh, reflect their business KPIs appropriately in order to meet those business goals effectively. Mm-Hmm.

So ultimately, is the KPI the root of all evil here, because that's what's encouraging people to kind of do some things that might be not in the best interest of the organization as a whole? Yeah, I mean, I respect the fact that business wanna be more agile and, and, you know, be competitive and push stuff out. Especially when you're looking at, you know, any, any malware or dangerous issues that kind of come up or viruses or whatever may come out, you have to adjust your code very rapidly.

But, you know, this is where I think there's a, where, you know, and I said I think we're in the maturity. We're early stage maturity here. There's a combination of AI that can work with DevOps that can do this, which is pushing, putting the, uh, adjustments and changes to push code out faster, um, within an automation process.

And if you can automate it, you can kind of cut out some of those challenges, right? So, you know, tying DevOps and DevSecOps together will help with the security elements of it, but also, um, the automation pieces, once you get those workflows built, that should alleviate some of that pain as well. All right.

Well, other things going on in the world of security as it relates to cloud native. zero or slash one, depending on how you want to count of, uh, tetrad, cilium, cilium being a virtual network that connects various, uh, Kubernetes clusters together. I guess it can connect other things.

It's primarily used for Kubernetes clusters. Um, thanks once again for the open source community for making it easy to track exactly what's going on there with those virgin numbers, because, well, I think maybe somebody should have a little chat with the marketing folks about that. But as the point of the whole thing is this is a nice lightweight framework for managing cybersecurity policies that plugs in agent software directly into eBPF running in the Linux kernel.

So, um, I don't have to have as many agents running around collecting data from all these, so are we getting better about security and observability? And maybe we don't need to lean so much on the Kubernetes community to do that. Yeah, Mike, you know, I mean, I, the, the, this, this announcement is from, uh, as of Allen is, you know, is really adding quite a bit.

You know, I, I, I can appreciate your comment around the version numbers and such as a former, you know, uh, you know, marketeer and such. It, it always used to kill me on the TikTok release. It's like, which version is the main version?

Which version is the, the DOT release and such? 13 release is, is very robust. I mean, we're talking about Kubernetes identity awareness policies.

We're talking about, uh, default rule sets that are being put in place, um, redacting filters, sandbox policies, you know, there's all sorts of things. Uh, kernel, cce, CBEs and protection, uh, OpenShift support. And then the list goes on.

There's just a number and number of the things that are here. But to your point, uh, the, the advancements to simplify, uh, you know, the security elements is really going back to our previous statement, right? When you look at the tech stack that's been introduced here, all these feature sets are really trying to say, Hey, look, we have these, these abilities to really focus in and tune in on making, making your code and your applications faster.

But now you have the ability to take those individual feature functionality that I just mentioned and put them into that automated workflow, right? You can add it to that automated workflow. So now you can actually speed up your development, but making it less complex, even though it seems like there's a lot here to, to digest, which there is, the complexity gets reduced because you can add this into your workflow to address all those different challenges.

Meaning one other part to that, meaning that you don't have to go to multiple tools to do the job that you need to be done right here. So very, very cool announcement here. It seems to me that eBPF is playing a rather strategic role in all of this, and maybe we don't give it enough credit, but if it was me, I would be sitting there saying, all right, if I upgrade to the latest versions of Lennox, I get eBPF.

And then I'd be turning around all, all my security vendors and for that matter, networking, storage and observability folks. And I'd be saying, you need to support this now because it makes such a huge difference. It, it improves performance, it reduces the overhead in terms of the amount of agent software that we're seeing.

So, um, is there some sort of groundswell around this starting to build or what's your thoughts here? Yeah, I mean, A BPF is definitely a, an approach for application communication that, that may, that streamlines the delivery process, it streamlined the application that can communicate better against each other. But what this release, the network observability really kind of really allowed for deeper host networking observability using EVPF, right?

So this is, this is, uh, an area that I think is really exp expanding and, and growing. Um, I do see vendors in the space, uh, like you just mentioned, like Cisco with the acquisition here. Uh, but I also see other vendors in the space using and, and, and taking cilium to that next level, right?

And taking that, uh, open source, uh, activities, uh, and, and using eBPF within their models because it is simplifying. I, I mean, I used to have this discussion when I was, uh, in a number of discussions with service mesh conversations and the, and there's, there was a, about an, a year ago, there was this argument in the analyst community of, well, is a service mesh really worth it? Right?

And is it really worth putting these things in place? And, and, you know, my opinion was always, yeah, there's, it's complex to kind of build it, but once you get it built, it's really worth it because now you have the ability to manage those applications, it communication appropriately. When you look at the introduction of eeb PF and you can have a sidecar list kind of approach of have that communication that opens up different doors.

And I think that vendors are seeing how this is working, and that in, and that integration, um, is really allowing for, um, you know, different exposure here. But I think, again, this is a maturity thing with regards to the application development that, uh, we're gonna see grow over time. All right?

I say to all those IT folks out there, take a good hard look at all that stuff that's running up in user space and try to figure out how much of it's gonna move into this new micro kernel architecture. Because the sooner you do that, better off, you're gonna be, I'm gonna shift a little gears away from security and talk about, um, there's a whole new approach to applying AI to the management of APIs from Postman that they were talking about. And it's interesting to me, we're seeing a proliferation of all these agents to manage various processes, but in, in the context of Cloud native, it's kind of important because every microservice has some sort of API hanging off of it and managing all these has become problematic.

So, um, do we need AI to manage the level of complexity that we're seeing in Cloud Native? Now, what's your thoughts here, Paul? Yeah, Mike, you know, I mean, APIs are, are, they drive the, they drive connectivity, right?

I mean, that's really what's we're seeing, right? We, I think the, the, the quote that we know noted earlier this week is, uh, you know, 71 million per, uh, uh, applications or something to that effect is running through APIs on the internet. So it's really just driving, uh, you know, uh, the, the, the, the activity level, um, through, through the API connection when we're, what this, this does actually have to play with security.

Um, when we look at API management, API management and, and what Postman is doing with the API tool set here and, and the, and the, and you know, and adding ap uh, uh, AI potentially into it, API management is testing the connectivity between these applications. Great. That works.

But it does tie to the earlier part of this conversation on the video we were talking about here, right? It ties to security because connectivity is only one part of the integration. The other part of the integration is making sure it's secure.

So, but let's talk about the integration piece and the, and making sure these APIs work. Well, an API management platform that's available, uh, allows for that. Making sure those connectivity points for those integration of applications within your ecosystem to work equally together, right?

And having a seamless way to do that, um, is, is only going to continue to grow. And, and what I mean by that is application development is happening so rapidly that, you know, it's the, I I think of it as the Stonehenge approach, right? If you put a bunch of hands under a rock, you can only put so many hands under the rock to push it up, right?

And it's, and you still won't be able to push it up 'cause it's just, it's a big rock, right? Think of Stonehenge, right? Um, I don't know how they did it, but however they did it, great.

The fact is, is you can't keep putting hands underneath applications to build them. You're going to need to have some type of automation to do that, some type of AI integration to do that. API management and API, um, uh, a, you know, would, would benefit from AI tools because as those, uh, API sets grow, and as those connectivity points grow, automation and workflow and AI to simplify the connectivity and making sure things are working properly will have to be the way to go, especially with the volume of applications that are being, that are being developed today.

Mm-Hmm. I can't help but wonder where API management will ultimately settle as I watch the emergence of platform engineering. And, um, it seems to me that the APIs and are core to that whole motion.

And so should that be part of a larger DevOps platform, or is it gonna always be this kind of standalone, uh, management framework? Well, API management, um, really does kind of attach, or it kind of goes across developer DevOps platform engineering, releasing the CI and the entire CI/CD pipeline. Um, once you, you know, when you're developing code and you have APIs in that code, you have to make sure they work right?

So the a the management of that connectivity is important, right? But once that code's released and pushed into the, into the delivery process, that also has to be continuously tracked, right? Because if there's changes in the ecosystem, if there's a part of the, uh, you know, a part of the, uh, the tech stack that changes, those APIs need to be, um, managed appropriately.

So it, it is, in my opinion, the API platform, uh, the management platform needs to go holistically across dev dev, DevOps, platform engineering, and across the entire CI/CD pipeline as well as the SDLC mm-Hmm. Yeah, it seems to me that the developers already have enough to worry about, but we wanna make it easier for them to create the APIs, but that doesn't necessarily mean that we wanna put the onus of managing all those APIs forever and ever on the backs of developers who may come and go over time. So it just seems to me there's a conversation to be had here.

Maybe it's tighter integration between the platforms, but, um, if, if I'm managing thousands of APIs, it just makes more sense to have somebody who sits on some sort of centralized function to do that. Yeah. Which comes back to the point, you know, developers are more than most likely going to have to de rely on AI tools in the future to offload some of those tedious tasks, right?

I think that if you, the more you grow into it, the more you have to do, you, you have to have automation to to, to manage it. Otherwise it's just going to be, oh, you know, daunting, overwhelming tasks that won't be able to get the, won't be able to meet those earlier comments that we made about delivering code faster. Um, and, and, you know, meeting those business KPIs.

All right. And then finally on the more esoteric side of things, there's an outfit called Upbound, and they're working with the CNCF to create, um, a thing called cross plane, which is a control plane and is essentially based on the same one used for Kubernetes, but it's extensible in the sense that it can run and manage environments that are not Kubernetes, which is great. Um, it seems like historically, at least if you look inside of a cloud service provider, they went in and built their own control claims and they did that by creating a layer of abstraction above all these APIs that the infrastructure manifests.

And that's how they kind of manage things at scale. Enterprises are cribbing this, and cross plane is a way to accelerate that strategy because you can use cross plane anywhere, and you can now centrally manage it through a single console. So you can have this kind of federated control plane architecture, which is, you know, good news.

And ultimately it should lower the cost of it because I don't need a dedicated IT team for every single management platform that comes out. If I can now centralize the management of that. Paul, when is your sense of, you know, how quickly are people thinking through the management of these hybrid environments?

Are they gonna, is it cost that drives that, or is it just, just I don't have enough people to throw at it and I need another way to think about it? Yeah, it's a great point. Uh, again, um, you know, this is one of those areas that, um, is incredibly interesting in the market right now.

Uh, uh, so Mike, I've been watching Upbound for quite some time, and control plane has, has been very attractive in my opinion because it, it does simplify a lot of things. It reduces complexity, uh, or in over at, uh, Upbound. Um, and I have had conversations and briefings about, you know, where cross plane is, is, is where it fits and, and honestly where it doesn't fit.

But, you know, in my opinion, I think it's a, it's a, it's a conversation about build versus buy, right? And, and if you're in the, um, discussion of if you're in the business of running your business, whatever that may be, right? If you're banking, finance, retail, whatever it is, I do, you have the technical bench to manage, manage and monitor everything that you need to do.

And if you do, that's great. That's awesome. That's a lot of resources, and then you have to keep maintaining it and, and grow it.

The thing I like about a upbound, uh, control plane, uh, approach is it, it, when I, when I talk about the story of modernization, as I, as you know, I cover application development modernization, um, I talk about in the context of past, present, future, and there's a lot of applications that are still in the heritage or past kind of state that are looking to modernize. And what this does, what Upbound does is allows for that bridging the gap between your heritage environment and your new modernized environment without having to use multiple different tools to do something, right? So, and that's really where your TCO kind of comes into play.

Your total cost of ownership. You don't have to have your DevOps teams go on from one tool to another tool to, you know, all these different areas. They can use Upbound control plane to have a holistic view across the different application stack.

So I think that's attractive. I also think that when you look at modernization, um, going, uh, there's a, uh, we know that 65% of respondents to our, our cloud native survey are running on four more clouds, right? And if you're running across four more clouds, and yes, you may have a control plane that's, you know, uh, GCP or AWS or Azure specific, but you have those individual tool tools and multiple panes of glass to look at.

I'm not claiming I never claim a single pane of glass 'cause there's no such thing, but I will say that what Upbound provides is a more holistic view across those different distributed cloud environments. Do you think that there is a slow, shall we say, underground movement and among enterprise IT leaders to take back control of these environments? I think they're getting beat up a little bit by the finance teams.

And part of their issue is they're all grew up in isolation with different decisions made at different times and different workloads and things are more distributed than ever. But, you know, do we need to, you know, have some sort of movement here? It's a good point.

Um, you know, when I look at the, the IT function and I look at how it's kind of ebbed and flowed over the last, you know, whatever, 20 years or so there was, you know, it at one point it owned everything, right? And then they, and then there was kind of, it owned nothing like, which was crazy, right? Right.

To a point. And it just became the wild west. Um, but like what I see it doing, and I didn't touch on the, the, the financial side of the control plan, but I will in in just a second, what I think it is doing right now.

There, there has been over the last say 10 plus years, a internalization of it, bringing it back into, um, into the, into the IT function, but more so as a business line manager or, or like a partner manager, right? Working with all these different organizations and these different tech stacks and different tool sets to manage what those relationships look like versus having to build it themselves and build stand up things themselves. So part of that business relationship is to have full visibility of the budget, have full vis visibility of forecasting what you're trying to spend on.

And without that, um, you know, I mean, uh, you know, DevOps and developers, they're, they just need the tools to do their jobs, right? They just need what, they need the resource and tools to do their jobs. If they, uh, are, if you leave it up to them, them directly, they'll over-provision every time because they'll get what they need and not worry about it 'cause it doesn't come outta their pocket.

But we see in our research that even though the tool sets are decided on by, say, DevOps and architects and developers, that tool set per uh, purchase may happen at that level, but the budget is the senior IT leadership's budget. So senior it is managing the budget. And, and you know, I think what the case that could be made across, uh, like the, you know, the Upbound case and, and, and control plane here is understanding a, a holistic view, reducing your TCO, reducing the number of tools you need, but also having a very clear understanding of what the spend is for what you, what you're, uh, what you're trying to build, not only for your current projects, but that also gives you the ability to forecast out.

I mean, how many times have we had these conversations when we're like looking at IT resources and go, we'll just do a 25% lift next year because that's what we're gonna need without, that's, you know, without any real data. These tools that are coming to market in the, in the finops space and you know, in the, in the management place, are really giving you the lens and the visibility to understand what the budget should look like more effectively. All right, folks, you heard it here.

The revenge of centralized, IT is on its way for better or worse, we'll see how it all plays out. Hey Paul, thanks for being on the show as always. Thanks for, enjoy having me on the show as well.

com. Happy to take it, happy to have any suggestions for future conversations. Of course, you can also find this on the Cloud Native Now website and we invite you to check that out.

Until then, we'll see you all next time.