Navigating Linux Container Challenges with Red Hat’s Scott McCarty

This is Textron tv. Hey guys, thanks with Throw, we're here with Scott McCarty, who's a principal product manager for R Server at Red Hat. And we're talking about what to expect from Linux containers going into the new year because, well, there's a lot of conversation about this as of late.

Scott, welcome to the show. Hey, thanks for having me, Mike. Where are we in terms of adoption and containers?

I feel like it's now kind of the defacto way we build applications and deploy them, but, uh, maybe not everybody's quite there yet, but as we go into the new year, where are we with the usage of containers and what should we be expect in the new year? Yeah, I think, uh, I am a long time container guy. I started literally 2013.

I think. Uh, it is kind of hard to believe it's been almost 11 years. Uh, time flies by.

So, you know, it's interesting. I think containers went through the standard hype cycle and you know, but I don't think they've hit the trough of despair, right? Like I still think we're finding new use cases for them.

So when you look, you know, you look at tools like a, you know, all the AI stuff that's exploded obviously, and you look at Ramal LAMA and uh, and tools like that, and we're starting to see people deploy models and things in containers. You look at things like, uh, wasm, there's still a lot of, there's still a lot of ground to cover with containers, but I, I'd say we're, we're seeing mass adoption now, but we're also still seeing breakthroughs happen. So I think that's, that's it in a high level.

There are multiple types of containers now, and there seem to be big ones and small ones. So are we getting used to kinda, uh, orchestrating these containers in ways that, uh, where the sum of the parts may be greater or the, where the hole might actually be more than the sum of the parts? Yeah, I think, I think there's always been a discomfort with large, you know, monolithic containers, and I think that's still the preference is to have smaller, smaller individual containers that are orchestrated by something like Kubernetes, um, such that each one could be killed individually, easily and restarted, et cetera, et cetera.

But, uh, but I do think, yeah, I think we're getting more comfortable with the primitives that containers offer. So the storage aspect, you know, the fact that container images and those OCI images can be used for all kinds of different things for cloud providers, for, for AI models, for regular applications, for all kinds of things. And then on the runtime side, yeah, I think, I think we're getting comfortable with, with putting more and more applications in containers, which is something we kind of expected, right?

Like we expected would, once people saw the benefits, they'd wanna put everything in a container, right? And some of those things are large, some are small, some are big monolithic applications, some are microservices applications broken down into a bunch of tiny containers, you know, I think we're seeing everything under the sun, basically. So what are you looking forward to in 2025 then?

What should people be on the lookout for as it pertains to containers? Yeah, I think for me there's probably three sort of groundbreaking areas still, if you will, where you've seen containers break new ground, uh, I didn't mean that as a buzzword, but more just actually breaking new ground. But, um, you know, I, I talk about, first I'd say, uh, if you look at bootable containers, so things like Bootsy, uh, this is the idea that you take a container image, you use it as the primitive for deploying an operating system, and then you lay it down onto the, you know, onto disc of a virtual machine or a bare metal machine, and then you boot it up as an irregular operating system, and then you deliver the updates to that os also as a container image, as a layer in the container image.

Um, I think that's pretty exciting because that brings the value that we'd found with collaboration with CICD, with testing, with, you know, building everything at the factory and then deploying it, you know, at runtime at 2:00 AM it takes two seconds to deploy an update. Uh, it brings a lot of value that we had in containers to the actual infrastructure, the actual, you know, historically we deployed applications as containers, but we still had to deploy the infrastructure the way we've always done it, albeit with more automation and things. But, uh, now we're, we're seeing with Bootsy and, uh, image mode for r we're seeing, uh, you know, the deployment of the OS infrastructure itself.

So say you have a thousand servers, you can now deploy those all from a single container image, for example, and do all the updates to those with the container image. I think that's one super exciting thing that I see for 2025. Um, you're seeing it also in projects like bluefin and things like that.

They're adopting these methodologies, so it's really cool to see. Another one is probably the second most exci, or the next exciting one I'd say is ramal lama. So what you're, we're seeing a tool that will build, uh, essentially bring a lot of the magic of containers.

So if you wanna deploy an AI model today, it's pretty tough. You go to hugging face, you read a page, there's very technical documentation. Um, when you look at what Ramal LAMA and tools like O Lama are doing, they're making, they're sort of bringing the same thing that Docker did back in 2013 where it's much easier single command, you know, you type ramal lama run, you pull down a model, so Alama run grant three B pulls down a model and runs it, and you're like, okay, that's pretty nice.

Um, and now I can test locally, I can literally fire it up in a terminal, say, Hey, uh, tell me a story about dwarves. And next thing you know, it tells you a magical story about dwarves. So you can start to test out a bunch of models, but later ramala is interesting and does is it actually runs all of those in containers, and in fact, it'll look at the local hardware and pull the right version of the model optimized for the hardware that you have.

So it's doing some trickier things with that container infrastructure, but, um, it's basically taking a model, running it on top of a container and smashing all the software together that you need to run the GPUs that you have, for example. So that's pretty cool. Um, and I still think there's a lot of excitement as the third one, I'd say around the cross platform, you know, potential for Wasm, if you can compile a single binary that's wasm based, have the runtime either in the container or the way pod man does it, it'll be installed on the operating system along with pod man side by side.

You pull down a was and binary and it'll run it on the local runtime, that, that has some interesting potential use cases long term with edge devices and automotive and all kinds of other weird hardware that people are trying to run. Um, so, so we, I, I think those three to me still have a lot of life left in, in them, you know, of where they're going in 20, 25 and beyond. There are still folks after all this time who are still critical of the approach of containers.

So what's your sense of what's going on in the Linux community, especially as it relates to containers? Is there kinda, you know, there're some tension, but it's friendly tension, or how does that play out? Yeah, I have, I have been in this, in this container space for a long time.

I've heard everything from over my dead body to, you know, you know, we're, I literally had a guy coming to me at a conference one time. He goes, we will never adopt Kubernetes ever. You know, like, I literally just started yelling at me and I was like, okay, all right, that's fine.

You don't have to. I, you know, uh, so yes, you, you have some of that, but I would say generally it's friendly. I would say that that containers are not a, i, I mean it's, I, it takes some work and brain power to put applications in containers.

I don't want to, you know, betray that. Um, and I'll admit I was even lazy with like my blog and my wiki and my WordPress instances. It took me years to get them in containers, although I did now get them in containers several years ago.

But I've been doing this for 10, 11 years now, and it took me probably five or six to get 'em in containers. So I guess I'd say we're still in that state where yes, there's some stragglers, yes, it can be difficult to put certain applications that were designed 20 years ago into a container. Um, but we're seeing progress and I think people are starting to understand where it's good, where it's difficult, where maybe I'll never be able to get this in a container.

It'd be very difficult, or maybe it's too much, too much work to try and we'll just kind of let this thing live off into the sunset, not in a container. I mean, we're, I think people are starting to figure all that out. Not everybody who builds application in containers is also using Kubernetes.

And some folks say, you know, Kubernetes is still too hard, and there's criticism that says something to the effect that, well, maybe we have 10 million developers who can build something on Kubernetes, but there's 50 million developers out there. So what's it gonna take to make the platform, uh, more accessible to the broader developer community? Yeah, that's a good question.

It's interesting. The, the, the spark I see is there's always, there's always startups and, you know, newcomers trying to find ways to enter a market and essentially disrupt it. I still thi personally, I still think Kubernetes has a ton of life left in it.

I mean, it's still wildly popular and it's so well tested. And I'd argue it's actually probably the best example of an open source project ever. You know, when you look at how many different disparate organizations and individuals have contributed.

So like, I would find it very difficult to find a project with more velocity and more diversity, more, more, uh, you know, differing opinions on how to get things done. And I have, and I, I have seen it get a lot easier over time. I think, uh, I think some of it is, is digestion of like, how hard is it now for, versus five years ago when I tried, you know, there's some of that life cycle that has to happen where people have to absorb how much more stable and easy it's gotten to, to deploy things on Kubernetes.

Is there some irony in the fact that I feel like we've come full circle initially there are a bunch of full stack developers telling it ops people, we gotta have Kubernetes. And they said no. And then we had a bunch of, now we've come full circle and the it ops people are telling the development community, we need Kubernetes, and you must write to it.

And by the way, hello, I'm your new platform engineering team. Yep. I, I agree on that wholeheartedly.

That is what's happened. I mean, that's the, that is the technology adoption curve, right? Like once it becomes the standard, then it becomes a little bit painful sometimes, right?

You know, you're like, oh, now I have to do this, you know, and everybody has to do it. Yeah. I, I do agree on that.

That's part of the growing pains. I think, though I think Linux was the same way, right? I think we went through that Linux.

I think maybe the next big challenge is we're trying to figure out how to manage fleets of clusters. And in fact, some people are arguing about, do I need fleets of clusters or do I need just one big cluster that I parse out to a bunch of different projects? And I guess that was the original vision, but yeah, how are people deciding which way to go?

I have seen this as an age old debate. I have seen this debate for probably seven years, maybe I, a long time even probably a couple years into the Kubernetes, so probably 2016 I've probably heard this debate. I've seen everything.

I've seen giant mega monolithic Kubernetes clusters that do everything I've seen. Let's spin up a thousand Kubernetes Kubernetes clusters for each application. And I've even seen people manage thousands of Kubernetes clusters with a central Kubernetes cluster.

So like a Kubernetes cluster that manages the other ones, it does all the orchestrating of the updates, et cetera. There. The short answer is I don't, I don't think there's a clear path there.

I think, I think people end up with like a production cluster for sure. For, for a central, you know, cloud or IT operations, you'll see some of that where there's like, you know, something in the DMZ type, you know, you know, uh, set up that's kind of tradit reminds us of the traditional DMZ world where this is where we run everything that's web facing. Um, you know, but then we'll see smaller ones could nearer to the edge.

So I think, I think use case is partially what drives it, and I think everyone has to get comfortable with what makes sense for them. But there's this, I forget what that principle is in math, but like, you know, there's a knife's edge, right? Like where too many, and that's not good and too little, and that's not good, you know, and there's sort of a happy medium, um, for most users and most organizations.

And while we're putting other longstanding debates to bed, um, staple applications, when it initially started, everybody was like, thou shalt not put stateful applications on Kubernetes. It's, uh, meant to be stateless. And now I look around and there were databases everywhere.

So what happened? Yeah, well, I was a contrarian early on in the Kubernetes world, and I knew immediately that we would've stateful applications because, because the world needs stateful applications, you know? And so I, I was a preacher of, of the fact that I, I'd seen enough technology adoption lifecycle curves that I was like, every big platform starts with whatever it's good at, and then it expands off and gobbles up everything essentially.

Linux did the same thing. We started with DNS and Web and a few things, and now Linux is everywhere, right? So yeah, I I think finally the world, yeah, the, well, the world caught up to the reality that, that yes, we're gonna run staple applications 'cause data is stateful and, and even, you know, these AI models, if you're gonna run 'em, they're gonna be, you know, they may be stateful, we don't, you know, uh, I mean technically those are not stateful, but like, they need to be in place for long periods of time probably.

Um, and applications are made up of stateful and stateless components essentially, you know, and so you're gonna of course run all that in Kubernetes, right? The recurring theme has always been, you know, Kubernetes is heart. And I look around and I see all these AI tools now that are making things simpler, and I can get explanations, and it's a natural language interface.

So are we gonna finally maybe put some abstractions in front of Kubernetes where it just becomes simple enough that, uh, the whole backend is headless, maybe, and I'm just using natural language to manage it, and I don't have to be this kinda yamo rocket scientist. Yeah. In our experience, people are still, you know, AI is in the early phases of the adoption cycle, and there's still a lot of fear around maybe going headless.

Um, you know, that's probably speaking into the future a bit. I guess I've been waiting for driverless cars since 2016 or whatever, but, uh, but uh, I will say yes. So if you look at OpenShift, for example, it released a capability or feature called, uh, OpenShift Lightspeed, where you could essentially pull a chat bot to help you with what's going on inside.

So if you're in the OpenShift interface, the graphical interface, you can click on something and say, Hey, help me with this. You know, you could imagine like tool tips for example, that are dynamic and, Hey, help me, what, what's, what's wrong with this log? I don't understand it.

You know, those kinds of interactions that are definitely useful. In fact, coincidentally we're, it's funny the OpenShift and, and Kubernetes maybe is getting that before rl, but, but there's also RL lights speed we announced at Summit last year, and we're actually working on that, uh, uh, a bunch of use cases around that for R 10. So like, uh, also exciting stuff, I guess I'd say.

So yeah, I'd say operations in general, I'd back up and say I think is being enabled by AI in a lot of ways. And, and I'll argue that the reason why operations is such a good place for that is because operations people live in a constant state of not being sure anyway. Like, I joke, when you wake up at 2:00 AM you're like, is this page real?

That's the first thing. Are you sure about? Then you get, oh yeah, this is real, but wait, is this really broken in this way?

All right. And so like, then you finally get it back running again and you're like, I think it's fixed. And then you go back to sleep, but the entire cycle of your 2:00 AM to 4:00 AM was, you know, you were living in a constant state of not being sure anyway.

So I think AI tools actually help there. If they can increase your confidence a little bit, say 15% or 10%, that's a win, right? But you're not gonna trust it as I, I don't think we're there where we trust it as headless, right?

We're not gonna have headless cis admins just managing the cluster anytime soon, Nor are you gonna trust if it tells you that everything is fine and you go back to sleep, only you wake up in the morning and discover that all hell broke loose. Right? Exactly.

Yeah, exactly. So one other debate that's been going around for decades now is this whole conversation about portability. And one of the reasons to, you know, have containers was I can move them from platform to platform, but I don't think many people paid much attention to that or actually did that.

Do you think going into the coming year though, it seems like with some of the arguments that are going on around licensing terms and repatriation is, is there a new fat appreciation for portability enabled by containers? Yeah, that's, that's been an age old debate since containers came of what does portability actually mean? I've written extensively on this more than I can even probably 20 blog entries on this kind of thing.

But, uh, essentially, does portability mean that you can just move, build an application once and run it anywhere in the world that you want? Like does it mean that, or does it mean if I set up similar infrastructure, can I do very easy Dr, essentially, right? Like, like where I, where we know from dr like the environment is the same on both sides.

I just move the app over and it works. I would argue containers clearly make Dr extremely easy, right? Like whether it's local resiliency in a Kubernetes cluster or whether it's failing over to another environment that you've prepared, I think there's very little debate that that will work in almost all scenarios.

It's pretty much bulletproof. Um, I think that that still exists today. I think the debate, and, and, and if you think about it, that's the hybrid cloud story in, in effect, is it, can I go build a, a similar environment in Azure or AWS or IBM cloud or whatever, and can I have a local data center and kind of move things back and forth at will?

If the environments are pretty similar, I would argue yes, absolutely you can do that. The question is, can I move from Azure, you know, to Google, you know, to Google or to AWS without regard for any changes or differences between those infrastructure with the same container image, eh, with some applications, yes. Uh, with other ones less.

So I, I think that's where the gradient and you need to hire smart people to figure all that out, you know? Uh, that, that gets a little hairier, you know, in a nutshell, that's always been my philosophy on this. And What is the one thing that we're not paying enough attention to?

What's the conversation of the moment that, you know, we're all distracted by these shiny objects over here, but what's this one thing that maybe we should take a minute and go, you know, this is gonna be a bigger problem than we think? Oh, that's a good question. Uh, I, I've been thinking actually mostly in the AI space, if you really, if I'm being honest about where I've been looking for problems, but, uh, in the container space, you know, I would say the one stickler problem that I've still been seeing nibbled is people try to reinvent the wheel.

You know, I, I'd go back to my, my argument about Kubernetes, Kubernetes is, is, is, you know, Linux was the quintessential open source project, right? That kind of took off and lit on fire, and I'd argue Kubernetes has even surpassed Linux. And in its in, its just how many people participate and how many use cases it's covering, and how mu you know, it's the, it's the, it's an, oh, I would argue it's a, it's an example of humanity working on a problem together, like at a level that just has not been explored before yet I still see people coming in and going, oh, isn't there this other tool we could use for this one edge case?

And I'm like, this is just like stateful applications, except for on the other side, we're arguing, you know, think about the Kubernetes and Linux operating system as sort of pieces of infrastructure in the middle, and then there's all the hardware and clouds below, and there's all the applications above, right? We were arguing before, can we bring stateful applications? And my argument was, of course, we're gonna bring stateful applications, we're gonna bring all the applications on the bottom side.

I go, I go, does this work for this one little use case where I need to schedule this thing in this weird way? Absolutely. You know, you're gonna see Kubernetes oz into basically every possible use case that is possible to manage, in my humble opinion.

Yet, I still see people trying to introduce new, you know, this is like BSD. People say, someday BSD is gonna take over Linux. You know, and you're like, is it really at this point?

Like, Linux is obviously one, Kubernetes is obviously one, yet we constantly, every couple years or year we reintroduce, is there this new OS that could win a against Linux? Is there this new unikernels are a perfect example of this, not that I wanna rant, but like, we thought Unikernels were gonna take over the world. I knew they weren't.

And then if you look at the Linux kernel, there's actually a way now based on some research that happened at bought bu Boston University with some red hatters where you can boot Linux into a unikernel mode and funding almost immediately dried up to kernels. I stopped hearing about Unikernels and everyone realized, oh man, that's probably a dead end. I think we're seeing the same thing at Kubernetes where we see these other schedulers and other things that are like, oh, we could do this other stuff that Kubernetes can't do.

I'm like, good luck maintaining that differentiation over any, any, you know, sufficiently long period of time, like the next three to five years, you know, it'll be very tough to unseat Kubernetes, yet I still see us trying, I guess. So that's, that's one of the, the pitfalls, I guess I see coming in the container space. Okay, folks, well, you're heard it here.

The good news is containers are everywhere. The challenge is containers are everywhere. Hey, Scott, thanks for being on the show.

Hey, thanks for having me. All right. And back to you guys, and seeing you.