GenAI workloads on Google Kubernetes Engine with Gari Singh at KubeCon Paris 2024

This is Textron tv. Hey everybody. Welcome back to Cobe Con.

We are on, uh, our last day here, and we're filled with more great interviews, just like the first two days, doing some great stuff. You know, I'm, I'm very privileged to have somebody from Google joining us. Um, Gary Singh, who is, what, what's the, what's your title for?

Uh, I'm just a product manager. Product manager. Go with that.

Okay. On on what? On, uh, Google Kubernetes Engine.

Okay. There we go. Yeah.

Thank you. Wanted to make sure we got that right. You Welcome.

Oh yeah. Get that in there. Yeah.

Welcome. Good to Thank for having me. So I Really want to hear the, kind of Google your perspective, uh, 'cause you know, how many people are coming from that, you know, era of helping create the technology, create, helping really build this ecosystem and fostering it.

What do you, what do you, what are you taking away from this year that's really kind of standing out? Yeah, I think, um, well, one, I, it's great. Like, obviously, I, I should plug that, uh, where, you know, closing in on the 10 year anniversary of, uh, of, uh, Kubernetes being donated to open source.

So that was fantastic. I think it's been nice. I mean, I've been to the last, like three or four of these, uh, you know, after the pandemic and just got a lot of people, um, more users here, I'd say.

Mm-Hmm. I mean, obviously ai, we can maybe talk about that a little later. It's obviously the big hype cycle, but I see a lot of people who are still just in the midst of, I don't wanna say getting started, but really starting to like, you know, really work with Kubernetes, right?

Maybe they've committed to, you know, move to the cloud. They're like, okay, now I've really gotta get my workloads out there. How do I use this?

What does this mean for my enterprise? Right? So I, I'd say, I mean, at least 50% of the people here probably are not in the, you know, not a product company or like, you know, on the open source side, they're just people who want to consume and use this stuff.

Yep, Yep. I agree. And, and even one of the things I've noticed is even the technology providers, so many of them have built now built in or built on Kubernetes as part of their offering.

So it's even becoming part of the substrate of so many products and services. Yeah. It reminds me of, uh, you know, a long time ago, and then we were talking earlier, it reminds me of like the Java world, right?

When there was a ton of people who built stuff on top of application servers, right? It became their sort of core framework, right? So I think you see more and more SAS offerings, uh, you know, database offerings, like all these types of stuff built on, built literally on top of Kubernetes.

And obviously you have the massive ecosystem of people building like side projects, not side, I mean, you know, things that work off of the side or compliment Kubernetes, Right. Complimentary too. Yeah.

But the number of things that are on Kubernetes is, uh, is quite impressive, which to me means that, you know, for the foreseeable future, Kubernetes probably can't go away. Alright. We were talking earlier in this segment.

Um, it, it's almost feels like, so someone described it as, uh, Kubernetes is kind of having its Linux moment where it's becoming pervasive enough, almost so pervasive that it's just gonna be part of the substrate of so many things that we use. You know, Linux isn't everywhere, but it is everywhere. You know, kind of Kubernetes is starting to get that kind of momentum.

Oh, yeah, most definitely. I think, uh, I mean, we've seen, uh, containers, you know, have become like, you know, so popular for the last whatever, you know, 10 sort of years. But, but yeah, I mean, I think, um, it's interesting to hear like, you know, why not?

Maybe, maybe the thing is like, why not Kubernetes more than why Kubernetes? Mm-Hmm. Um, which, you know, is, you know, you know, pushing the sort of boundaries of that, that usage everywhere, right?

I mean, and everybody's got an offering. Um, the ai, you know, I think, uh, I mean, maybe everybody obviously wants to use the buzzword of ai, but I think you start to see, look, it makes a lot of sense, right? It's all about compute, orchestration, running your stuff, packaging everything up.

Why go to another paradigm, right? Mm-Hmm. Um, and it's great to see like the, I like to say the why not Kubernetes, like, we're already doing this type of stuff, and like, let's not criticized, uh, you know, I don't think people are looking at the next thing right now.

They're just looking at how does Kubernetes expand into these other workloads, right? Um, I think things like the data on Kubernetes community Mm-hmm. Right?

And I think like five years ago, people would've said, don't run Yeah. Databases on there. They don't.

Yeah. Don't put it in, don't put it in a container. Yeah.

Don't put it in a container. Exactly. Yeah, exactly.

You're right. Back then, put it on a vm, don't put it in a container. And now you look at every major, look at how many of the vendors here are all like, you know, database vendors or data vendors, or data processing vendors, which is impressive.

Mm-Hmm. You know, it, it's interesting, you know, I have, I have a bit more, quite a bit more gray hair than you do, but we've, we've gone through a couple generations of these cycles where new things kind of come and usher in and become pretty pervasive. We were talking about Java before.

We're talking about a lot of different things before that. Um, what do you think are some of the lessons we've learned in those successive generations about adoption, solving problems, you know, hardware for now for ai, right? GPUs is a challenge.

Throughput, scaling, Kubernetes, those are all challenges we've had in successive technologies, right? Right, right. What have we learned in those, those cycles that you think we can build upon now?

Yeah, I think, I, I think, you know, I think one of the interesting things is, you know, if you look back on all this stuff first, everything sort of becomes like this, like technology play, right? You always try to appeal to technologists and kind of make it the cool or the hip thing to do, right? Mm-Hmm.

But in a way that, that it generates some level of adoption, but it doesn't get the, like, um, you don't get like necessarily big enterprises or whoever are gonna do it. And these are the folks who are gonna, you know, when once a, it's adopted by like a tra more traditional company, as we know, they invest in these technologies for the next 10 50 SoCo a bank or insurance company, or, yeah, yeah, Absolutely. They pick this thing up, it's gonna be their platform foundation for the next, like, sort of 10 to 15 years.

Yeah. So I think the one thing is to learn like what is the, you know, business value, what are the problems and real use cases, you can do that and also learn that it's a means to the end, right? So don't, don't focus so much on the means, but focus on the end.

Focus on the end game, right? I think that's interesting in the AI space, right? Because it should not be so much about, yes, there's low level plumbing details that we have to work out to solve people's problems, but what we'll really have to explain is, well, why should, why, why should I go do this?

What is it that's, you know, great being ai, right? The simple fact that there's 9,000 frameworks every day, right? Different versions of Python, sound familiar, different versions of Java.

How do I package this stuff together? Different teams picking something new of the week, right? How do I standardize on a core platform that's just gonna enable all of those things, right?

So it's the means, but not the, you know, not the end necessarily. It, it's interesting, um, you know, Kubernetes isn't a, uh, a feature, if you will, right? It's, it's part of the substrate of what we're able to do things with.

Yeah. Right? The kind of workflow moving workforce flows to the cloud, handling challenging workflows.

I mean, that's, I think one of the strengths is being able to do more than just the normal stuff. Like, how do I distribute this? How do I handle varying changes in load?

Maybe drastic changes in load or geographic, all that kind of thing is, those are problems we've had for, you know, forever now, even more so because it's just commonplace to do that kind of thing. Yeah, I mean, exactly. Right.

I mean, I think you see whether, you see companies that have to have, you know, disaster recovery, high availability across, you know, multiple regions. Uh, you know, there's the interesting parts of the cloud, right? Where people, you know, you, how do you not treat the cloud like you treated a data center, right?

So how do I make it easy to deploy in like, you know, the US east region and like the US whatever, the central region, right? And then what do I do if I want to have load, you know, in, in, in emea, right? And how do I have my replicated stuff?

And I think if you look at, you could rebuild all your compute infrastructure over and over, but to me, Kubernetes, we always talk about it as like container orchestration. To me it's actually compute orchestration, right? And I think that's what we're also seeing, like with the GPUs and everything like that, right?

You have to have load balancers. You want to fail over across zones. Sure, you can map all this out in the cloud or wherever it is yourself, but in the end, you just take Kubernetes, you kind of install it, it's the plate in zones, it's gonna restart your containers for you, right?

If it's down in one zone, it's, you know, you're connected to another zone, your network overlay is already there. Like, all these things are just kind of there Mm-Hmm. Which allow apps to communicate with each other.

They have to transparently move them around. And I think it's almost like, uh, we forgot, I think we focus so much on containers that we actually forgot that it actually, like orchestrates, like you said, it substrate, but it orchestrates that whole layer and it simplifies it fundamentally, because you could do it all yourself, but Why? But why, yeah.

That's, that's a huge problem to go solve again, right? Yeah. Yeah.

How about platform engineering? You know, that's obviously a big topic and many organizations are adopting it sometimes from a developer productivity standpoint. You talk about standardizing your environments and kind of making things more, uh, manageable.

Uh, how does Kubernetes fit into a platform engineering kind of group's role? And what, what should they, if they aren't, what kind of things should they be doing with Kubernetes? Yeah, that's a great question.

I think, you know, once you, once you get, once you get above thinking about managing the infrastructure, right? I think yes, there is some infrastructure with Kubernetes, but as we just said, it covers a lot of the gamut of what you need to do. I think, especially as people move to like, you know, the cloud especially, right?

But so now how do I look at what can I do specific to like my like environment? What are some frameworks that I might need to do? What do my developers need for like, more productivity, right?

Invest in those tools rather than like, oh, how do I, you know, what's my container build system or what's my, what's my communication protocol between things, right? We've, we've handled all that in sort of Kubernetes. Um, and then the fact that the ecosystem is so big, right?

Whether you wanted to have an overlay, you know, a, a service mesh, whether you need, you know, CICD, um, I think you see people going the other route with like backstage, right? Building this sort of self-serve platform, right? Right.

So to me, it serves as a, you know, great foundation for that because it's abstracting a lot of the compute services, right? For you. And now you can just build whatever you want on top of that, but focused on the value for your business, as we were saying, right?

Not the value of the actual, like plumbing itself. You know, one of the challenges, maybe even NOx I guess you could say with Kubernetes been is the complexity. You know, there is a, you know, there is a place to get to, to be really proficient at it.

Um, is, do you see that as getting better or what do we need to do to kind of help close that gap for folks that are adopting it? Yeah, I think that's a great question. I think, I think there's still some stuff left over from the, I think some of this is like left over from the earlier days of Kubernetes.

Like, uh, I was talking to somebody the other day. I said, imagine what would happen if we had had mini cube on day one of Kubernetes, right? Because what are the confusing parts about Kubernetes networking, you know, people, and you shouldn't have to like, worry about that part, right?

But, you know, the more majority of people don't know networking, right? As we were saying, right? They don't.

Right. So that gets confusing. So then you start to focus so much on getting it up and running that you forget about like, the power of the deployments and the power of, of actually the higher level resources.

So I think now, um, I mean even whether it be a managed distribution, obviously all the stuff in the cloud, you know, obviously we have, you know, where I work, where I come from, we have that, but Yep. But I think we've tried to take away that toil mm-Hmm. And now sort of move that, move that up, right?

Such that, uh, it should be fairly easy to sort of deploy. But it'd be interesting to come back and do sort of a paradigm. I'm like, what are the five things that like you should think about now instead of like, these things that I think have been keeping with Kubernetes for the last 10 years?

I think it has gotten a lot easier and a lot simpler to do. I think some people just hear that, or they want to keep repeating that message. Um, and, and, and maybe that's, maybe that's what we should focus on, like, you know, the rest of this year Is just, it's not as long.

Maybe it's more about picking your starting point. Yeah. You know, as you know, we don't, we no longer go back and compile, you know, build Yeah.

Kubernetes, right? That's what we all did in the beginning. Correct.

Let me build it. Let me, then a lot of us got to the, oh my gosh, I don't think I really want to, right? Keep doing this.

Let's see, what can I do? Right? Who's got a distribution I can use?

Yep. Now where's a managed, uh, service that I can use? And so to the things like security and networking and let's face it, distributed workloads fell over load balancing all that.

That's, that's complicated s**t. I mean, that's not, you know, not every programmer's Correct, correct. You know, thinking about all the, or proficient at every one of those issues.

So build on where there's already expertise or that's been solved or as a good service to do that. So don't, you know, we don't go back to try to build Kubernetes from scratch. So also don't start from networking from Scratch.

Don't start from networking from scratch. You know, look at the sort of paradigms that are in there, but you do bring up a, you know, I think the one interesting point about, I think that on Kubernetes too, is maybe looking at how do we explain the different, I, I think there has to be this sort of, um, translation. I mean, we were talking about stuff that we've done in the past, and I call, and you can call it reference learning, right?

Mm-Hmm. And I think when you don't, if you don't give people like an existing point of reference, like what's the analogy? If you're coming from a VM world, like what is the analogy in like, you know, Kubernetes, right?

If you're coming from the network world, right? People wanted to have explicit firewall rules, IP chains, IP tables. You're like, understand, oh, network policy, right?

Like, I think we don't do a, we need to do a better job of mapping people to the analogs, right. For their specific role. Like what, what does that mean in like Kubernetes, right?

And instead of making it seem like it's all new, it's just a different way of configuring the same thing. Well, it's like message bus. We've been doing MQ series and Tuxedo.

Yeah. And you know, cor, was it corba? Yeah, corba.

corba. Yeah. We, we've been doing those in successes generations since with different architectures, right?

Very large scale monoliths. In some cases now we do it with microservices. So in some ways it's easier and it's more kind of the fundamentals of how communications talk about instead of opening sockets and weird things, we had to do that.

Now we just, you know, call an API and you know, do a restful or, or graph ll kind of call. So there, there's analogs to what we've done. Yeah.

And how to now how do we leverage that in this newer kind of paradigm, uh, where we've got a lot more flexibility, maybe workloads are smaller, tasks are smaller, and it's kind of interesting. 'cause to me it's one of those that gives us a lot of capability to deliver sort, uh, code faster, functionality faster. It also means a human can't manage it all in their head.

Right? Right. You can't just lift it over to the operations team.

Exactly. Or throw it. Yeah.

And, you know, over the wall, you have to have that orchestration, you have to have that substrate and management capability, scalability, all of that. Exactly. Exactly.

Yeah. The, um, yeah, you don't wanna just like sort of throw it over the wall and Yeah. I, I think if you look at, and maybe, you know, maybe sometimes people have said like, you know, what is the role of the developer?

You know, what do they have to know about sort of like, you know, Kubernetes and then what's the value? You know, I some, I think sometimes we just, again, we don't tell them explaining like the what, the value of like, what pieces should you care about, right? Mm-Hmm.

And then that's where like the platform engineering can come in. You know, I think if people want to build abstractions, I always say be careful not to like abstract everything. Like, how many layers do you really need on top of this?

Um, is, it's not really that hard to write a basic like yaml Mm-Hmm. Right. And I think hope not.

Yeah. No, no. And I think, you know, we see operators out there now for the databases and things like that, right?

The, the experts are kind of doing this stuff for you pick that stuff up rather than, you know, do it. And then I think the, what's the express? My, I made up an expression, I think, you know, 70 30 is the new 80 20, right?

I think so many people always want to sort of tweak, tweak, tweak. I always think about what's gonna get me there faster, right? If I just do, I really need to do this extra 10% of the things that maybe it doesn't do today.

Yeah. Am I willing to give those up for the, you know, 90%, you know, in productivity boost I get, Yeah. You're really going to get the big, a big enough gain to make it worth getting from 80 to 90 or beyond.

Exactly. Exactly. Yeah.

I think, I think also is I have this theory of we have a generation of technologists, developers, et cetera. We're coming up in the world of I can get it off of YouTube, I can watch this, I can download that, I can build it this way as opposed to I have to construct it myself, which is certainly more what we had to do in a lot of my generation. And so hopefully that promotes that leverage build, build upon.

I don't have to do it all. Yeah. Know it all.

Right? Yeah. I mean it's, I mean, it's amazing that's out there, right?

I mean, you can, I mean, who knows what like this uh, whole LLM, you know, thing will do for people, right? When it gives you the answer and sort of code generation, I think you have to at least know what we have to be careful of is to make sure we instill like the right concepts, right? Mm-Hmm.

We won't make this an AI conversation, but as you, if you listen to anything out there, LLMs can't just do stuff for you. Like if you don't ask them the right, you have to know to ask them something. Right?

So what is it I want to ask you to do? Maybe build a distributed app? What are the key sort of terms that people need to know?

So I think it's how do we make sure that people understand those, like what the tool's capable of, so they know how to sort of ask the right question. Understand it enough to know how to use it. Yeah, yeah.

Or know enough about how it works. It's like the security people, they don't need to know how to write code and Exactly. Build API gateways, but, you know, use the networking concepts that a restful call makes.

Oh, I recognize that. I see how that works. Okay, great.

Now I, I have enough knowledge to be able to talk to the developers about how we're securing our APIs. Right? Exactly.

Right. Understanding that within with, you know, how the, within a, within a cluster, right? That we have just an overlay network that's not publicly exposed, right?

For service to service communication. Oh, we have a closed boundary. It's not open to people outside, right?

So I think, you know, you just have to figure out, you know, that people can understand some of those sort of core, basic kind of tenants, right? Um, and it's not overly complicated. I think once you break it down to the, the pieces, I think it's like we, we just, to your point, I think a lot of people started from here's how I built it.

Here's all the stuff that was in here, here was all the sort of corps. And I think, I think that's like maybe slowed things down. I think now we've moved, I think we're starting to seriously move past that though now, right?

More and more, like I said, more and more people here are using Kubernetes, right? Yep. Not asking and asking how to use it for real use cases, more so than, Hey, I need, you know, x, y, Z field in my like deployment.

Or what does this mean, right? We're not arguing about like silly things, right? It's like, alright, I got this massive AI workload or I have this massive thing.

How do I get the right, you know, GPUs? How do I, what's the right metric to use your autoscaler for? Right?

How do I get notes spinning up faster so they're asking the right questions? Kind of a bad analogy, but I don't need to know how a combustion work engine works in my car. But I do want to think about when's the hybrid gonna be really good for me?

When might I go ev when stay combustion, you know, to run my farm on or something. But Exactly. Hey Gary, it's been fascinating talking with you too.

And uh, thanks for your good work at and you and the team at Google. Yeah, Yeah. Thanks for Having us and continue to move the ball downfield and yeah, we'll, we'll do what we can look forward to talking to you at the next, uh, Kon event.

Definitely. Salt Lake City. Okay.

Salt Lake City. There you go. See, fantastic guest.

Yeah. I tell you the truth here on Techstrong TV at Kon Paris. And we're fan, we're really privileged to have Gary join us today.

We've got more guests coming up, so stay tuned. We'll be right back.