Alan Shimel & Tim Hockin – Fireside Chat: Original Google Kubernetes Team Member Discussing Frontiers in Kubernetes

Hey everyone. Welcome. To Cloud native event here from techstream.

We appreciate you joining us today, you know, it's become a bit of a tradition for our Cloud native event that I do a fireside chat with our friend Tim Hawkins. Tim for those who don't know so a principal software engineering Google and was actually on the original founding team for kubernetes the folks who really, you know work as we transition from Borg to kubernetes. and and all that it's become Tim and his cohorts on the team were the kind of the The people doing it so Tim welcome back to our Cloud native event a pleasure to have you on again.

Great to be here again. Thank you. Tim two areas.

We wanted to cover today. I've been to it. I I wanted to maybe just give you a chance to say hello to the audience and if there was any kind of big message or met, you know anything you wanted right off the bat for them to take, you know notice of Sure.

Yeah, it's great to be here again. You know, I love seeing these things go on. This is the fourth or fifth year.

I think so. You know kubernetes. If you'd ask me eight years ago, what's this thing going to look like in 2022?

I would never never in a million years of predicted this the number of events. The number of people engaged the number of companies and you know people developers that I get to work with on a daily basis. It's it's phenomenal.

So thanks for being part of it. Thank you, really? Look, we should be thanking you.

but more I mean it really has be. Come the centerpiece of the new compute stack, right? I mean, it's just phenomenal would who would have thought but you know what Tim I before we jump in just one other quick word.

I used to like it a lot better when we could get to come out to Google offices in the valley and do this in person. I'm hoping one day we can again next year. Let's hope it's it's kind of like my seders every year for Passover when they say next year in Jerusalem next year at Google.

Well, we'll try to do that. um anyway let us jump in now though and talk about Two of the things, you know, the first thing we wanted to talk about today is this notion of cloud native? Of course Cloud native look it's it's the name of of our virtual event is the cloud native con.

Cloud native Computing Foundation of course manages kubernetes kubernetes project in some ways kubernetes has become synonymous with Cloud native. there are others that tell you the cloud native is more than kubernetes, and I'm not really here even argue that but This whole notion of cloud native. That's It you know, I don't want to get all Beach Boys Beatles metaphysical sixties on us, but it's a state of mind as much as it is an actual place in some respects.

I want to kind of your thoughts on it Tim and if you want to expand Yeah, you know this is something I think a lot about as we are part of the cloud native compute Foundation, which is a huge Foundation. Now, we've got so many projects in there but kubernetes was the the anchor project right? That was the the seed Crystal for this whole ecosystem.

And while I certainly don't pretend to have invented the idea of of cloud native. I do think a lot about what it means right? I spend a lot of time with customers and and users of kubernetes trying to give them advice on how to modernize their applications how to approach the evolution of software from you know, what they might have built 10 years ago to what they might build 10 years from now and you know software it's eating the world, right?

And anybody who pretends that they're not a software company is in denial and so, you know, what is it? What does cloud native mean to me? Right?

I've got a The words that that I like to throw out but the one that jumps out the most is this idea of decoupled. And people say yeah, I decouple them a software engineer. I get it right modularity Etc, but it's it's more than just.

Modularity of code or even a microservices right microservices is a Hot Topic but that alone doesn't make you Cloud native and it's not actually necessary to be Cloud native. When I say decoupled I like to think about all the things that you use to build your systems, right? Not just a piece of soft.

We're not just a program but the systems itself the storage the networking the compute the apis that you use right, you know for an example storage Right storage is always a Hot Topic. It's that one subsystem of our industry that never really keeps up with the rest and people need storage. It's at the bottom of every stack and so, you know, what is cloud native for storage need well.

Historically, it's a it's a database. Right? It's a my sequel or a postgres or an oracle or something that's sitting on top of a file system, which is sitting on top of an array of disks, which is sitting on top of an array of servers and there's coupling all up and down that stack your couple to a specific database your couple to a specific file system.

You're coupled to a specific operating system your couple to specific disks that hold your data. Right? And the more of those couplings that you can break the more flexibility.

You get the more Mobility you get the more automation can take over and fix your problems for you and going the other direction the more things you couple to the more brittle you become more delicate and possible and likely to break. So, you know, when customers ask me. What should I do for storage?

My go-to answer is always Try to use a blob store system try to use something that doesn't depend on a particular file system. That doesn't depend on a particular operating system. That doesn't depend on a particular disk.

Look for something that gives you an API and some slos and rely on those and now you've got mobility in a way that you didn't before. you know when you look at something like kubernetes, if you are running a container on kubernetes and your couple to a particular volume, which is using a particular file system, you are only as mobile as that volume is right as that date is sometimes it's network attached and you can get You can get it within a particular zone or within a particular region, but you can't move beyond that and you know, you have a file system installed which limits what you can do with it. You you can't use the same file systems across different operating systems.

And so I encourage people to think about you know, where are the places that you couple to anything whether that's hardware or other software? I'm also asking people when you when you look at. Becoming Cloud native think about dynamism and dynamic management of stuff everything changes.

And if you accept that as a fundamental premise, then you can sort of build your systems to adapt to it and be ready for it because that's what the cloud native mindset gives you. If you're ready to move around and grow and shrink and adapt to the incoming load requests, then you have a system that is much more malleable and can be deformed in ways that don't destroy it. Right.

It's like the difference between a clay molding and a piece of glass right the glass and the clay can both hold water, but the clay can take whatever shape you need to take. So, you know, these are some of the things that I think about when I talk to people around Cloud native I said, I have a list of you know, 15 words that I like to throw out to people but those are the two that I think are most important. so let me See if I can internalize this and share with the audience Tim.

for you Cloud native It sends almost like George Washington's in Oregon address don't get involved in foreign company and foreign country treaties right stay out of those kinds of alliances for cloud native. You want to decouple? from being too strongly or too closely attached to any one asset of class of that like storage instances as you mentioned go with something as general is malleable as possible that you could hook and unhook from easily.

substitute something else in You know one of the paradoxes about Cloud native for me. Is you don't necessarily have to be in the public Cloud to run Cloud native? You can run Cloud native in your data centers stuff.

Maybe not. Exclusively, but there's no reason that part of your Cloud native strategy doesn't include your own data center assets. Sure, and you know I should be clear.

It's not that you need to decouple from everything. It's that when you couple you should be doing it very intentionally right where you're getting value out of that coupling and so many people couple accidentally. It's just so easy to do it's the way we've always done things and if you think about it, if you start to look for those couplings, you'll find some that.

They restrict your Mobility they don't offer you the value that their cost would entail versus other coupling certainly do right, like trying to build an application. That is completely Cloud agnostic. Is kind of a Fool's errant right?

Like it's very difficult to do and in Practical terms. So you're gonna pick a coupling at some level, but hopefully that's a couple that brings you value and then whether that's your data center because you have value in your own data center in your you know, operational stack. That's that's fine.

Cloud native is an ethos. It's a mindset not literally I have to run in the cloud now. Running in the cloud certainly gives you some advantages, you know, and that list of 15 words.

I throw out, you know, elastic is another word that's out there and you know elasticity is sort of hard to do when you own the data center, right when you are the only customer it's hard to have infinite capacity like a cloud wood. Yes, it is. No doubt about that.

Tim here's another thing, you know this whole mindset versus destination. Way of looking at Cloud native is and you brought up elastics. That's actually a great.

example the marketplace for quote unquote Cloud native applications that you can use has sort of well, it's not sort of it had exploded. Right, there are so many applications that. Are either Based On A Cloud, you know cncf.

sort of project, but this is the commercial version or They make no bones about it. They're not open source. Maybe they have a free version, but they're not open source, and they're not necessarily based on the cncf project or they don't acknowledge.

They are. But they still. considered themselves part of that cloud native toolset and Cloud native mindset right and you know and then you have purots to say wait a second.

That's that's not really cloudinative. Right? It's no, you know, it's not open.

It's not you know, there's no there's no like real test if you will as to whether it's something Cloud native, it's Cloud native or whether we're giving setup is cloud native. Yeah, you're right. There isn't a test and there is no there's no yardstick where you can say now, I'm done now.

I'm Cloud native, right? It's a it's a journey and it's a gradient from you know from black to white and there's an infinite number of Grays in between it and you talked on an open and clearly I'm passionate about open source, and and you know, it's been a big part of the success of kubernetes. I would say the fundamental ingredient to the success of kubernetes.

But it's not a prerequisite for everything you do to be open sourced or every component you use to be an open source component. I think personally, you know over time open always wins. And and so that's something to think about as you're Building architecture for your applications.

But if you find a closed product that solves a problem for you and helps you further your own goals right help you make your own software more elastic and dynamic and decoupled then you should embrace that and if it makes sense for you, that's cool. That's an intentional coupling. Right and that's okay as long as you're aware of the fact that you know, I'm using this thing and it's not replaceable.

It is not swappable. I can't detach from that and make a different decision later without some significant pain every portfolios going to have some of those. That's okay.

That's normal. Just be aware of them. It's just something to think about.

Absolutely. Let me ask you another sort of paradoxical question. Could you be Cloud native without running kubernetes?

Absolutely, you know, we we like to talk about containers as the Cornerstone of cloud native but containers is a enabling technology. It's not a goal in and of itself right containers allow you to build self-contained applications that are you know, more easily moved around than VMS, but it's not That alone doesn't make you cloud-native serverless platforms are I would say also Cloud native. Now, you have a different sort of coupling in there whether you use, you know, this providers implementation or that providers implementation, but you're now decoupled you don't you don't have to care about which data center you run in right?

Because the serverless framework will handle it for you. You don't have to care about which operating system which kernel version you're running on top of because the framework handles it for you for some of these Frameworks. You don't have to care about which language version or runtime you're using because the framework handles that for you, right?

So there's all these little couplings that you can get rid of by going to something like serverless. You can also be Cloud native VM ish way, right? I mean even physical but more VMS because VMS decouple you from say the physical machine I can take a VM and I can move it between physical machines that physical host and run a different operating system.

IVM will never know the difference, right so there's a sort of coupling that you can break there and certainly many big companies have made their large-scale systems around VMS and orchestrating VMS and manipulating. Dynamically. Elastically PM, that's fine.

somebody might pay 61 billion dollars for a VM company go figure hypothetically. Anyway, but you're right. I mean Tim I we're gonna jump into another topic here no moment, but I you know, I want to leave our audience with that message, which is don't get hung up on the particulars of whether or not something is cloud native based upon some artificial checklist that someone has made right or litmus test really it really is kind of a state of mind more about you know, you want to experience the advantages of cloud native brings and if what you doing gives you those advantages do it?

Totally, you know, like I keep saying coupling you look for those couplings. Think about how you can break them. And what does it give you if you do right and that, you know, you drive towards the value for your own business for your own software.

Everybody wants to release faster. Everybody wants to cut a daily build that they push to production automatically without humans touching it that the automation moves around and makes happen and gives them higher slos. That's when we're talking about Cloud native.

Not any particular one technology. Absolutely good stuff Tim. I want to move to the second half now of our of our interview today and in this I wanted to ask you more about hey, as you mentioned right from the get-go of eight years ago.

Someone had said, what do we you know, what do you think is gonna become of kubernetes and what did some of the interesting things that can happen? It would have you know, You would have never guessed it. But as we sit here today, I mean there is some.

You know areas that are still kind of being settled out in the kubernetes world things like multi-cluster and Fleet Management. I wanted to ask you about those Tim and and if don't feel obligated to stay on those two You know, I'm talking what's on you about, you know, Cutting Edge kubernetes if you will, but those two are you know, really kind of one area, but you know, it's the complexity of today's kubernetes. You know installs and and case.

There are studies. What do you think? You're touching on my wheelhouse.

I mean this is this has been my my mission my passion for the last year and a half or so, you know part of being part of the kubernetes process from the beginning. I'm guilty of this but kubernetes for a long time has assumed that our cluster exists inside of its own universe and there is nothing outside of the cluster there is no there's no rest of the universe and maybe the Clusters are expanding but like the the Paradox of what is outside the edge of the universe. Well, there's there's nothing there and I was string theory for Google Nettie.

That's right. But the reality is there's not a single customer that I engage with who doesn't have something else outside of their community's cluster, whether that's another kubernetes cluster or their VMS or their on-prem or their existing infrastructure. They all have something they've already made decisions and kubernetes needs to do a better job fitting into those things.

I think we haven't done a good enough job of being adaptable to fit into the the spaces that people have for us. We've instead tried to be a little bit insular. And so this leads to thinking about and talking to customers who have multiple kubernetes clusters.

We'll start with the simplest problem one where I control the whole access, right? I have two or three or five clusters. How do I make them work together?

What does work together mean for those different clusters? So, you know I've spent time the last year and a half or so working with the multi-cluster Sig in particular to think about what are the principles and the core tenants of having multiple clusters that can make it easier for people to use them together. Right?

For example. I'm a customer who has different clusters for different teams and there are lots of good reasons for that and I want my workloading cluster a to talk to my workload and cluster B. That's it.

I just want them to work together. How do I do that? And the short answer is kubernetes makes it really hard.

We pretend that at the end of the cluster you sort of convert phases of matter into something else and it's it's now your problem to go figure out so, you know as a sing we've started thinking about How do we break down those barriers? How do we extend the universe to incorporate other universes? Right a metaverse sort of theory around kubernetes and we started to build some Primitives and some some philosophies around building clusters that work that are designed to work together.

Right? And if you do X, then we can do why for you sorts of propositions. And so you can see this for example as a community we built this idea of multi-cluster services.

We're now talking about multi-cluster Network policy and we're starting to think about multi-cluster storage. How can I take a stateful application in one cluster and fail it over automatically into another cluster right within gke and Google we're talking about can I upgrade your cluster by bringing up a new cluster and automatically shifting all your workloads over to that second cluster and then turning off the old the blue green trick, right but on a whole cluster level. what do we need to do to make those things happen what sorts of Dashboards and metrics and Telemetry do we need to make those things useful and manageable?

What sort of tenancy model do we need? How do we describe the humans who are engaging with these clusters? And how do we manage that relationship between humans and clusters?

It's huge. so You know when it comes to these kind and let me say Tim both you and I've been in technology a long kind of a long time. You know the the ability for giving technology to scale.

Into multi-cluster or multi-install multi-home, you know. Managing Fleet wide management. It's pretty common task in technology, but it's also a sign of maturity.

Right into it. If no one's using it in that big away you don't worry about it. But now if someone's using it you do.

so this is not like a a case of first You know impression we've dealt with these. situations across other Technologies in the past what? What inherent in advantages or disadvantages I guess do we have with kubernetes because I mean it was kind of made to be scalable to begin with right?

So you would think. It's got to be a little easier this time or is it not you know for for many people kubernetes is their first foray into the idea of cloud native. They're really taking their old server-centric hardware-centric or even single VM Centric monoliths, and they're starting to break them down.

Right and they're starting to think about apis as the the switchboard operator between their components as opposed to these sort of direct linkages and and they're starting to think about declarative infrastructure. Right and again kubernetes didn't invent declarative, but we certainly help popularize it and so I think there is an opportunity here where people are going through a bit of a sea change of how they think about how they manage their infrastructure right the rise now of get Ops or no Ops as a mindset, right? Don't let humans touch the stuff right put it through all these audit processes where get becomes the center of the workflow.

They they're an opportunity for people to change other things now. I talked to a lot of customers and I always tell them don't change two things at once, right? But as they adopt kubernetes they can bring their sort of Monolithic or semi-modolithic applications into kubernetes and then they can start thinking about how to decompose those into services and peel off logical.

Things and as they're doing that they can also start to think about multiple clusters. For for reasons, right and and I have a lot of reasons why people would want to use multiple clusters and it shouldn't be a day two or day three problem. It should be a day Zero or day one problem.

How am I going to plan for multiple clusters, right? You know, it's 2022. There's no excuse for your website to go down for maintenance and these days, you know availability is assumed right if you're if your application isn't up all the time, you're kind of not doing it, right.

I assume you're being attacked. That's for sure thing that pops into my mind these days is when I see a site even if they have the maintenance sign up. I assume they're lying and there really has been some kind of detox or something.

Yeah, exactly. And so, you know, how do you build a highly available application? Well, you decouple again you say well, I'm not gonna couple my application to one cluster because that's a coupling I'm gonna couple it too multiple clusters.

Now, I'm less couple I'm not gonna couple it to one Cloud region or even one continent right? I'm gonna bring up multiple clusters on multiple continents and I'm gonna set up my application across them and I'm gonna bring in traffic and I'm gonna route it to whichever is closest to the customer and has available capacity and I'm gonna be dynamic. I'm gonna shift the traffic between these things and maybe my customer base wakes up in Asia while the us is going to sleep and I'm gonna dynamically shift traffic or shift capacity between these different places because I'm trying to also econ lines and you know as somebody who's going to build multiple clusters.

How do I manage that that is super chaotic right? I'm sure that some of your audience right now are going oh my God, I don't want to deal with this. And you know most people this is a distraction from their business, right?

This is stuff. That should just be day rigor. It's part of the process.

Agree, you know? I mean from where I said Tim to me it kind of breaks into two two camps greenfields and brownfields. Right.

Look if you if you're planning a green field deployment of a new app or even just you know, your whole new infrastructure stack. You can. You can really plan for a scalable.

Stack based around kubernetes and all Cloud native thing. Right? And I think you can do a lot of the things you're talking about.

But unfortunately for a lot of people out here watching right now. They don't have a lot of greenfields. They got a lot of brown muddy fields.

And and saying, you know, it doesn't even scale now and it's not on kubernetes. So now I'm gonna you know, try to transition or microservices infrastructure make a cloud native and I know yeah, I want it scalable too. And and so I got to start thinking in terms of must a multi-cluster and Fleet Management and you know all of these other Kind of things that pop in with the migration.

Right because it's not the old lift and shift or shift and lift from on-prempt to Cloud where you just put it up on the hypervisor and put it up there. It's still thought. Right.

It wasn't truly Cloud. It was just shift and lift and shift. No, transitioning through microservices means like you know, it's like getting a deconstructed dessert in the restaurant, you know, the plate has a lot of pieces on it.

It's hard to imagine what it is when it's when it's all together sometimes. so You know, what? What's your advice to people like that?

Do they real look you go through the pain of deconstructing it. Anyway, you might as well plan for it to be I Uber scalable and and multi-cluster. Is that the kind of model?

I said before like I would always tell people don't change two things at once right in other Pearls of Wisdom that I think I picked up from Kelsey Hightower at some point. You can't take a system that's in trouble and sprinkle some kubernetes on it and have it get better, right? That is not It's not medicine and it's not even like seasoning right here has to be fundamentally stable in order for you to try replatforming things.

Right? And I also I wouldn't take a system even a stable system and then move it to kubernetes and break it up into a hundred microservices at the same time because you never you're not gonna know what's going on. You have no clue what's happening.

So, you know for those customers who have brown Fields all of them right? Think about what are your what are your biggest pain points is it? The ability to roll out releases is it that you have subsystems that are particularly problematic and are regularly causing rollbacks of the whole monolith.

Think about how you're going to attack those individual problems. And then those are the places where you can start to bring out the the Chisel that is microservices and say, okay look the the something system subsystem. Well, that is always having bugs.

And so we're going to break that off into a micro service so that we can version and roll it back independently. Okay. Well, my storage subsystem is the one that is causing most of my scaling bottleneck.

So if I break the storage system out into a separate component, then I can scale that one independent of my front end applications, right or maybe it's vice versa. I need more web server front ends, but I don't need more databases, right and so you gotta you got to look at your systems and understand what's going wrong and that starts with being able to understand your own systems which you know, sadly a lot of people have a hard time with because they built these things over the years and they've accumulated a million bits of of cruft and so you can't just say I'm going to solve all my problems with kubernetes does not make your application scalable. It does not make your Microsoft microservices magically decouple themselves.

It does not design good apis for you, right and it does not. Get rid of bugs, although occasionally it can hide them. So you have to approach this from an engineering discipline first.

And you know, the reality is everybody's got a brown field. There are very few Green Fields out there. So you start moving them one by one look at the ones that are sort of most impactful for you.

Those are the ones I would focus on the ones that are most difficult. I would focus on right the go through the hard problems first and start to identify. How do I scale this?

Right? Is it going to be horizontally scalable or is it vertically scalable? Do I just need bigger servers or can I actually start to break this up into replicas, right?

This is not new kubernetes didn't invent any of this. very cool All right. Hey, Tim.

How much year after? year share here your wisdom and knowledge and in this area and it's it's so appreciated. Keep up the great work.

That well next year. Hopefully I'm coming out to California. We're doing this in person.

God willing and until then though man keep doing what you're doing and thanks for joining us on cloud nativecon today. Thanks so much for having me good to see you again. All right, that's it for our Fireside chart with Tim Hawkins.

Enjoy the rest of your day of cloud native. There is so many great sessions check them out and thank you very much.