Runxia Ye, CloudBees | DevOps World 2023

This is Textron tv. Hey everybody. Welcome back to DevOps World here in Santa Clara, California.

We are having a fantastic time talking with some amazing people, of which I have another amazing person joining me today. This Aye, who is, uh, director of product management with CloudBees, which knowing what you just got through working on and delivered to market is, you know, a gross understatement of how much work you've put into you and the team. So thanks for being here.

We'll, we'll dive into what that is you here. Great. It's a pleasure.

Re pleasure talking with you. So, you know, so many of us have used Jenkins since early on if you started DevOps way back in the dawn of DevOps, right? Yeah, exactly.

Uh, and and it's still heavily used. A lot of organizations still use, uh, Jenkins. Yeah.

And of course it's evolved over time. Um, but in talking with people like you and others enterprises have adopted it so much, we started kind of hitting the ceiling of what you can do on a larger scale with Jenkins and how many instances, which is you, what you've been working on, and then release. So tell us about that.

Yeah, absolutely. So indeed what you're saying, saying is exactly the case that we have been seeing the last couple of years. You know, the amount of Jenkins controllers and jobs just increases, uh, a lot in these big companies just now is meeting with someone and they're, you know, six, 8,000 plus 50, you know, up to 50 developers.

But if you count the users, because people not only use Jenkins for ci, they use it for anything you can automate. Yep, very true. It's very useful.

So up to 15 k you know, users and, and then you have 200 controllers, and then people would start thinking, how am I going to manage that at scale? How am I going to ensure governance and my organizations, et cetera? And so different aspects, when you come to that scale, you will encounter.

Um, and that's where clubby so, um, and my team we're looking at which of these problems can we help solve today. Uh, and so super happy that, uh, we did the first big one. Yeah.

And a couple of others will, you know, more focus on the developer experience because that's also very important Of course. Um, Excellent. You know, thinking about this, you know, you make c certain architectural decisions and software about how it's, how you're gonna scale it, what mechanisms you're gonna use to do that.

And not always, but inevitably you'll bump up against those assumptions of whether they're, they're, you know, kind of survivable beyond a certain point or whether they serve you in the same way they did when you were operating at just a little bit smaller scale. Yeah. To bring this, to bring HAHS to Jenkins, did you have to rethink kind of how to scale it?

Or did you do things to just give it a bigger boost to be able to scale more? How much did you have to re-architect to, to do this? Yeah, so this one actually is a really good question.

Um, 'cause it's a big problem that we have been actually looking at for a while now, right. Um, and we've done a lot of different researches, different approaches, and tried, um, this particular feature that we have released this year. Um, I actually gave the team a little bit of a, a constraint in how they design it.

I said I would prefer to have one that has actually not too much fundamental change to the existing architecture. Mm. And the reason for that is it's such a critical feature that I want all our customers or our users to be able to benefit from it.

If you make a major architectural change, it'll mean a really high adoption kind of threshold. They'll have to migrate data and, and whatnot. So what is the best option we can have without doing that fundamental or, you know, change?

And the team came up with a really good solution, of course, based on a lot of research that was done before as well. And, and so we run it, uh, we keep the, the file based systems as is. Um, but then they thought of quite, you know, creative ways to make sure we can do the replication and, and also to have these automated kind of handing over adoptions between the replicas and So on.

That's a pretty brilliant thing to do, by the way. Absolutely. I'm so proud of them.

That was just amazing what they did. Yeah. So let me ask you, taking that constraint to your team is that the, a lesson learned through sch school of hard knocks, you've gone through those, not just major upgrades, but you know, it is a full kind of reinstallation conversion, massive effort problems.

It never goes smoothly. If you had those experiences and you're like, well, let's avoid that path If we can't Yeah, yeah. Every single time.

Right? You, you make those decisions and you get this feedback. Is that Yeah.

Okay. Lessons learned. That was not the best way to do it.

And to be honest, also for developers, um, for our engineers, sometimes you're more creative when you're constrained. Mm-Hmm. Oh yeah.

Absolutely. Right. So that was also kind of one of the motivation as well to say, okay, let's simplify the problem a little bit for you.

What is the best you can come up with? Mm-Hmm. And, and we were just happy that it was satisfying all the other, uh, requirement that we had all the benefit that it could get.

Sometimes no constraints are a lot harder than having constraints. Absolutely. Because you're like, I have too many possibilities.

What do I choose to do? That was actually how it started. The very first con conversation going into this feature.

It was like, okay, so I can do this, I can do that, I can do that. Here are the pros and cons. And then we narrowed it down to say, okay, these are the key use cases we want to support and here are some design constraints I want you to have.

Yeah. How far do you think this takes us in terms of scalability ha kind of capabilities of Jenkins? I mean, you had to test it to a certain point.

Yeah. Maybe you couldn't test it as far as you wanted to go, but do you have an idea where you think you might have the next threshold? Oh, well we really have a threshold, which is still the file system, right?

You can't have many replicas, but if they're all writing to the same file system, you'll have a limit somewhere. Mm-Hmm. And there is a cost associated to that as well.

So yeah, we're thinking about the next step of, um, making that a lot lighter so that you don't have that problem or less Yeah. It's actually a better way when you can do it that, because then you're solving incremental problems. Exactly.

Not changing everything, not Boil the ocean. Yeah. We do it step by step, but they're very logical and they take you every time a little bit further Right.

To yeah. To Get much more the DevOps way. Exactly.

Exactly. Exactly. Interesting.

Okay. So what's the reaction been? Have folks been like, oh, I'll need that someday.

Or like, we've been waiting for this for Robin the second. Yeah. Or that ladder.

Okay. Yeah. Even had a customer like jumping up to say, okay, where were you 10 years ago?

But yeah, Let me ask you, not that you will do this, but sometimes when you go down this path and you're solving a a, a particularly tough time, you have to create ways of testing it. Mm-Hmm. You have to create maybe some tools to manage it.

There can be different things. Sometimes those things can be products themselves like, oh yeah, I would like to be able to, would you productize the CloudBees test suite that you use to test ha because I wanna do that on my whole application. Right.

You foresee something. Not, not that you're going to not, not that you're gonna announce anything today, but do you see that as a possible, because sometimes, you know, that need that mother of invention, you know, kind of philosophy, all of a sudden you've got something you could take to market. Absolutely.

Yeah. We're looking at these, I think they, another benefit of that approach is indeed you have already been test driving yourself this solution, right? So productizing it.

Yeah. We have a couple of candidates like these lined up that we'll be thinking about, you know, which one to would bring benefit value to, to a larger customer. Um, set.

Absolutely. It, it always is one of the hardest things to test high volume failover. Yep.

Availability, you know, it's not every, every, maybe some testers nightmare, not their dream to have to write test suites to do that. Yeah. That particular use case is, um, can be tricky because every customer, their infrastructure they're set up is so different, right?

Mm-Hmm. So tools may not work in their environment as, as such methodologies, yes. You can carry them over.

So that's why sometimes we do help customers rather with kind of advice, professional service consultancy to tell them, this is how you can test with your workload and your environment, et cetera. If we provide a tool, it's not always going to work. Mm-Hmm.

Well, what you said, since there's many ways you can architect, it's set it, set it up, operate it, right? Yeah. There's no one standard way to do Jenkins.

Did you have to spend some time, maybe quite a bit of time talking to customers and understanding Oh yeah. The, their use cases, their scenarios, and kind of like how you may, may, may have edge cases, you may not accommodate everything, but you want to kind of figure out, these are the scenarios we definitely wanna make sure we can Support. Yeah, indeed.

We started with these kind of interviews, conversations. Um, some are actually from other topics that, you know, naturally kind of flow into that. Um, specifically, but also for this, uh, one, we have actually worked with some design partners, right?

Hmm. So really in the, in building these features, that was kind of also, um, when we talked with the, with our engineering team together, you know, to make sure we're bringing value to the, in the right way, it would be good to have a customer who can actually develop with us, right? Mm-Hmm.

Provide quick feedback and, and we can help them because, and, and that's also gives everyone the confidence, like, okay, not just works for us, but it works, um, in their kind of Yeah. Somebody who actually will use it solve problem. Exactly.

Exactly. So that was super helpful as well. So you, you mentioned, um, I think you mentioned developer experience, kind of an area you're gonna focus on.

What are, when you talk about developer experience for Jenkins and some of the capabilities you've brought out, what are the next things to work on to make that better for developers? Not saying you're doing that exactly what you'll release next, but what are some areas, you know, there's need to improve that? Yeah, so there are a couple of pain points we have learned from, again, our, our conversations with, with customers and users.

So with Jenkins there, there's the high learning curve. So, right. So if you talk to our own engineers or those who are very Jenkins expert, that's no problem.

We can do this way, you can do that way. But when you have a, a fairly junior developer who has never worked with Jenkins before, the learning curve is pretty high. Mm-Hmm.

So that's where we want to see where we can, you know, make it easier. There, there are ways, different ways again, but we're just leaving it open currently to every customer, every organization to figure out how they can want to support them. Um, so that, that's an area that we're, um, actively looking at.

Another in terms of developer experience is, um, you always want to get them closer to just one tool. They're, they want to work in, right? Which is typically their IDE, they want to write code, that's where they are.

So can we create, That's what they Live in, right? Can we bring all these just as close as possible, if not inside the IDE for example. Yeah.

That's, you know, it, it's a, it is a great point and you can, you can develop plugins and extinctions for ides that flow really naturally with how you develop Yeah. You can also create 'em so that it's just like having the command line built in. Yeah.

And I can just do it there instead of opening a terminal, which isn't always that, you know. Yeah. Is it worth, worth it just to be able to do that sometimes, I guess it is.

Do you see some of the changes in kind of improving the workflow Yeah. Of developers in the IDE? Yeah.

We see that because, you know, that's, it just saves them that time to have to log in or to go to another tool and, and to get another used to another interface, et cetera. You know, everything could be, many things could be there, um, right. From, you know, where they're actually writing and providing as quickly, you know, feedback as possible.

'cause indeed, you know, you saw the workspace caching one, but you still need to run the build. But what if you can bring that even closer and quicker feedback that Yeah. If you perform this step now as you're writing curve, easy to deploy versus Okay, gotta go back, do some work to get ready for it Yeah.

Kind of thing. Yeah. Yeah.

Interesting. Well, stepping back in time. Mm-Hmm.

When did the, um, we've kicked the idea around we're gonna go, we're gonna go for it, we're gonna make this product enhancement and launch it when it's ready. How long ago was that, when that kind of decision was made? That was, I would say last summer.

Okay. Yeah. So it wasn't five years ago or three years ago.

It was No, in the last year. Plus year and a half. Yeah.

Yeah. Because I mean, five, three years ago we did, let's say, explore the different options that we have. It was not fully mature yet, I would say.

Mm-Hmm. But last year in the drums, end of summer was like, okay, we're gonna do it. Good.

Wow. Yeah. I I bet you had to have a few people on board with you to make that.

Oh yeah, yeah, yeah, yeah. Yeah. It was quite a journey.

I would say Uhhuh, I'm very excited. I'm very happy about the results and everyone is, I think I'm, I'm so proud and I think they're very proud of themselves as well. Yeah.

Yeah. And when you've launched products, you really, you, you feel for the good, for people that Absolutely. When they get to launch products.

'cause it is so it's satisfying 'cause of the amount of work that you put in Yeah. To get it to that point and to have customers say, this is what we've been looking for, This is this What we've been waiting for. Exactly.

Exactly what we needed. Thank you for solving this problem for Us. That's a nice part about my job.

That's a nice one that happens. Well, congratulations. Thank You, Mitch.

Where can, if you're one of those people like, hey, I could solve, I need help with this, uh, where do people go to download, you know, the whatever they they do to install and leverage this? com and um, if you look there for Cloud BCI or find, actually I think on the page about high availability and there you get a lot of information how we can move further. Okay.

Well Rancha very gave very, uh, very much a pleasure to talk with you and hopefully we get to have this conversation somewhere down the road about some new, um, developer experience improvements, looking Forward to that. They come Along. Yeah.

I'm glad. BS great. Looking forward to that.

Congrats again. Thank you Mitch. You Bet.

It's really nice. You know what? It's great to talk to the people that create the technology that we all use.

You know, some of us every day, maybe for many years, maybe we're new to it, but we could be a part of our own journey of creating software. So it's one of the great things about coming to a conference or listening into conversations like this at DevOps world. So thanks for joining us.

We've got some more interviews, do not go away. A few surprises here and there, maybe a turn or two, a curve on the road and we'll show you some great stuff, have some great conversations. So hang tight, we'll be right back.