David Sandilands and Ben Ford, Puppet | OSS North America 2023

This is techstrong tv. Welcome back to the Open Source Summit here in beautiful Vancouver. We're here with David and Ben from Puppet, and we're talking about open source software and DevOps.

This is peanut butter and jelly. Right? It all comes together.

It's perfect. Uh, I don't know if I would go as far as to say perfect, but we try, we try. So what are the challenges with managing open source software as you build applications through a DevOps workflow and you got all these different pieces and versions and the pipelines.

So from what you guys have seen, you know, what are people struggling with? It's an inter interesting way of looking at it. I, I, I think because, uh, open source software isn't inherently different from proprietary software when it comes to just managing and installing, uh, uh, it across your infrastructure, there's less license management.

Uh, so you don't have to, you can just do it, which is really nice. Um, but there's a ton of benefits and that, that's the way that I, I like to look at, it's like benefits of, of open source. And, um, I think there's two real big benefits that I kind of want to, uh, talk about.

And that's trust and collaboration. And the thing about trust is like if you look at DevOps pipelines, you look at, uh, configuration management platforms and whatnot, there's a lot of things that you're trusting us with, right? Like, we have root access on your systems, and we're configuring your systems with a lot of of things.

And, and being able to look at the source code allows you to trust us, allows you to trust the content authors, uh, the modules, you get off the forge. It gives you a, a much, much, much greater level of trust in what you're actually deploying into your, to your infrastructure. And then the collaboration part is, in reality, a lot of these tools don't offer a ton of value by themselves.

It's the content that that gets deployed by them. Uh, and puppet's value, like is the what you can do with Puppet. And that's all enabled by content.

And we do build and manage, maintain a lot of our own content, but the vast majority of it comes from our community. And so engaging with community and working with them together on their open source projects or open source modules, that's how we build this vast, vast value for the entire ecosystem. How do I maintain the context given all the dependencies that can exist between all those different tools as I'm going along?

Is that kind of what you are providing? Is the value add, or what is the role of Puppet in that conversation? Do you wanna give a stab at that or?

Yeah, I mean, I think, um, you know, when you're looking at big organizations often have, uh, almost a mistrust of open source because they, they don't, you know, when, as you're saying, there's a lot of open source products, um, they know they can get the value, but you know, they have a certain fear of understanding how to work for those communities and, um, how to contribute into those communities. I think when an organization like Puppet comes in, um, you know, with our strong history and strong relationships in those communities, we're we're able to kind of, um, provide that sort of enterprise level support for, for our enterprise level customers. And, uh, you know, at an open source level work, you know, work alongside different communities, um, and, you know, work on the right kind of integrations and partnerships, uh, you know, to, to, you know, to give everyone confidence using the product from, from those who are using it as purely open source to those who are using it commercially.

Are you becoming something of a conduit back to those open source projects about how their software's being used in the enterprise? And they may or may not take that into consideration, but there's a dialogue starting to happen. It's, it's definitely a dialogue.

Cuz I mean, what, I mean, i, I come from the, the, the enterprise side originally, you know, I worked for a bank for 13 years, and one of the strongest aspects we found was it's very hard to deliver open source, you know, but frankly, any software into those kind of organizations without, um, good, uh, usage examples. So we often found it was very powerful. If, you know, you're looking at Puppet, if we had kind of, um, good open source customers who showed how they used the products, that was often one of the big feeders into the enterprise customers so they could understand and see how people are already using it, rather than having to try and, you know, take something and immediately putting it into, you know, the production of a bank, for example.

Um, and it equally does feed back because, you know, you know, in the, in the Vox people eye community that we, we work with directly, um, you know, we kind of message the things that our customers are trying to do. Um, and they kind of discuss the challenges that they're working on. And, you know, often by going back and forth, we can come up with the best solutions, uh, you know, through that cooperation.

One interesting, uh, part about this ecosystem is that we're, we're basically talking about two layers of, of open source is that there's the open source projects that, that everybody's all familiar with. There's Maria idb, MySQL, uh, Apache, et cetera. But then the puppet modules that, uh, allow Puppet to manage 'em and maintain them, those themselves are open source projects.

And so while we, we don't necessarily have direct engagement with the maintainers of, of the managed projects like Cappa, uh, we do engage very, very directly and, and practically every single day with the, uh, the, uh, maintainers of the open source pilot modules. Mm-hmm. Um, like for example, I have a, a weekly, or sorry, a monthly sync with the, uh, the leadership of the box for Blood group.

And we talk about what challenges they're having and what challenges we're we're having, and maybe give some like, heads up of things that are happening. And we, we just kind of work together on ing some of these things that we're struggling with. And not all those folks are puppet employees.

I think it's, so they're, how big is the community around these different modules? It's very large. Uh, but there's different levels of activities.

We have about 8,000 modules on the Forge. Uh, probably one to 2000 of those are, uh, actively maintained, uh, like people are making active, uh, uh, commits and updating them, uh, uh, month by month, uh, year by year. And then the, uh, Vox Pule group is about 300, uh, individuals, uh, uh, with probably, uh, 10th of that are, uh, highly engaged.

Uh, those are the ones who attend this, uh, this monthly sync. We have a, a small leadership group that is, uh, that attends these. And we, we do literally talk to them like they're on our, uh, uh, slack channels and we just like, uh, chat back and forth and, and, uh, a lot of them we even consider, uh, uh, personal friends.

So are you gonna add more of these projects going forward? Or, you know, do you got enough and just enough to handle as it is? I'd say that's up to the community.

They'll add, they'll add projects, they'll add puppet modules for the things that they need to manage, and the things that they, uh, are finding less use, uh, uh, managing those are kind of like, uh, we'll let them, uh, sunset. Mm-hmm. So the, uh, the number of modules I think is, is growing steadily, but it's not like exploding.

It's a mature ecosystem. So it's like in that state where it's a, a slow, steady process. David, how is, after we build all these applications, how is the delivery process evolving from your perspective?

Historically, at least from my understanding, everybody does CI pretty well, but CD's been a little bit more challenging. And so where are we on that maturity curve? Yeah, I mean, I think, I think we've made a, a kind number of assumptions in the past about how sophisticated, how mature people are and understanding how they should test and how they should manage pipelines.

Uh, I mean, I think, you know, at, you know, the events to, you know, in the past few days, speaking to, um, some of the, you know, supply chain security and understanding how to have better, you know, be, you know, better, uh, security as things are published and, you know, things are delivered through pipelines is definitely something we'll be investing more in. And, um, looking to create partnerships with, um, with other groups, uh, to he to help standardize that sort of thing. Cause I think a lot of, um, you know, even kind of, you know, we talk about where Vox pupil are, you know, they work on a community standard to understand how best they should do pipelines.

Uh, you know, our customers can struggle as well, cuz I think sometimes as an assumption they're going to, you know, have the institutions and organizations internally in big enterprises, but often they don't. Uh, so I think trying to help build more in the community and, you know, bring it to the enterprise as well. You know, so just like, you know, we've talked about the, um, how the open source examples help, um, them see how to use our products, showing again, how pipelines can be done well in the community and bringing them to enterprise.

I think one interesting aspect of that is that one thing that we're, we're learning and we're kind of messaging is that configuration management and, and, uh, a lot of these DevOps tooling it's software and we deliver it with software pipelines. Mm-hmm. And people are still kind of struggling to, to get used to that.

Mm-hmm. We hear about the rise of platform engineering. Is that an attempt to kind of bring some order to the chaos?

Yeah, I mean, um, this, this was very close to me cause I I, I've been recently presenting on, you know, puppet does its state DevOps report. Uh, this year's, uh, was, uh, platform engineering. Um, and, you know, we'd, we'd looked into it as a result of, uh, last year's result kind of showing that platform engineering was one of these things that we saw, um, uh, as a kind of prominent feature of highly evolved DevOps companies.

Um, I mean, there's obviously been a lot of buzz around it, but I think when you kind of break down into those who do it well, um, I, you know, I I think it has a huge effect on people delivering well and, uh, freeing up the developers so the, you they're not having to do, uh, you know, reinvent the wheel and do repetitive tasks that aren't, you know, their core domain. Mm-hmm. Um, and yeah, uh, you know, it is still relatively young, I would say.

You know, the, the survey showed us that the, the majority are kinda under three years. So we're gonna be interested to see how people continue to commit to that. Because often with these sorts of phases, people will, you know, everyone will start up their own platform.

It'll be, you know, there'll be money or there'll be interest from community or, or leaders. And then it's, you know, when it gets into that, um, longer period where it becomes a challenge to keep it going Or somebody may rebel because they're gonna perceive that this is the revenge of centralized it. And It's real big challenges, right?

Is is that we need to present it as being mutually beneficial. And, and once people realize that they get, uh, value out of this, they're just in as invested in, in the platform as the people building the platform for them. Uh, I mean, there was, there was one really interesting thing in the, the, the state day op support on that point, which was, you know, the, the, the survey was done to try and make sure, you know, it wasn't just operators saying, of course platforms are wonderful, but you know, you had the users as well and in terms of, they were talking about how, you know, how centralized people wanted them to be, uh, anyway, I mean, centralized in terms of multiple business units, you know, having a platform, uh, you know, it was kind of universally popular, uh, and viewed as university beneficial for delivery from both operators and from consumers.

Mm-hmm. What surprised you in the report? What was in there that you didn't expect?

Because you've been around the block a couple of times now? I, I I, I, I think the biggest thing for me was actually the, there was a sort of universal positiveness. You know, you normally, you normally less get, get at least a, some cynicism where it's kind of, um, I think it was almost 94% reported it was a, a benefit to, um, it was a benefit to, uh, their adoption of DevOps.

And I think also a very large number reported it was, uh, increasing their, uh, delivery time. And, you know, I, I'd normally expect to see a much more substantial number kind of saying, this is, you know, this is just hyper, this is his buzzwords. Whereas it seemed to be quite universally believed in, and Again, that was users and operators both, right?

That was users and operators, both. And, you know, the, the report also, you know, I did have control groups to try and, um, you know, to try and, you know, ha have, you know, the views of people who weren't using platforms that only seen platforms. What do you guys think is the state of the relationship between developers and the software engineers that run the DevOps teams?

Seems like, you know, we want to shift a lot of things left and sometimes the developers are using DevOps tools and sometimes not. And how do we kind of bring all these folks together? Well, I mean, both silent.

Yeah. I mean, I mean, it, it's all a challenge. It's always a balance.

I mean, you know, you know, as we're talking about the platforms there, you know, we don't want to over decentralize things in such a way that people, uh, don't get the best tools that they need. So I, I, I, I think, you know, when we're highlighting one of the, you know, just touching again on platforms, one of the things that was highlighted was it needs to be product managed. You need somebody going to the developers and, you know, finding out their needs, finding out the directions are going in, and making sure that, um, making sure that, you know, the platform or whatever you're providing is evolving to their needs.

Yep. Um, and equally, there's always gonna be some exceptions. And, you know, we need to give enough freedom for developers to develop in the way this effective for them and allows the, you know, the release to be effective.

Um, but, you know, as far as possible, you know, providing those tools centrally and, and through platforms, I think that, like, as a developer myself, the thing that I want out of a platform, uh, like this is I want it to shift left the things that are going to provide value and benefit to me, but not the things that aren't going to. And it's like, I don't really want to have to think about writing ingress routes and everything. I, I just wanna say like, my application should be available to, to the internet.

Um, and, and having that ability to like, abstract away the things that I don't need to care about, but while still being able to like, drill in if I do need to care about 'em, that's like, to me, that's the, the way that we get to, to, like, we, we get buy-in and we get people, uh, talking to each other and, and, and both into the, the platform Is the cognitive load on developers too high. I mean, ultimately you wanna write business logic and the, and the fun stuff, right? That I, I, I think that is, is so very, very concisely what I'm, what I'm trying to get at is like, if I have to understand all of my business logic and all of the operational logic, it, it's, it's a lot to, to, to hold in head.

I mean, I think one thing we commonly see is, you know, you know, organizations who do it well quite often form communities of practice and they bring in lots of groups of developers and you know, the engineers and have those discussions of, well, how are you doing things now? What tools are you using? And they look for those things or commonalities.

So, you know, you're then able to identify, well actually, you know, we can see these things are in common in our organization and those are clear targets. We can create platforms around and, you know, find useful ways to work together. And equally, some, um, some developers you may kind of be, we were only using this because, you know, we had to have something may actually find that something they can drop.

Uh, you know, and, and I, you know, focusing on where we can reduce the cognitive load, you know, you know, well, and not kind of just saying, this is the one platform to rule them all and you have to use it. Have we, to a certain degree, DevOps has always been about ruthless automation. Hmm.

And yet you go visit some folks and you talk to 'em and you find out there's a lot of manual processes floating around and there's a lot of toil. So do we kind of need to take a giant step back and rethink all this stuff? Because I think we get into a mindset where I bought this thing five or six years ago and it doesn't occur to me that maybe the world's moved on a little bit.

You Know, there was a cartoon that it, that I saw, I think it was X XKCD that talked about like the, the, the value you get from auto, uh, like the time you saved by automating something versus the time you, uh, spent to, to do that automation. And I think that's, that's really it. It's like if we get value from automating something, then we need to do it.

And if it's not, if it's, if it's a manual task that doesn't have to be perfect, done perfectly, and we only do it once a year, then we don't get the same amount of value from it. But, Oh, sorry. Go, go on.

I was just gonna say, there was one key thing just on that experience recently where, uh, I was working with a customer who was kind of disappointed. They hadn't got enough automation, um, in one of their pipeline setups. Uh, but actually I then spoke to the other side and application teams were delighted cuz the key thing they got out of it was they were in control of their own deployment.

You know, they'd previously been in a gateway sort of environment where, you know, a a a central infrastructure team had to approve their code going to production. And even though they had these manual steps, they had some change process things to do. The fact was they could do under all, all their own steam.

And now, you know, they were able to say, when we wanna deploy code, we just have to put the effort in to get through those steps. So sometimes it's not even just about being fully automated, it's about, you know, developers having their own control over their own, you know, their own code to deploy. Are we measuring the wrong things?

We're all obsessed with Dora metrics, but in reality there's this thing you call developer experience, and maybe we should be measuring that more so than, you know, what my uptime was for a release cycle or whatever other thing that we're kind of obsessing about at the moment. I think one of the things that I like to measure is frustration, developer frustration. And I think like key over everything that that's the, the, the summary of, of all of the metrics.

One question I want to ask you, where we've been talking about security all week long, where does security fit in this DevOps workflow? And, you know, is that a security team coming to the DevOps team, or am I training the DevOps people more on security? Where do you think this is all gonna play out?

I mean, I, I, I think it has to be all levels. I mean, and, and I think again, you know, mentioned community of practices. I think when you can bring those teams together to discuss how security needs to be done and you know, you know, work out where best things should go into pipelines, what tools should be used and just make people aware where the risks are.

Um, you know, cause I think, you know, if you look at, if you look at the kinda supply chain management that's been talked a lot about today, and you can see a lot of vendors here today talking about it. I think there was a kind of naivety around, you know, the publication of open source software, whether, you know, it was packaged software or modules going up into shared resources where, you know, you know, the bar was often relatively low in terms of releasing, you know, releasing to those areas. And now, you know, there's big questions about, well, to make it easier for people to consume that we should actually have, you know, not necessarily gateways, but you know, just kind of, um, you know, almost like being scanned, coming through the airport to make sure, uh, you know, there's nothing bad on you, there's nothing kind of obvious patterns or, you know, we're hearing some from some other vendors about how, you know, they showed there was kind of 30% of MPM packages that did nothing.

Uh, and you know, there was a suspicion around, you know, well, clearly that's, um, looking to pollute and looking to kind of create problems. Uh, but again, like what, uh, you kind of hinted at earlier is that it's, it's multi-level. It's like you can't just like buy a product stick somewhere in, in your pipeline and, and call it done.

It's like we, we shift security left, but we also have to keep, uh, checking it in. Uh, we, we secure our pipeline. When you deploy your code, you need to, uh, to, uh, secure that when you put it in your, your infrastructure, you should still be scanning that and make, making sure that what's actually deployed, uh, is, is still meets all of your cloud security, uh, requirements.

So it's many, many levels we have to pay attention to. All of 'em. When I go to the airport today, it takes me a lot longer to get on that plane than it did 10 or 15 years ago.

Do we have to accept that maybe the pace of software development is gonna slow down a little bit as we put more of these controls in place? Or can we keep going at the pace we're at and be secure? I think that we're gonna have a little bit less security theater than the the tsa.

I mean, I, I I, I think the, you know, I think it's a funny thing is almost the difference is, you know, if you, you know, if you get your developers, you know, so they, so they understand what they're going through. Because you know, the difference is when you go to the airport, there's a good reason, which can been in a while, so you're gonna stumble through find out, oh, I've gotta take my shoes off now. And I think it's that same thing.

If your developers aren't really sure what they should be doing, then they are gonna stumble through that kind of pipeline in that queue. Whereas if it's kind of well reversed and well understood, then, you know, I always think it's that key thing. If people understand why the security steps are being done, then they can be far happier to follow them and, and, you know, and see them of value is when you're gonna get these arbitrary steps that people can of, you know, why the heck can I take my shoes off?

I, I don't know. And, you know, then they become more resistant and more fumbly with it. So I, I I think it's that thing of, it doesn't have to be, if it's done well and the education's done well and the promotion's done well, um, but you know, I I think it'll take time to get right.

Is he right? Do we just need a known traveler number for secur, for developers? Oh, Uh, no.

I, I think he's totally right. But one of the things that we're doing at Puppet is, is like our secur, we, we do have a security team, but in reality, uh, our security team is all of our engineers because we all work together, we all talk together, um, and we all participated in even some of the decision making and, and how we're going to go about, uh, uh, solving some of the problems. So there are some steps that we, we have cut down and may, may take a little bit longer.

Um, but, uh, we all understand that at that is going to save us time in the long run is going to, to make a both make us mo more secure today. And it's also going to save us time in the long run because we don't have to resolve, uh, uh, things coming up, uh, down the road. And I think we're, we're all on board with it.

What's that one thing you see DevOps seems doing over and over again that just makes your shake your head a little bit and go, I think we're better than this, folks. I mean, I mean, for, for me, I think it's, um, again, this is coming from my banking experience. I think the thing is that common thing of when tools are just dropped down and, um, you know, they may, maybe they've bought a book or something and said, right, we're doing scrums, we're doing this.

And you, you can see that in terms of the kind of the, the, the, the, the teams and the, and the kind interactions with each other, nothing's changed. So they're more or less kind of doing exactly what they were doing before, but they've just kind of translated it into Jira and, you know, sit sitting down every morning and saying what they were doing, exactly what they were doing before anyway, you know, so I think it's this assumption you can almost just buy DevOps rather than actually working together to work in a different Way. You mean that waterfall approach to DevOps is working really well?

Yeah. Well Also, you know, you see some sprints and they're essentially waterfall sprints, you know, the no, you know, nothing's changed and how you deliver, here's An assignment, go do it. Yeah.

This Is really funny because you started talking and you said almost exactly the same thing. I was gonna say. It was like, look, like I, the thing that bothers me is people think that they can buy a thing, flip a switch to uh, do a toggle and, and they're done.

And you have to think like, there are paradigm shifts that are happening every day. We're learning new things every day, and we, we have to take those into account. We have to learn, we have to adapt and, and we have to, to like change how we think to match where the world is going.

All right, folks, you're hurting here. There's no getting around doing the work. You just can't like show up and drop the tools.

You gotta actually do the thing. Guys, thanks for being on the show. Thank You so much.

This was a lot of fun. All right. Yeah, Thanks.

Having Us Pleasure folks. We've come to the end, believe it or not, after three days. We're gonna call it a day here.

We want to thank you all for spending your time with us. We want to thank the Open Source Summit folks for having us, and we look forward to coming back to the next show, and we hope that you'll join us once again On behalf of Textron Group, thank you all and good night.