AI Exposes Open Source Documentation Debt

Hey everybody, we're back here at the Open Source Summit in Minneapolis with my new friend, Paul. Paul, nice to meet you. Nice to meet you, Mike.

Don't worry. You don't look like the Paul I know, but it's nice to meet you anyway. It's all good.

Thank you. Thanks for the invitation to be here. We're having a little chat, though, about documentation, which is having a moment.

It is. It's a long overdue moment, I might argue, but... Yeah.

We heard the fearless leader speak yesterday about the technical debt associated with security. But we're staring in the face of another whole bucket of that related to documentation. Right.

And it seems to me, with the AI agents, they're looking for the documentation first. They might arguably be the first entity that's reading the documentation. Yeah.

No. We like to say that nothing will find bad documentation and expose it faster than an AI system trying to program against your platform. Yeah.

So what are we supposed to do about this? Because we've all been kind of paying lip service to documentation. God bless the ones that actually do it well, and there's been instances out there where clearly they put some thought into it, but a lot of the documentation is, shall we say, sparse, and especially around open source projects.

What do we got to do? Yeah. We need to make it a priority.

I think we need to consider it part of the whole product. And I think there are ways, in terms of education and really upping that level of effort, and then leaning on people that know how to build good documentation. I think those are the elements of the programs that we're finding some success with.

Can I maybe use AI to create better documentation because at least I'm getting a leg up on it a little bit? It's certainly a part of the answer. We have seen some companies make some decisions about reducing staff in the areas of documentation.

I'm blessed with a very deep bench, which I don't usually have right now because of some of those decisions. But those far more common is the case where writers are being asked to enhance what they do or augment their efforts with AI. And so the tools are getting better.

The AI is certainly a tool that can be very productive and help writers get the work done. There's always more documentation to do than writers to do it, so we need automation to have a chance at keeping up, and AI is providing some of that. When it comes to documentation, what is it that most development teams overlook or don't think through well enough that you've seen a pattern where it's kind of like a recurring gotcha?

Yeah. No. There are some common themes to the projects that we've worked on.

One of those themes is there tends to not be a crisp definition of the different kinds of users that are using the documentation. We've seen that the informational needs of someone who's a systems administrator are different than those of a contributor. Both of those are different from a developer who's just trying to use the API to the system.

And there are end users and different kinds of roles within those users. And now we're seeing that same kind of definition apply to different kinds of development environments that are being automated. So what Claude needs might be different than what Cursor needs.

That might be different, again, from what Gemini might want. So, I think that being crisp about who your readers are, who your targets are, and then understanding what they're trying to accomplish, can help inform documentation efforts to make sure that they can find what they're looking for, and get the answers that they need to be successful. Mm-hmm.

We're at an open source conference- Right ... and there's a lot of projects, and they're always looking for people to contribute to the projects. But you actually work with some of these projects to provide that documentation.

We do. So what is it that you guys do, exactly? Well, most of our work, especially with this conference, is organized by the Cloud Native Computing Foundation.

One of the benefits of joining CNCF is they have a documentation assistance program, and we've helped define and power some of the solutions that they use with their community. In their system, they have sandbox projects that are just getting started. For those projects, we've built some free training that is available on the Linux Foundation training portal that maintainers and contributors can take that are interested in learning a little bit about documentation, and kind of understanding how to talk about it and what the right approaches are.

For sandbox projects, we've also built a primer and a workshop where we can help those smaller teams get off to a good start with documentation early in their evolution. For projects that are incubating and looking to graduate, there's a little more sophisticated analysis that we can provide. There's a methodology that's been documented for how to assess, "Hey, how good is our documentation?

Do we have everything we need? What's missing? " So we'll use a rubric that has been established and refined over the years to kind of evaluate where you are or where a project is, and then we'll meet with amaintainers to talk about our findings, and then recommendations for what steps they can take.

But so far, there's generally pretty good consensus about what the work is, and then we'll help by actually writing issues that contributors can close that are a little more consumable for someone who just wants to be a contributor. If they want to help with documentation, it helps if there's a little bit of a plan so they know where to start. If you don't have that set up as part of your project, as a contributor, you go to help with docs, it's kind of hard to know where to start.

True. Do you think also that enterprise IT organizations, then, are consuming some of these projects? They evaluate the documentation, and it's kind of part of the maturity thing that you're discussing.

" It's true. For a technical professional, like a software developer or an IT person, the line between really effective documentation and really effective marketing is indistinguishable. It's really the same information.

Professionals, they just want to know what it does and how they can use it. And the stakes are high now, because if they can't find it, there are many competing alternatives. And so if you go to evaluate a solution and you can't quickly see how you're going to get value from it, you're onto the next one.

And that's terrible, but worse than that, in today's day and age, you tell all your friends that whatever you just looked at is not quite ready for prime time. Mm. Now, the next person that comes to evaluate may have a negative impression based on something that they read or heard from a friend that may or may not be accurate, but that's where they're starting.

That's true. That's where they're starting that process, right? Also, there's these compliance and security people walking around.

Right. And they also want to know what's going on with this project and to evaluate it. " And it seems also with the IT folks, there's not a lot of feedback loop.

There's just an opinion that they have, and they pass through. So- That's right ... are the maintainers making some assumptions about their projects, not realizing that there isn't really a feedback loop for documentation?

Well, there can be. There are things we can measure what happens, but you're right. We may not capture a key piece of information from a user.

So it's important to have someone on your team that is anticipating the needs of the audiences you're trying to reach, and then making what they need obvious to find. In fact, oftentimes when we get started, we try to document all the really crucial stuff that should be obvious, but is not. Mm-hmm.

So- So what is your best advice, then, to the maintainers, or for that matter, anybody else who's building software, about how to approach documentation? Because I also think they view it as a chore, and it's not their least favorite- Yeah, well- ... adventure ...

we're all cursed with the experience that we have or don't have. So my advice is to get someone who has either done technical writing or has some experience with technical writing. Get them involved early, even on a part-time basis.

Sometimes just naming things in an intuitive way, getting the things organized in a way, and the repo named, and the different parts of the system with a little bit of overview so that you know how to navigate a particular system as you're getting started. A little bit goes a long ways. Mm-hmm.

And check out the free training from Linux Foundation. Check out the primer and some of the documentation. They're in that CNCF Docs repo.

And if they need help, reach out for help to the CNCF, or contact Expert Support. We're happy to brainstorm. How do people reach you if I am the maintainer of a project, or for that matter, if I'm an individual technical writer and I want to work with you somehow?

Yeah. Well, great. Our URL is easy.

com. And there's a Contact Us button sprinkled all over that website in various places. It's easy to schedule a meeting with me, or call us up.

We're not hard to find. All right, folks, you heard it here. Documentation.

It's the beginning and end of everything. So make a good impression the first time. Right.

Thanks, Mike. Hey, buddy. Thanks for being on the show.

Thank you. Thanks for having me.