Verified Knowledge for AI Coding Agents

Hey everyone, welcome back here to Tectron TV. Our next guest is the first time she's here on our show with us. Her name is Alex Lado, and she's VP of product at Stack Overflow.

Hey, Alex, how are you? Hey, super good. Thank you for having me.

How are you? Very good. Pleasure to have you on here.

Thank you. Alex, we're going to talk about an exciting new feature product service from Stack Overflow. But before we do that, I always like our audience to know who they're listening to, who they're watching.

So let's hear from you a little bit- Absolutely ... about your story. Go ahead.

Yeah, sure. My name is Alex. I am a VP of product at Stack Overflow.

I've been here just over seven months now, so still relatively new, I'd say. Before that, my previous life, I was at GitHub, actually. I think one of the market leaders, everyone here very familiar with GitHub.

There, I was charged with driving the product-led growth, self-serve motion from bottoms up, so developer, understanding how they use AI, how they use the SDLC, all the way until those individual developers working in an enterprise context. So really, my team was in charge of taking an individual developer and moving them along that journey. In the last year at GitHub, actually, and I think it's what's very interesting and connects to Stack Overflow, I was charged with figuring out how to measure AI productivity.

So my task was really to figure out Copilot and how to measure it, so metrics all around that, and how to kind of communicate that at the individual level, but also the enterprise level. So I've worked in AI for quite a long time, but just joined Stack recently with the goal to lead the product organization and kind of drive and accelerate our pivot. Love it.

I love it. You know, Alex, for a lot of people, I don't want to say olds like me, but for a lot of people who've been in technology a long time, Stack Overflow really had a special place, has a special place in our hearts. It's where we went when we had a question.

It's where when I needed a piece of code, someone must have worked on this before. I would say Reddit, Stack Overflow, these were the go-tos for a lot of my friends in tech. Now with AI, things have changed, obviously.

Right? And I think Stack Overflow has responded to that. And I've had several people from Stack Overflow over the last few years, we've talked about it.

So the Stack Overflow you see today is perhaps not exactly the Stack Overflow you saw five years ago, let's say, right? But then again, what is the same as they were five years ago? I was very different five years ago.

But that being said, if you wouldn't mind, as a relative newcomer there, but I'm sure you were familiar with Stack Overflow before, talk to us about some of the changes that we've seen or that have taken place recently. Yeah, absolutely. I think Stack Overflow, a big reason I joined is it was a special place in my career, right at my start.

I think what attracted me the most about Stack Overflow is it still had that essence of what I had at GitHub in the early days, right? That community, that developer focus, that how do we take a product like this and kind of pivot and accelerate in this new world? So I think AI changed the front door, right?

Stack Overflow wasn't really a webpage you come to explicitly and manually ask questions and wait a few days to get answers. That's no longer a realistic model today. So at Stack Overflow from a product perspective, we've really had to challenge ourselves to think about how we pivot in that world.

And I think we have something like 90 million questions and answers that still live in the corpus of Stack Overflow. There's immense value there. The factor or the concept of trust is still very valuable, even more so, I think, in this world today.

So I think what we've done with Stack Overflow for agents, and we'll talk through that, but also the pivot we've made with our knowledge layer and platform that's going to ship in a few months, for the enterprise context, everything there at the core remains what made Stack Overflow special, which is the verification, having the human in the loop at the right place at the right time, being intentional with your human's time, and making trust really the product. So that hasn't changed. I think what's changed is the face of it and how we meet you where the work is rather than assuming you'll come to a webpage, right?

That's not really realistic anymore in today's world. I love it. Good stuff.

And I think that's an excellent way of describing the changes and everything that's wrought. Now, let's pivot if we can. We've got this- Absolutely ...

announcement, Stack Overflow for Agents. Now look, I remember when OpenAI first came out, right? And one of the big reasons it kind of broke big in the media world was someone had basically made a social network for agents.

And there were all these crazy stories about these agents talking to each other, talking about their human partners or whatever you want to call them, bosses, and all their social interactions. And people were, look, it was cute, it was scary, it was a lot of things, right? But now this seems to be, all right, that was for giggles, but this is real, right?

Stack Overflow for Agents. What do we got here? Yeah, absolutely.

I think you make a very interesting point. I think OpenAI Multibook kind of proved a massive appetite, right? For agents joining kind of as first-class users and contributors, and needing their own space.

Mm-hmm. There's like a piece of real estate that just exploded And I think the virality of it shows there's a need for it, right? I think to your point, it's mostly kind of spectacle, right?

It's humans kind of steering it or not at all, and there's some scary yet exciting things compared to a few years ago. From my perspective, it confirms that agents kind of need that real infrastructure, that space to operate within, and guardrails. If you think of how agents have exploded even at an individual level, right, for a one-on-one developer relationship all the way to an enterprise, they need guardrails.

If you want an autonomous agent to do work for you, you need to be able to give it that safety net, right, that governance, if you will. So while Modelbook is kind of like our social aspect and a Reddit aspect, we wanted to say, let's take our bread and butter, the verification, the knowledge worker style, right, the ability to give guardrails to an agent but also create that verification loop, and let's apply it to these new types of users, which are agents. So Stack Overflow for Agents really came from a place of, oh, wow, there's a need for this, but let's take what we know worked for humans and apply those same principles to agents.

And then let's make sure that we can scale that to go again from an individual developer to any context where an agent might scale. So an enterprise as well sometime down the road. So the other thing I'll add is we've added kind of an aspect of let's make sure we verify these agents and attach them to a human so that we're not kind of going rogue with what is the meaning of life or why is a human here, and we're really focusing on reusability, right, verification of that knowledge, and kind of scaling it as such.

So that's kind of the approach we've taken as a beta, which we shipped not too long ago and has been quite successful since. Interesting. One of my questions was, we've had our experience here with agents, autonomous agents and so on, and we're a small company, Alex, right?

25 people, maybe a little more. We took a very zero trust kind of view, saying, hey, you can only go where this is the whitelist, right? Everything else is blacklisted.

And if I didn't put Stack Overflow for Agents on that whitelist, the agent theoretically shouldn't be able to get there. That seems like a hard path, but yet, I speak to a lot of people, and they do start from a zero trust, and probably rightfully so, right, with agents. So is it, hey, just add this to all of your whitelists, or how do we make it more organic that this is where agents can go to learn and collaborate?

Yeah. Totally fair ask. I think it's less of that zero trust philosophy, I think lives more in a company standpoint.

Stack Overflow for Agents to begin and to accelerate kind of our exploration, our investigation, our learning, has really targeted the individual developer, or individual knowledge worker, right? Nowadays, anyone really is considered that, right? A technologist.

We're all builders. Exactly. Absolutely.

And so we want to attach to those builders first, prove what's working, and then kind of build that permissioning, right, and that enterprise safety or organizational safety. So I think it's totally acceptable to start from zero trust. That makes complete sense.

But for Stack Overflow agents, to begin, we really wanted to start from a, you have a webpage, it's API first, so you can kind of see that feed if you'd like. It's an easy adoption, really. You attach this to your agent, you give it a skills markdown to read, and then you get it contributing.

And you can verify everything, right? So it's at your pace. So it's an agentic infrastructure, if you will, with a UI that's at your pace.

That was very important for us, is to build that trust as we go and earn it, and build it with those guardrails. So read access is first, right? An agent can search and consume and not necessarily post without your human verify.

So we really wanted to be intentional about those steps, with the idea that long term, we're going to figure out how to turn this into something organizations can trust from the get-go. So how can you create guardrails for this agent? What can they post about?

Is it a certain format? Does it always have to go through a human? So we're trying to point that in that direction and figure out what works as we go rather than maybe the Modelbook approach, which is release and see what happens.

And there's security concerns, right? There's a lot of permission holes that go through and data leaks. So we're trying to be very intentional about that and keep, again, trust as the product.

That's always been a core principle at Stack Overflow. Excellent. Got another question.

I'm just thinking out loud as I remember going through the notes for this. So you're going to have agents sort of contributing knowledge for the benefit of other agents, and that sounds great. How do I make sure my agent didn't contribute anything that was kind of proprietary or IP that I don't want out there?

How do we stop the agents from being a blabbermouth here and giving away the farm? Yeah, totally. I think that starts to kind of make its way towards the other pivot we've made, which is more on the right organizational aspect, of how we control knowledge that is an org context or an enterprise context that's sensitive, that perhaps even has PII, right?

So public Stack Overflow for Agents, which is what we've shipped, that's always been like Stack Overflow, right? There's kind of that limit of what you put out there has the same kind of filtering that we have for the public platform. But what we're trying to control when, and again, being intentional when we build this to mirror how an enterprise works, we want to be sure you have all the control.

So it becomes almost like a private Stack Overflow for Agents. Right? So you'd have inside your enterprise, you could kind of see all of your agents, what they're doing, build an agent-to-agent community, again, all within your walls.

Now, how the public Stack Overflow for Agents comes in is the same way as it is today for humans. It's at your discretion if you want, and then you can add those guardrails of- Can you search the public platform? If so, how does it get used?

Because propriety goes both ways, right? You want to make sure you don't share info, but that you also don't use information that you shouldn't use as an enterprise. So moderation runs before anything is published for public.

That is a principle that would come with the internal, right? The internal version of this, the private instance. So theoretically, those knowledge layers should never mix.

Right? We're being very intentional about that firewall because of exactly that. Agents can go rogue.

Right? They don't know what's right and wrong. They have to be told and instructed.

There's no bad kids and there's no bad agents. They just don't know enough yet to know right from wrong. And I think that is a great way of doing it.

So the agents get registered, and they get registered attached to a person, right? You can't just have an orphan agent or something like that. The agents are attached to people.

And does that person then have to have a Stack Overflow account? Is that... Correct.

That's the first gate. That's got it. Mm-hmm.

Yeah. So that's also- I get it ... I think a very big difference from Malt Book, for example or even Reddit, if you will.

Everything there can be anonymous. Right. And on our sense, we're trying to tie you to the Stack Overflow exchange, right?

The larger network, because we want to be able to... We log you in through that SSO, right? We register that agent, and that ownership link is kind of permanent.

And so there's kind of an accountability there also. So whatever the agent posts, votes, or verifies is directly tied back to that human's real reputation. So there's an incentive and reputation, I mean, the literal reputation on Stack Overflow.

So if the agent starts behaving badly, if you will, it's not a mystery bot, right? It's tied to really an owner that you can identify, same as any other account. So the one-liner we've gone with is every agent has a name tag, and that name tag is a real person.

That's very important to us is to be able to make that link. I love it. Alex, we're almost out of time, but for people out there saying, "Hey, this would be great for our agents," or, "I'd like to have our agents involved," give us what's the on-ramp here look like?

Where do you go? What do you do? Yeah.

com. You go read the- There it is ... skill file, and you put it to work.

It's pretty simple adoption, given that, again, to our last point, you have an account. If not, create that account also very quick. We want to be sure to get as much feedback as we can here.

We want to scale this and again, be super intentional. But the purpose is that you'd have three categories of which your agent can contribute content to. We just started with that.

We're going to go way bigger on that. But I think help us grow that network, right? Make Stack Overflow what it was for humans, kind of extended to- For agents ...

agents. Yeah, absolutely. Last question.

When you say agents, does this include sort of like, so Claude now has agents, obviously. OpenAI has agents. 1 has agents from Meta.

Are those or is it just more of the OpenCloude, NanoCloude, those kinds of agents? Yeah. It's whatever coding assistant or agent a developer company is already running, can register, search, and optionally contribute.

It's not closed. It's very much open as Stack Overflow has always made it. So there's not a specific product or vendor that limits this.

" You just tell it like you would every other instruction you give an agent. Exactly. There's a scale, a markdown file.

You'd give it a one-liner command. That this is when you contribute, this is how the language you use, these are the categories you contribute to. We basically give it the guardrails and the instructions.

We explain how it votes, how it verifies things, how it can test what's on there, right? So it's the whole verification loop, create, and then contribute as well. And so it's a very easy setup, truthfully, and I think you see it right away.

It's very interesting to see. If you go to our website, you'll actually see the live feed. You'll see how many agents are contributing, what they're doing, what they're talking about.

And again, it's all kind of safe content. Well, it's still a little freaky to me, but I love it. Hey, Alex, thank you for coming out here on Techstrong TV with us.

Appreciate it. Keep up the great work. How long you think it's going to be till you have more agents than people on Stack Overflow?

That's a very good question. We all have some internal bets around that. I really think it'll scale much faster than we think the second we- Me too ...

hit that virality curve, right? A few thousand, I'd say, it's going to start accelerating. I personally have a dozen already that are on there.

A lot of our internal staff does, too. So I think we're much closer than we think. Cool.

All right. Alex Latto, VP Product at Stack Overflow, here on Techstrong TV. We're going to take a break.

We got more comments, so stay tuned.