AI Agents in Action: Enhancing Social Media Oversight with Chris Taylor

ai video series. I'm your host, Mike Bazar. Today we're with Chris Taylor, who is CEO for fractional ai, and we're talking about an interesting use case involving content moderation and AI agents.

Chris, welcome to show. Thanks for having me. We have seen a lot of controversy over this, especially over the last year, where, uh, social media platforms, uh, for many different reasons or kind of stepping back from moderating content 'cause Well, there's just too much of it.

They're being overwhelmed and they may not have the right people to go look at the right thing at the right time anyway. But you guys are working with some organizations who are applying AI agents to that challenge. So walk us through what's going on here.

Yeah, so ha, happy to, happy to share a little bit about what we're doing. Uh, basic idea is you really want to take the, what the team at, uh, at one of these companies that's doing the content moderation is, is doing and empower them with AI agents to do it more efficiently. Uh, and, and so what that looks like in practice, um, with, uh, with, with a project that we did recently, uh, was to mimic the decision making process that the humans are applying to their content.

So when they review a piece of content and they, they see is it, is it spam, is it following, uh, the, uh, the policies, uh, of, of our platform, uh, you know, mimicking the decision making that they're doing and building that into an automated system. We're using LLMs to make different determinations, and then as it spots problematic content, flagging that content for human review, uh, in, in a way that it allows them to kind of have a, a, a first sanity check on every piece of content, the second that comes up, put it in a queue for humans, and then ultimately still leave the decision making about which content stays up or down in the hands of humans, is the, the approach that we have, we have, uh, been a part of. So how did you go about doing that?

Did you have to take all this content and train the LLM somehow or other, or what was the process for that? Yeah, so the process really involves, uh, in, in day one, what you're doing is trying to get together a source of ground truth, uh, and, and that, that, that's very similar to a lot of, a lot of AI projects. You know, in day one, what you're trying to do is, uh, fully wrap your head around what is the decision making process that that human is doing?

And the first place you start is by seeing, okay, where where has that decision making applied been applied in the past? Um, and you know, in this case it was looking through a lot of, uh, a lot of these petitions that have gone up on, uh, on change dot org's platform, uh, for, for this project and what was the human determination in the past, um, and then mimicking the decision making that went into that behavior, uh, and, and, and building that into, uh, building that into the, the agent that was, uh, going to assist with, uh, with that going forward. Um, so getting to ground truth is really step one.

Uh, once you have, uh, once you have that ground truth dataset, uh, you, you then need to set up evals, uh, to measure how accurately you're doing. So you're running through, uh, you're running your system on top of that dataset, uh, having it make the determinations and, and judgements about, about the content. And then seeing do those judgments match the judgments that came from the humans who did the review.

And where you see large discrepancies, you go back into, uh, uh, your development effort and you, you refine the, the, the system underneath the hood to continually improve the, uh, the, the quality of the determinations that the agent's making. org is the customer that you're working with, but not everybody knows who they are, so bring us up to speed on what they're doing. org is, uh, it's a nonprofit, um, think about it as, uh, uh, Facebook but for social change.

Um, and, uh, and so a lot of the, the content that goes onto their platform, uh, is to raise awareness about different causes, um, and then try to get to the, try to get support. Um, so you, you know, each, uh, each post on their site is a petition and, uh, and then, you know, you're trying to draw up support for your cause on their, on their platform. And, and they get, you know, thousands of new petitions every single day.

Uh, and, uh, and then, you know, the, the, the agents that, uh, that we're building for them are helping them, uh, be more efficient in what they're able to do with that, with that content. And, and one of them was focused on helping their moderation team, um, and we're also building, you know, agents with them to help, uh, help their, their marketing team and, and, and do other work with their org. And unfortunately, we live in an age where you can't have nice things.

So I imagine that attracts a lot of trolls, and that's what they're trying to weed out. There's, I mean, there's quite a bit of content, uh, that that goes out on, uh, on I think every platform, uh, that violates content, policies that is spam, um, that contains, you know, contains things that clearly need to be taken down. Uh, and, uh, and, and so I think, um, yeah, I think that you're, you're exactly right.

org per se? I, myself, uh, am not an experts in, uh, in what's happening inside of some of the, the larger social media platforms, uh, on, on this topic. Um, but I think the, the promise of the technology to take some of the, the human decision making, uh, and replicate it at a, a very high, uh, degree of accuracy to assist the, the, the, the human team to review things at scale and put it through, uh, whatever process they, they determine as, uh, as the correct one, um, I think is, uh, I think it's a very powerful technology.

I think it has that capability. Now, all models are prone to drift over time. So how do I keep track of the AI agents are doing the right things, not just when I put 'em out there initially, but six months, nine months, a year later.

So every, uh, big part of every single project that, uh, that we build here at, uh, at, at fractional AI is measurement and really understanding, uh, from, uh, from a numbers perspective, uh, you know, how, how well is this performing across every key dimension? And, uh, and, and you know, one, getting comfortable with that before you ship it so that, you know, this is exactly how, how well this thing's performing. This is production grade.

I feel comfort, you know, confident putting this, putting this out into the world. Uh, and, uh, and then, you know, from a long-term, uh, you know, drift perspective, uh, I think, I think in reality, the, the thing we see more often is that over time you see improvements in all of these models. And so you see improvements in both from, from both a performance perspective and from a cost perspective.

Uh, you know, the costs of running these systems drops about 90, 90%, uh, on a every, every 12 months. And so, uh, and so I think you see, you see benefit there. You see, uh, potentially some improvements on performance and the ability to swap in new models and, uh, and update your system with, you know, a couple lines of code, uh, to, to the latest and greatest.

And then you, you, you still rely on that, you know, that, uh, the, the metrics that I mentioned before, when you're making those changes to the system to tell you are you making the changes and, and driving the positive change that you want in the, in the system. Um, but a lot of it comes down to measurements and that that source of ground truth that you establish early on on a project. So ultimately, these systems are not static.

They're gonna be dynamic in the sense that, uh, AI agents may get pointed at different LLMs, the LLMs themselves get upgraded with more reasoning capabilities. Um, do we have the right workflows in place to kind of manage all that Early on in, uh, in using AI or adopting AI agents to do tasks? Uh, it's, it's very important to think about what is the, what is the eventual user experience as you're building one of these things?

Um, and I, I think some of the best practices are you, you know, you don't want to just build things behind the chat bot where you need to drive usage in certain ways, and you're leaving, you know, a a lot up to, to the end user of this thing to, to figure out how to do it. And, uh, and, and you're relying on like heavy handed change management policies. And, um, you know, you, you really, the, the best practice here is to build the functionality of the agent in a way where you can expose it in the flow of work for, uh, for, for the, the ultimate user, uh, in an intuitive way, uh, such that the new default path becomes the agent is adding value and, and fitting seamlessly into, into your workflow.

And, uh, and I think, you know, that that takes a different shape for different types of agents that, that you may be building, but, uh, but, but ultimately, um, the most powerful AI functionality that I've seen is the, the, the functionality that is built into the core experience in a way where it just fits naturally. Um, you don't need, you don't need some lengthy training, you don't need a lot of time with it to figure out what it's good and bad at. It just works and it works the first time you do it, and it, it can create some pretty magical experiences.

And then a world of software development, we have this notion of observability where we look at the systems and the interactions is that same concept need to be applied to AI agents, which are in turn interacting with all kinds of other AI agents. So do we need some way to kinda keep track of what's going on with all these? Yes, I think beyond the, uh, the, the going beyond the type of performance metrics I, I've been talking about and, and going beyond evals, um, you know, I think with, with agents you can see the various steps that go into a process.

So, you know, the way, the way I think broadly the definition of AI agent is something that's capable of performing a task for you. And when that task becomes very complex, there's a lot of subtasks that it's performing under the hood. And so the ability to see, you know, take a peek behind the curtain and when it gets the answer wrong, look at, okay, well what are all of the different decisions that it made along the way?

How did it arrive at this, this conclusion is, is is a necessary thing to be able to see going, going further than that? I think, uh, sometimes where, where I, I've, I've heard conversation, uh, I think go into territory that's, that's tougher is, uh, when you're replicating human decision making, sometimes, uh, I think there's a, there's a propensity to hold the AI agents to a higher standard than you might hold a, a human being. And when it comes to observability, you know, you can, you can imagine if, if there's a, a decision that was made and you ask, you're trying to ask like a question of like, well, why did you make that decision versus that decision?

Like, sometimes you just get to the level of like, that's just what happened. That's just what this human, human did. And there's no real answer to that question.

That's just like, you know, you, you can't like take their brain and like, put it under a an MRI machine and get some magic answer. And I think sometimes that that is the, the same thing with, with these machines. 'cause they are able to replicate that human decision making.

Um, but, uh, but to a large degree, uh, what we've seen a lot of success with is what I was talking about before. If you break that task into all of the different sub tasks, you can see it every step what it did and measure the performance of what it did. Um, I think when you start to get into the, well, why did it do that?

And, uh, and, and the parallels to like you, you ask why a human did what they did. Um, I think think, uh, I think not expecting a different level of answer when it's a machine versus a human is a, is something to keep in mind Through that. I mean, it almost seems like to me that, you know, in humans they are subject to emotion and, uh, some days the decision is based on insight and other days it might be indigestion and maybe AI agents have, um, you know, similar variables to consider.

Yeah, I don't know what the AI agent equivalent is of indigestion, but, uh, but yeah, there's probably something there. So what's your best advice to folks then about how to think about managing all this? I think people are starting to realize that there might be, uh, hundreds of agents, there might be super agents that are managing other agents and, um, it's getting highly dynamic.

But, um, how do I wrap my head around all this? Yeah, I, I think, uh, speculating about five, 10 years out and thinking about, you know, what, what is possible? I think some of these frameworks are intellectually interesting, but I think in 2025, uh, I think you can safely think about these things as very powerful software and no different than, you know, different in the fact that they can do new things now.

Um, but the way that you're, you know, you're not hiring an employee and managing an employee here that is some, you know, floating head avatar. Uh, you know, it, it's, I think that framework is, it can be confusing. And in reality what this looks like is this is high performing software that can handle tasks for you.

And, uh, and then it's, it's sometimes a little harder to measure than classic software because, uh, because LLMs have, uh, have that, you know, u unique ability to apply judgments and judgment can be squishy and, and hard to, to put into numbers. Um, but I think thinking about these agentic systems as software that can perform tasks, uh, and uh, and, and that, that is the, the right framing. And I think the more and more tasks you can identify where this software can do it, uh, you can, you can really make a massive impact on your business.

Um, but uh, but, but fundamentally I think it's helpful to think about this stuff as super powerful software. All right, folks, you heard it here. We're kind of having a variation of the pets versus cattle debate we had about, um, software artifacts in the first place and processes.

And we need to understand that, um, pets are things with names on it and cattle might be things that we're just kind of keeping around for various purposes. But, um, don't get too attached to your agents. Hey Chris, thanks for being on the show.

Thanks a much driving me. All right. And thank you all for watching the latest episode of the Textron AI video series.

You can watch this episode and others on our website. Until then, we'll see you next time.