How AI Agents Navigate API Barriers with Reed McGinley-Stempel

Hello and welcome to the latest edition of the Techstrong AI video series. I'm your host, Mike Vizard. Today we're with Reid Reed McGinley-Stempel, who's the CEO for Stitch.

And we're gonna be talking about, well, how AI and AI agents in particular might be bypassing some of the controls we put in place to limit their access to our data and applications. And that has all kinds of fun implications going forward. We welcome the show.

Thanks for having me on. Excited to chat about it. A lot of folks think that, well, all they gotta do is tell the AI not to scrape their data and somehow or other good things will happen.

But, um, apparently there are ways to bypass this, and I'm not quite clear how much of it is happening, but you're closer to it than I am. So walk us through what's going on here. Yeah, definitely.

There, there's kind of two different ways to think about this. Um, one is scraping for the sole purpose of like LOM education training, et cetera. So that's like the LLMs, uh, and like the AI foundational companies doing the scraping themselves.

The other side is agents, uh, which I'll touch on in a second, which is more, uh, directed by the end user using the LLM with some tools or some functionality like browsing built on top of it. So maybe I'll, I'll start on the former first. 'cause that's probably what's more readily known today though.

The other one is becoming, uh, quickly increasing in terms of regularity. Um, on the LLM side, as everyone knows, like LMS are only as good as, uh, how, how much data, uh, they're trained on. And so the way that a lot of companies figured that, um, this training would work is that you could just update your robots text, uh, file on your website, tell them you don't want your website to be scraped and that you'd be fine.

And what we're seeing, there's actually a big report on it from some FOSS infrastructure companies running into, uh, high bills being caused by scraping from ai, uh, from LLM, uh, research, uh, foundational companies. And the issue is that not all of them are respecting, uh, kind of the websites that are making it clear that they don't want their data scraped. And so that's one issue that's kind of been happening for the last couple years.

Some of the companies, um, have been particularly aggressive though in, in kind of locking that down, like you might have noticed now, when you open up a tweet, uh, and you try to, you know, and you're not logged in, it makes you log in every time you wanna look at your Twitter data or Twitter feed. Same thing with a lot of Reddit threads where they're pushing you to authentication. The reason being is that those are examples of companies that have a lot of proprietary data that they don't want scraped because they may want to have their own AI fueled product.

Think about grok within ai. They want to have that as kinda like specialized access data and only want data sent to LMS that is being paid for in some other way. So that's kind of the conventional scraping problem that's been around for a few years since LMS have kind of been on the rise.

Um, what you're seeing though is that a lot of those kind of wishes not to be scraped are not being, uh, fully adhered to. And so companies have to think about how do I actually put defenses in place to stop that. The other side though, when we kind of extended a bit to agents is that now LLMs are starting to get better tooling built on top of them, such as like computer using agents, or if you've seen things like OpenAI operator where now I effectively am telling the LLM to go do something on my behalf, like maybe it's even going and logging into my LinkedIn, or maybe it's going to Hacker News or some other news site and compiling information for me or doing research for me.

And that's an example of where it's not the, you know, foundational like AI companies that are doing the scraping themselves. Um, even if they're part of it, it's really that there's an end user that is trying to accomplish something, whether that may, may be scraping or it may be a legitimate automation use case that they're trying to build on top of it. Where now it's actually the agent that is taking, uh, the user's directed approach to go get this data, go log in and take this action on this site.

And overall, I think there's kind of two ways to think about this. One is like, it's pretty awesome, right? It it is amazing that we're starting to get AI with the, with hands, with arms that can actually interact with the world around us.

Um, obviously it's nascent, but that's probably where a lot of the promise comes from. Not just a chat interface, but that AI can actually start doing things for us, automating things. The other side of it though is you can imagine not every website feels comfortable with what's happening here, both from, uh, what data's being scraped, what user actions may be taken on behalf of the agent, um, whether it opens them up to like more fraud use cases where people are using these agents to go create many fake accounts or do a credential stuffing at attack, uh, through headless browsing and things like that.

And then there's also the business element where if you think about it, um, someone like say an open table or resi somewhere where like you, you, you traditionally go to their UI to book a a reservation, those are types of companies that probably fear disintermediation if they are take, if the agents start taking these actions on behalf of users rather than a user coming directly to their site and they don't wanna get aggregated upon. So that's a bit of a kind of overview on on what we're seeing, but happy to dive into anything else. Will this push more folks to put content and data behind some sort of gate that is inaccessible unless you log in as a user and you need to be authenticated and, um, kinda runs counter this original spirit of the web, right?

Definitely. Yeah. com, OpenTable, uh, has done this as well.

Uh, Reddit, if you try to send OpenAI operator to go do something on Reddit, uh, you'll get blocked. Um, because they actually have invested a lot in detecting these types of, uh, kind of virtual machines that are operating on behalf of the AI or the agent. Uh, and so those are some of the main kind of examples of places where I've seen people move very quickly because they stand to lose a lot, uh, if they don't, you know, retain their data in the way that they want.

Uh, but we're also starting to see even smaller companies have to think about this as well. And there's obviously kind of two sides of this. Some, some companies want their data scraped, right?

Like the, it is helpful. I I think about this for Stitch. Um, you know, as search engine optimization increasingly becomes like generative, uh, you know, AI optimization of, if somebody asks about auth, we wanna show up in chat GBT or perplexity or wherever it is, we are actually okay with a lot of parts of our site being, uh, scraped.

And we even wanna make it easier with things like LMS text, like our docs, things like that. Um, but a lot of companies are in another scenario where they may not want that data scraped, or if the, as the agents start to take more, um, concrete actions like actually signing up for an account logging in, they start to see more potential like risk of abuse because there's not the same guardrails that are in place for a human on the other side. So those are, we are certainly seeing a trend where people are starting to lock it down more behind authentication, behind like bot detection, things like that.

So do I need a strategy for that? I mean, who in these organizations is kind of sitting down and figuring out that which we wanna expose to some sort of, uh, AI tool that's rapidly replacing a search engine versus the value data that we don't? It's usually a collaboration between, um, there's usually some guiding light on like the business team of like, what do they want, what are they okay with versus not okay with.

Uh, and that may be coming down from an executive team of we are okay with these parts of our site being indexed and scraped. We actually, you know, are happy to have it done. Or these parts or actions on our site like booking a reservation or booking a calendar or things like that.

Uh, we need to have more, uh, security around. And then for the security side where they wanna have more preventative mechanisms, that's where it does become an engineering question of how do they work with a security engineering team? Or if they just have an application engineering team and a dedicated security team, what security tools do they integrate to make sure that they at least have observability into what traffic is on their site?

Because you used to only care about, um, is this a human or is this a bot? Um, but it's, and you know, there were good bots, but increasingly a lot, a much higher percentage of that bot traffic can actually be split between like good and bad bots. Um, and obviously there are good agent use cases and bad agent use cases, uh, for any particular business depending on how it's being used.

So will there someday be, or maybe there already are lawsuits about all this and you know, the lawyers are gonna make all the money? Uh, great question. Yeah, I think those are already in flight for some of the bigger companies that are.

Um, I, I believe a number of the media, like I feel like New York Times, I think New York Times and a few of, the few other of the major, uh, media providers, uh, in the US at least, and I believe globally as well, have started lawsuits in terms of what the data, uh, in the past has been used for training, which I think has updated how some of these companies like OpenAI and philanthropic approach it. But obviously there are many other players outside of those that are not necessarily adhering to those, you know, court orders or those, uh, you know, threatened lawsuits. Um, but now I do think we're starting to see some more lawsuits crop up where I think X and Twitter had one, uh, within the last year that they filed.

Um, and I, I think you'll see more of those just because they'll both have to think about defense in terms of like, how do we technically stop this? But then also how do we rec recuperate damages for things that have been done in the past that we can't go back and, uh, undo like, you know, training of a certain model. Uh, so I think you'll see a mix of, of both like technical defenses, then legal defenses.

And I think you're right, lawyers will probably make out, uh, in a pretty pretty penny, uh, with this. Some of it comes down to various business models, right? So let's just say I am the Washington Post of the New York Times and I'm pushed more behind a gate, but then I have other people covering the same thing or the same type of content that are not, they might get more eyeballs and traffic for that content pushed to them by people who are, uh, moving away from search to use these AI tools to go look for the latest and greatest thing.

Whether I think we've seen Deep Sea kind of be a little more current than chat GPT, but all these things seem to be getting, uh, faster and smarter about the latest and greatest. No, that's exactly right. That's what we're seeing on our side as well.

Um, and obviously companies will have to make the decision of what's right for them. Are they pro uh, this type of interaction? Are they anti it or do they fall somewhere in between where there are certain areas of their application or their site that they're okay with, uh, AI interaction because they wanna make sure they're keeping up with what user exp the customer experience wants.

Um, and so I think we're seeing, we're seeing kind of like a full spectrum where we have companies that are embracing it and actually want to enable agents on their site. Uh, that's something that stitch that we actually do. We make it easy for, from an authentication standpoint to delegate an agent to take action on behalf of your user account.

Uh, and then on the other side, there are also customers we're working with that are purely in the, um, they want to monitor, detect, and even stop, uh, these agents from interacting with their site. And this is where they'll use other tools that we have, like device fingerprinting, uh, traffic analysis, network analysis, et cetera, to understand is this a real user on a browser and device or is this potentially a computer using agent or is this a headless browser being operated by an agent? Uh, and so we've seen both sides of the spectrum and then folks that are somewhere in the middle where they want to enable certain agent use cases and, you know, allow this agent to do X, Y, Z on my behalf and authorize that, uh, but stop, stop them from changing information like my, uh, payment, uh, details or other things that are sensitive within the application.

So that's more of the kind of like hybrid approach. Um, but we've seen, we've seen companies and customers fall across that entire spectrum. You alluded to this, but I just wanna dive in a little bit deeper.

Historically, um, there was a lot of debate when it came to creating some piece of content. Am I gonna optimize this for um, SEO purposes or a machine can find it, or am I gonna optimize it for a human to find it through something like social media? So now it feels like there's a third leg to this stool and we gotta figure out whether we're gonna optimize for, you know, an SEO machine or a person on social or a AI agent of some type.

Definitely. And I think there's a few different areas to think about here. There's like the general, you know, general content that, uh, companies might publish, whether it's like blog, blog posts, um, thought leadership, et cetera.

And from that perspective, um, being optimized for AI and LLMs is a little bit different than SEO, but not, not incredibly different. The reason being is that a lot of LMS still do use search functionality in order to decide what to surface in the chat interface. And so being still, like still having a good SEO strategy does bake into are we showing up in chat ft responses?

Are we showing up in, um, you know, perplexity responses, et cetera. There's certain companies now that just fully track that for companies like Profound is an example of an early stage company that, you know, in addition to the way people track SEO and what terms you show up for, they now actually run, uh, thousands of example queries on behalf of their customers per day to see what percentage you are showing up versus competitors in a chat LOM interface, uh, from some of the bigger providers. So that's one side that's like general content is slightly different, but it is aligned with an SEO strategy where you wanna rank well for keywords, uh, especially, you know, you wanna be in the top, you know, 10 to 20 results.

Um, on the other side there is the question of like content that is actually, um, either instructional on how to use your product or actually, um, content, you know, in the actual application interface itself that allows you to use the product concretely and that you may have heard of CPS have be become very popular online in recent months. Uh, and recent weeks in particular, MCP is a protocol model, context protocol that was debuted by anthropic, uh, in late 2020, uh, four. And effectively what that allows is a common standard for websites that want their data, uh, like say for example, our docs or API to be easily read by an, uh, LLM and then take action taken on on behalf by like hitting the API.

And so there's actually different ways now that you can just structure the information for the LM but also m uh, MCP servers do allow for, um, better presentation in a way that the LLM understands what to do. You've probably heard of like function calling or tools that can be added to lms. MC ps are kind of the next graduation that we're seeing so far in 2025.

And of course it's one standard, it's possible another standard comes out that gets more popularity, but right now, uh, MCP has really taken off in terms of how you actually structure your data and information for your application so that LMS can use it. What would that structure look like? 'cause back in the day, you know, you had all these people that were engaged in keyword stuffing and yet, you know, the machines have gotten smart enough to recognize the relationship between various terms.

So keyword stuffing is kind of silly and maybe slightly obsolete. So how do I optimize so that, uh, for an AI agent in a way that doesn't wind up, uh, making some piece of content borderline unreadable for a human because it's just, you know, the same keywords shoved up in the top three graphs and, and then metadata that just kind of has, you know, a thousand unrelated things in it. Yeah, definitely.

So, uh, to your point around keyword stuffing, obviously with SEO over the last decades, there's been a lot of hacking that's gone on. Um, fortunately I do think a lot of the major search, uh, providers have pretty good intel into what content is actually valuable to an end user. Obviously they can see how long i, I spend on a page after I search for something, if I quickly back out of it from Google because I find, you know, in four seconds, it was not actually what I was looking for.

All of those things are obviously reinforced into the search algorithm. And so they still do have, um, the right defenses, I think for the most part on how you can stop low quality content, uh, from being surfaced or, or winning in this atmosphere with, I think they have debatably good, uh, policies for is how do you stop that from being AI generated content? But at the end of the day, I think actually a lot of them worry less about whether it's AI generated, if it is actually high quality and helpful to the end user.

The end user spends a minute plus in that blog post 'cause they found it answered the question that they were searching for, et cetera. And so even with, um, you know, all of the hacks we saw with SEO, I think a lot of the defenses that have already been built into stopping the keyword stuffing and things like that will continue to be the case. Now, it doesn't mean that you can't have like a AI generated content.

Um, I've seen some companies that have generated AI content that's actually quite good. Uh, and then obviously they've editorialized it heavily, uh, and they've had success from an SEO or generative output perspective. Um, but I, I think overall, fortunately, a lot of the guardrails that are already in place from a search perspective will aid in terms of stopping too much, uh, slop from effectively being, uh, prioritized and sent to the top of search results.

So what's your best advice to folks then, as we kinda look at this brave new world here about how to think about their data, the content, the way they're presented? Is there, are there a set of best practices emerging? Yeah, definitely.

So the first thing we help customers with when we talk about this, we talk about this concept of are you agent ready? Um, are you prepared, is your application prepared for this rise of agents? And then we, we talk about that spectrum that I was talking about earlier where, where it is a different response depending on the business of what does agent ready mean to you?

Some businesses are on the side of the spectrum where being agent ready means they can observe them, monitor them, and stop them like they're anti these agents. Um, disin remediating their user relationship and that's fine. There's one strategy for that, which is using a lot of these device fingerprinting, anti-bot, um, agent detection tools that we have to make them agent ready in that sense.

There's the other side of the spectrum that I talked about earlier where it's being agent ready actually means these companies don't wanna fall behind. There's almost like a fear of missing out that their competitor will become more accessible to agents in a way that helps end users and the customer journey faster than they will. And so what they're moving towards is how do I actually give a first class agent experience in addition to a first class user experience?

And that's where it comes down to how is your, how is your information structured? How can the LLMs read it? Uh, do you have an MCP server?

Do you allow authorization so that, uh, an LA user's agent can actually take actions, either read or write actions on behalf of that end user and authenticated, but, uh, consented and, uh, auditable way? Um, or are you somewhere in the middle where you wanna make sure some actions are taken and others are not? Obviously that question of are you agent ready has a lot of like loaded context in it around what does that business want to achieve or not achieve When it comes to agents, I would say most companies we're talking to are actually quite welcoming of agents.

I'd say maybe like 70%. Uh, and then 30% are on the more like agent skeptic side. And so we have solutions for both that we recommend, uh, really on the, if you are embracing it, you wanna make sure that you are an OAuth provider.

You can allow an agent, uh, to authorize on behalf of a user. So it comes down to just core OAuth protocols, uh, and authentication on the web. And then if you obviously want to be, if you're an agent skeptic, being agent ready means having the detection tools and preventative tools in place to stop this when you see this traffic on your site.

All right, folks, you heard it here. I suspect the correct answer is once again, all of the above. Hey Reid, thanks for being on the show.

Thanks for having me. All right. Thank you all for watching the latest episode of the Textron AI video series.

You can find this and others on our website. We invite you to check them all out. Until then, we'll see you next time.