Navigating the Impact of AI on Web Traffic and Content with Bharat Guruprakash

Hey guys, thanks for the throw. We are here with Barat Guru Pish, who's chief product officer for Algolia, and we're talking about ai, web traffic bots, and all kinds of interesting things that are happening these days, both for good and Ill Barat, welcome to the show. Thank you very much, Mike.

Uh, thanks for having me. Uh, excited to he, uh, be here, have this conversation. The world is changing, uh, rapidly.

I think anybody who runs a website has noticed that there is traffic coming from bots that are essentially crawling their content and then using that content to, uh, we assume train AI models. The issue it seems to be though, is that the traffic they generate isn't necessarily something I want to monetize. It's not a human, it's not something that I'm gonna get paid to go do, and yet I am absorbing the cost of processing all that stuff.

So what's to be done about all this? And how much of this is just the new cost of doing business? Or is there something I can do that maybe will minimize the impact?

Actually, you know, there's, it's worse than just the cost of processing all this bot traffic. Um, you know, if you take a step back, whether you're an e-commerce or media or whichever, you know, type of website that you're running, there is a supply and demand component to these things. You as a media or an e-commerce company, you have a supply your product catalogs or your, you know, music podcasts and streams and all of that.

And you have a commercial strategy that goes with that. Like how you distribute it, who gets access to it, ads are involved, et cetera. There's a lot of things, engagement, customer experience.

There's a lot of thought that goes behind the supply side of this. You know, if you are Spotify or if you're Netflix, or if you are an e-commerce company, like, uh, name any big e-commerce company, Amazon, whoever it is, right? There's a lot that goes behind there.

Then there's the demand side that is coming in from these lms. You get demand from Google search engines. And more and more we're seeing bots now coming in.

And part of this is, it's, this is this complex equation because what is happening, I think you recently saw Amazon banning the perplexity stealth bots that was scraping all of their sites. It's because of this complex supply demand commercial, uh, strategies that these companies are thinking about on a daily basis. And on the other hand, you have these stealth bots that are coming in trying to scrape the information so that it can then be discoverable in a LLM for example, when you're looking for, Hey, what's the best sort of like ski gear that I can wear as a beginner going to Tahoe, for example.

So, you know, I think, I think it is not just about the cost of processing, it's actually the commercial sort of like strategies that these companies now have to think about when it comes to distribution and, and discoverability of their, uh, of their products, whether it is media, e-comm, whatever is the industry. So I think, I think it's, it's, it's extremely complex and this is why I think Amazon banned them. Mm-hmm.

Yet Amazon may wanna do that, but for a lot of organizations who have invested heavily in things like search engine optimization, there's a lot of the new search tools are AI driven, and a lot of people are starting to use those things. So will I just be cutting off my nose to spite my face if I block that traffic? And, and how do I strike that bounce?

No, it's a great question. And every company has to have a, um, a sophisticated distribution strategy. There is promiscuous distribution where they want sort of like more of their commodity items to be widely available to as many people as possible.

And then there is there actual premium stuff and what they want to control the experience on. So if you take a look in the e-comm world as an example, you might have, uh, like a Wilson's, uh, tennis ball or baseball, so whatever, these are commodity items, you want them to be available in as many different channels as possible, easily discoverable, et cetera. And then you have at the other end, if you go right to the other end is like super luxury items, like the air messes of the world.

Like they're not going to take a wide distribution strategy. And then you've got marketplaces and, and, and people who sort of like aggregate both premium products and commodity products. Like they want to have a mixed distribution strategy.

So I don't think it's as simple as I'm just gonna block them because I don't like them. I think companies are evolving to figure out what do I want to make available to a, you know, promiscuous, uh, channel strategy versus what do I want to control more of? And this is true in e-comm, again, in media, in, uh, any of these other fields where transactions are happening online.

Is that a quote unquote static process or is it more likely to be a lot more dynamic? 'cause I might make one decision today and another one tomorrow, a hundred percent dynamic, which is why at the very least, if you are the web, uh, you are owning that particular application or that web property, you at least wanna have the control to make those decisions. Um, today, like if you do these, these stealth bots, like that decision is being taken out of your hands.

And so, uh, this is where, you know, companies like Algolia come in because we can actually, you know, help customers control what type of products or what type of items, what shows up where, depending on whatever their commercial strategy looks like. Mm-hmm. Well elaborate a little bit on that.

How does that work exactly? Do I put what in where on my website or somewhere else? I mean, it could be, uh, your website, right?

Like let's, let's take for example, um, let's say Chad, GPT or perplexity or any of these LLMs wanna get access to. And let's say you are a, um, you're a media company. Let's take a media company as an example.

Um, let's take someone like, um, like an iHeartRadio for example. You know, they have, they have very, um, um, uh, media properties that they want it to be everywhere, like top 40 platelets, uh, hourly traffic bulletins, like these are all commodity type of media that you just want everywhere as possible. Then they've got their, because they're also a creator like Netflix, they have their own podcast, they have their, you know, investigative journalism, et cetera, and you want a slightly more, uh, narrow distribution strategy.

So Aldo powers all of that as an example. And if perplexity is questioning, uh, iHeart, hey, what is the latest sort of like podcast that, you know, this user is asking for? Um, with Algolia we can answer back with iHeart's control.

What is the appropriate things to send back to perplexity based on whatever it is that iHeart strategy looks like? So that's sort of like how it works. And you know, you've got technologies like MCP today that allow for that sort of like querying and that interaction to happen.

Mm-hmm. How much visibility and foresight can I get into that? Because, um, I may wanna know that a certain topic or subject is building with those new, how we say AI browsers, and then I wanna be able to respond and surface that content.

Uh, but I may only wanna do that to my previous point for a week or so. So I mean, how much visibility can I get, um, visibility into, into a control of it? Yeah, but also, but like, what kind of queries are the AI engines starting to make and 'cause that's gonna inform my strategy, right?

A hundred percent. So you take any of our customers, right? Like we, uh, all of these queries come through Algolia.

So we actually know what is being asked and we actually surface that back to the customer, our direct customers, so they actually can see what is happening and they can see what the results look like. And this can see what the interaction and the engagement model looked like based on the results that they gave. And does it line up with their personalization strategy?

Does it line up with their commercial strategy? Does it line up with, uh, their customer experience strategy? So we give all of that back to the customer.

They can see all of this analytics and what used to be a human analyze sort of like recommendation. That's what most companies used to do, is to look at this, put them in these gigantic spreadsheets and like compare them and say, Hey, you know what? We should probably modify our, we're, you know, uh, uh, returning results back in this way.

We are changing that because today agents can actually do that and they can do that at a faster clip, at a more dynamic pace. And as a result of that, um, we're entering with this world of like, what I call hyper hyper-personalization, uh, because we're at machine scale now. And so it's very exciting, um, in, in that sense because you know, who wants generic answers, right?

At the end of the day, you want it to be, you want it to be as personalized and relevant to you. How quickly is all this happening? Uh, 'cause some people are still using traditional, say Google search, other folks are relying more on, um, AI and it's not quite clear which one of those is more reliable or useful.

'cause you could probably argue both ends of that equation. But what's the balance and mix that you're seeing out there? We have, if you think of it from the innovator's curve, you know, curve, I think we are at the phase where we're still on the left side of that, of that equation.

But I would say that, uh, so we have a lot of like challenger customers who are ready to like, go to the forefront of it. You know, they're like, we're done with the traditional sort of like interfaces that we had on our website. You know, they're even talking about and pushing us about what is the future world?

Are there going to be websites or is it all going to happen through these like, um, headless browsers who are just communicating with other, you know, headless LLMs and et cetera. So we do have that class, but I would say that, um, they are still in the minority. But what is interesting and what I'm seeing is we have a lot of enterprise customers, by the way, and what I'm seeing is that the rate at which the bigger and the, and the more sort of like the middle category and the later stage categories in that innovator's dilemma curve, how quickly they're actually trying to get on board with all of this.

I think that's accelerating. And I've never seen that before at this rate that it's happening at. And it's, uh, it's, it's interesting.

It's almost like people are ready to skip a generation or two of technologies that they hadn't adopted yet and just go straight to the latest stuff. Hmm. So what ultimately is your best advice for folks as they try to navigate all this?

I mean, is there, uh, something you're starting to see that amounts to a set of best practices that customers are putting in place? How are they navigating all this? Um, I would say that most people are stumbling through it right now.

I think that is the reality. And, um, you know, there is a, I think there's a little bit of a cognitive overload as well going on right now. The number of tools that are available today, the number of LLMs, the differences between all of these, and then how do you apply it into an enterprise setting?

How do you then take it into production? Is it going to work? Is my current stack the way I've built it as, as a company going to support this new stack that is coming in?

So there's a lot going on. What I'm seeing is that most companies, especially the enterprise, they have sort of like, um, shifted their budget from where they were traditionally spending their IT budgets on to all of this stuff. But while they've made the budget available, everyone is still digesting what it means to actually operationalize this.

And so this is where, you know, we are trying to do our part because, you know, we know what it means to operate at scale. We run something like two and a half trillion queries per year, second only to Google. Google runs about 5 trillion, and we are not a five and we're not a public consumer search engine.

So we understand what it means to be a production scale. So we're trying to do our part, um, by abstracting away all of this complexity, this orchestration, this, um, this deployment that people have to do if they want an age agentic experience. And, uh, you know, with the respect to that, last week we actually launched Agent Studio Al Golias Agent Studio, where you can deploy agents that are grounded in your index with any tool that you want, that it wants to call out for any experience that they want to provide, customers wanna provide.

And what we are also seeing is customers experimenting. They don't know what is the right customer experience themselves, because this is happening in real time as well. Like, my own experience of using search engines, and I don't know if yours is similar, Mike, is I found that I am, I'm using Google Less and Chad, GPD way more.

And it's just been an interesting, uh, evolution observing myself. And what I've noticed is the way that I'm asking questions and the way that I'm looking for things, it's also changing the words that I'm using, the, the precision that I'm using, et cetera. So all of this is happening in real time.

So we are trying to make that, um, easier for customers by abstracting that away so that the system can auto configure itself to a new world as it keeps changing. So that's sort of like how we are approaching this, which is why, you know, we released Agent Studio a few months ago. We released our MCP, and so you can like enable these things to happen in real time and not have to go in and like crank the, the, the, the, the screws again.

Uh, 'cause it's complex. Yeah, it's challenging. 'cause on the one hand, uh, even with Google, you're seeing AI results pushed to the top of that page, but they're a little more generic.

And the more precise answer might be from the original search, but that's now 10 items below the fold because I got 47 ads that I gotta go through before I go find that thing. So I'm kinda stuck on either one of these things and I feel like I might not be well served on either end of this thing. But let me ask you this last question.

It's pretty clear that AI agents are emerging and they are similar to bots in the sense that they generate traffic. But, um, do I optimize content for their consumption? Are they a new type of end user per se, or are they some other entity altogether different?

And I need to kind of have a different mindset about this from the get go. Our a GI overloads are coming. Is that what you're saying, Mike?

I don't know. No, I, all I know is they're launching a heck of a lot of queries. Yeah.

So I'll, I'll put it this way, right? Like, and a, uh, um, generative AI was all about creating content and, and you know, putting it in a way that addresses whatever is the query, but it was still content creation at the end of the day of some form either summarizing or answering a question, whatever agents are, even though they rely on the underlying technology of foundational models, agents' role is to execute tasks. They are there to take content, take language, take behavior, and go execute something.

So as an example, we actually worked with one of our customers, which is a cruise lying company. They wanted to create a travel booking agent. So if you came in to their app or their website and said, Hey, I want to go from Seattle to Alaska, uh, my family and I, when is the best time to go and what are my options?

Gimme some ideas and can you book it for me? So the agents sort of like handles all of it at that point. Now, um, when it, uh, of course when they're trying, when these agents are trying to execute this task, they're going to be querying a lot more because they're querying different data stores, uh, because that's where information is.

The booking information is in say, a Salesforce CRN. The assets are maybe in an A EM, Adobe, a EM. So they're querying all these different systems to get the right context and the right information back so that they can execute it.

Um, so I think it is different, uh, from just what we used to know as bots. I think this is more about these agents ultimately becoming our representative on the internet to execute tasks on our behalf. And the example that I, that I give, which is, which is kind of funny, is, you know, about 20 years ago or so, 20, 30 years ago, we had travel booking agents.

If you remember that world, you would go to a travel agency and say, I need to go to the Maldives or whatever, and they would like, recommend to you, et cetera, what to do, and they'll do the booking. Interestingly, we're going back to that world, but in a machine, uh, led way. So if you have a agent that is representing you now travel companies are gonna have like these agents that are representing them.

And so we're gonna get to a world of agent to agent interaction. Um, and so that's how I see these bots and agents evolving. We're not there yet, because again, like I said, the orchestration layers to do all of this in a seamless, satisfactory way is still being built as we're speaking.

But it's clear that that's the world we are starting to head towards. And, you know, whatever is the consumer endpoint device, whether it's the mobile phone or the new thing that OpenAI will one day announce, um, that's gonna be your interface to communicate with your agent. Who's gonna be your representative.

At least that's the way I see the world evolving. All right. Well folks, Ja heard in here.

To paraphrase Bob Dylan, the times are a changing. So buckle up buttercup. Ding Barrett, thanks for being on the show.

Hey, this was wonderful, Mike. Thank you so much. All right.

And back to you guys in the.