Implementing AI-Driven Product Innovations: Strategic Insights and Practical Applications at SKILup Days 2024

Hi, um, hi everyone. Um, my name is Vik Vardas. I am a lead product manager at Groupon.

Um, my education is actually in engineering. Um, I graduated in computer science and I used to be a developer way back in the day. Um, and then I noticed that there's quite a lot of discrepancy between what we are building and why we are building a certain thing.

Um, and that's when I moved into this whole line of product management, uh, in, in India. There was a massive surge of startups around 2011, 2012. Uh, and that's when I'm, I moved into product management.

Now, product management deals with, in, in my humble opinion, it deals with what you're building and why you're building it. It doesn't really dive into how you're building it. Um, but yeah.

So that's a little bit about me. Uh, we'll move on to the next slide, which is we'll just, we just get right into it, right? Some of you might have seen this, some of you might have not seen this.

This is called the hype cycle. This is made by Gartner, and what they try to do is they try to explain the different stages that any emerging technology goes through. The specific one in front of you is for ai.

Now, for ai, if you will have a look, there are five different sections. So there is a trigger. There's, uh, inated expectations, there's a Clough of disillusionment, enlightenment, and then plateau of productivity.

And what that mentions is that anytime a new technology comes through, there's always, uh, an industry trigger for, they either be either mass, uh, adoption because of a, of a certain large company, or it'll be, as an example, Google. Um, or then there will be a new technology that comes out like ai. And that's the one that we're talking about over here.

Then comes the peak of inflated, uh, expectations. People think that this new technology will solve everyone's problems, it'll solve all the problems out there, and that's when we start expecting massive things out of, uh, any new emerging technology. But then as the next phase shows, we very quickly realized that our expectations were two grand for that particular feature, for, sorry, for that particular technology.

And that's when we start seeing that the benefits that we expected from that technology aren't really meeting, uh, what our expectations. Then we start understanding from our learnings from our deployments previously, and then we said, this is when our enlightenment begins, where we truly understand what the technology is going to be able to provide us. And then finally, we come into the plateau of productivity, and that's when it's, it's the most stable.

Why this is important is because as of right now, AI is generating a massive amount of hype. There's a lot of adoption, and people really want to get on the train and say, AI can solve this for us. Well, that's great and, and that's amazing for adoption.

We also need to have something that will be coming a lot in this, in this presentation is cautious optimism. We need to understand why exactly we need to say, AI can help us do this thing, but we need to know to what extent it can help us do that thing. And we will get more into it in the next couple of sites.

Talking about, as a product manager, our, our main thing that we work on on a day-to-day basis and on on a quarterly basis, is something called a roadmap. A roadmap defines what a product will do over a larger period of time. We are not, a roadmap doesn't only talk about, let's say, an agile practices sprints, um, but it talks about quarters, uh, in advance.

So here are some of the steps that you might take in building a a particular roadmap. So one thing that I always say when it comes to product management is you need to understand what the pain points are that you're trying to solve. Again, harking back to the first slide when I said, what are you building and why you building it?

So we're talking about the why. You need to understand that there are some business needs and there are some customer needs. And if you build a product that sits beautifully in between these, that's gonna be the best product that you've ever made.

So try to understand what your business requirements are, try to understand what your customer pain points are, and once you understand that, you will be able to be in a place where you can start deciding or start, start trying to figure out what exactly are the features that you want to build to solve that, um, that problem both for the customer and for the business, right? Once you've done that, you start looking at how AI can help in those features. Now, why I specifically say how AI can help in those features is because you don't want to start by saying, as unfortunately, quite a lot of companies are doing is they are saying that AI exists.

How can we use AI to solve this problem? Now, you are sure wanting, or you are forcing AI into a solution where it might not be the best thing to do. So in this step, in, in the checklist step, what I would say is you, I you identify potential places where AI can actually help in that whole process.

So you don't say, this is an AI feature, and you say, this is a feature and this is how AI can help. Right? Then you need to, and, and you need multiple ideas.

You can't just say, this is the one solution that can solve everything. You come up with a couple of ideas, and then you see where we can go with that in the next step, which is the rank step. Now, there's multiple frameworks out there that you can use.

The most common, one of them is something called the RICE framework. Uh, rice, uh, is an acronym. It stands for, uh, reach, excuse me, impact confidence and effort.

So the reach is how many people are gonna be affected by this impact is gonna be what's gonna be the metric impact of this particular feature. Confidence is how confident are we in that impact being reached? And effort can be either engineering days or it can be the amount of money that goes into that building, that particular project, right?

So now you've got a list of all the features that you can possibly build to solve a, a, a, a, a customer problem or a business problem. So you start taking stuff off the top, you start prototyping everything that you can. So if you are in B2C, you might, uh, be able to build a prototype in something like Figma, and you might say, okay, let's do a design review and let's do a customer in interviews and see how they interact with your, with your feature.

But if you are in B2B, or if you're building internal services, or if you're building platform services, it's a bit more different where you do have to go and work closely with architects. You have to work with engineering leads. You have to understand exactly what the, the pipeline might look like, and it's a bit harder to build prototypes there, but you can still have, but a flow of what each stage would look like and what the, the outcome will look like at the end of it.

Then you move on to the coding step. This is your standard agile practice. You know, you'll have your sprints, you'll have your stories, you go through them one by one, and, and you get the, the feature developed.

The last one, um, on, on this particular screen is launch. And it's, it's a bit misleading. It's, it's not the end of your, of your product roadmap, because on top, you'll see I've mentioned measure loan irate.

So when your feature goes live into production, you need to understand how it is being used by your customers, or either internal customers. External customers could be either one, but you need to understand whether it's hitting those metrics that you had defined earlier. So when I say metrics, let's say as an example, you might have a metric, which is your engagement metric.

You want to make sure that a product feature that you have launched is being engaged with the way that you expect it to. If it's not, then you'll learn from it and you understand why exactly it's not hitting those metrics, and then you irate on that, right? This is broad strokes what product management is.

Um, right? For a moment, we will pause a video and we will go back into ai. The, the problem with ai, and most widespread among this is because of nelms, because of the popularity of open AI chat, GPT or generally, uh, from, uh, from Alphabet or Google.

Uh, and there's another one called cloud. Why, why these are, uh, are super popular in, uh, in, in regular use is because it makes it very easy to get an output. And what I mean by that is everyone treats it like a black box, right here is a black box, which is open AI chat, JPT, we will give it some input and we expect the right output out of it, uh, which is fine, uh, and that, that's the holy grail, right?

Like, you don't need to understand what exactly the system does as long as it gives you the output. But the problem here is that because AI tries to understand the context of the input, no matter what, you know, garbage in, garbage out, no matter what garbage you give it, it's not always garbage out with ai. So you might get something that's believable that's, that's legible, and you might say, yeah, this AI feature is impact working, and this is something that we have to be fairly capable of.

And again, I will go back to that saying of cautious optimism. You need to deep dive into the use cases. You need to understand what exactly is going on with the output and why exactly it was generated.

And you always should test with bad input to see what the output's gonna look like. And the last point I would say, is always have a feedback loop. Now, what I mean by feedback loop is not, you don't just say that, look, we will have a, a, a questionnaire at the, at the end of your product Qs, and we will store that, and, you know, for the next citation, we will, we will use that feedback.

I mean, yes, do that, but if you can use that in the process itself, that's fantastic. So as an example, let's say you are using an LLM to generate text for some website, and a customer generates text and they don't like it. They should have a feedback form right there.

They should say, okay, the text is too. Um, let's say it's, it's too funny. I don't want it to be funny.

I want it to be a bit more business-like, and when they give that feedback, you should be able to generate the content all over again so that loop is maintained without the customer dropping off. Uh, and you can also use it for the owning, uh, at later stages, right? So let, let's talk about three main things.

Uh, when it comes to ai, uh, again, in, in my humble opinion, uh, the first thing that we are gonna look at is bias. The second explainability and the third evolution. And the reason why I talk about these three is because in, in your day-to-day, uh, workings with ai, unless you want to go into it and understand how AI works, if you just want to utilize ai, these are the three things that I think that you should be looking at.

Now, bias AI is just a machine learning model, which has trained on a lot of data. Um, you need to understand that there might be biases in that data, and you need to understand how you can afford to avoid those biases. So I'll give you an example.

I work at Groupon. Now in Groupon, we've got a, so just to give you a background, Groupon, Groupon sells deals online. So if you want to go for dinner, you can search on Groupon, get a discount, go for dinner.

If you want to go rock climbing, same stuff. Search for rock climbing, get a discount, go for rock climbing. Now, let's say our data says we, we just get all the deals on Groupon, and we drive feeding it into an ml, and we say, okay, find out what's the best deal that Groupon says.

Now we find out that the best deal that Groupon sells is, let's say, a spa deal, right? That might be a bias, because if there is a majority of, sorry, if there are a majority of deals on Groupon that are spa deals, spa obviously will be the one that gets us the most amount of revenue. So you need to dive a bit deeper to understand if there's a bias in the data that you're starting with to train your AI models.

The next one is explainability. Now, like I said, you know, most people try to treat AI as a black box, which is fine, as long as it gets you the output, you should be okay with it. But you need to be able to understand how it gets that output a little bit.

Now, what I mean by explainability is that you give it some input, it gives you some output, and it's, like I said, it's, it's completely unknown in the middle. And that's why as part of explainability, I think the atomicity of the task that you ask of AI is quite important. Again, I'll give you an example.

The the same thing as as group on, when you look at a particular deal, you don't just say, all right, make this deal better. You say, okay, for a particular deal, any product that you see on a marketplace, it'll have a title. It'll have images, it'll have a price, it'll have content plug up.

So you break it down into these small chunks, and then you have individual outputs for each of your sections, right? So you have an individual output for your titles. You have an individual output for your images, for your text, for your prices, for your options.

Then when you build a deal with this, you know exactly why that deal was built, because you know what the building blocks for those deals were. But if you just were to say will be the best deal that you possibly can using any ai, it'll just give you an output and you have no idea why that output was was generated. Uh, lastly, we'll talk about evolution.

5 turbo. And we were all in love with it. We were like, oh, this is beautiful.

Fast forward to today, uh, the most powerful, uh, model that they've got is GPT-4 oh, and the difference between these is massive. 5 turbo versus four, oh, the output's gonna be vastly different. Now, I'm not, I'm not gonna say which one's better, which one's worse.

I'm just saying it's gonna be vastly different. And the fact that it's different means that every time there's an evolution in that technology, you need to make sure that it works the way you expect it to work in your pipeline. Because if it doesn't, then you're just gonna start having breakages, and then your explainability becomes a lot harder to come across, and then that will actually exacerbate your biases as well.

So I think these three in your day-to-day, are very important to look at when it comes to, uh, to AI implementation in your product features. Alright, enough doom and gloom. We, we've spoken enough about how you have to be cautious.

Let's talk about some optimism. So again, I explained Groupon is a website that sells deals, right? Um, we wanted to understand how we could use ai, uh, for a particular problem that we had.

So the particular problem that we had is a certain percentage of sellers, we call them merchants at Groupon. So certain, uh, percentage of merchants that were coming to Groupon were not selling anything in the first 30 days that they were on the platform. Groupon, primarily because it's a discount platform, primarily is, uh, a marketing tool for these merchants because they come outta Groupon.

They say, here's my service. I'm gonna give a massive discount so that I get new customers coming in through my doors. Okay?

So that's the background of what Groupon does. That's certain percentage of merchants who wasn't selling anything. They had a terrible experience because they spent all that time getting onto Groupon.

They did all the work, they, you know, bought the services, they uploaded images, they did everything, and they made no sale. That's bad for Groupon as well, because there's a certain process that goes along with onboarding a deal, which is dollar spent. So we wanted to improve that, and we wanted to optimize that.

Now, again, the example that I gave in the last slide, we want to break it down to understand what exactly is quote unquote wrong with that particular deal. We try to understand the metrics for that particular deal. So as an example, we try to see whether the impression were bad, we try to see whether the clickthroughs were bad, we try to see whether the conversions were bad.

Uh, these are different points in, in the funnel when a customer is looking at a particular product feature. So with that in mind, we looked at deals and we try to say, here was a list of let's say 10,000, 20,000, 30,000 deals for each of these deals. Understand what is the metric that is failing?

Just as an example, I give an example. If the metric that was failing was, let's say, clickthrough rates, right? So a clickthrough rate would happen when you see, as an example, let's say you see something on Amazon, you would see a listing.

So you would see an image, you would see a title, you would see a price. So if we see that there's a problem with your, your clickthrough rate, that's probably gonna be a problem with one of those three. Then you come further in and if you see that there's a problem with your conversion rate, then that's probably gonna be a problem with your text, with your pricing, right?

So we try to understand what the problem is with a deal first, and then we run it through an AI and we said, okay, find out what's the best solution for that one specific problem. Don't just try to make the deal better. Here's a specific problem with that deal.

Make that a bit better. So if the titles were bad, so, uh, problem with, uh, with your click-through, if your titles were bad, if your images were bad, what we tried to do is we tried to go to the merchants online properties and get new images. We tried to write better titles for those deals.

If there was a problem with, uh, conversions, we started looking at whether the prices of those deals were correct or not. So what we could do is we could look at a deal and we could say, for this particular merchant, so a merchant has a category, they'll either be a high-end merchant, there'll be a low-end merchant, or there'll be a big range merchant. So for a merchant in a particular category, in a particular zip code, what is the, the price that they should be selling that at?

And we are able to then suggest the right price, which is a more competitive price than what the merchant had initially said, um, because this is a live service. So in that way, we were able to increase our conversion rates across multiple problems that we had by 45%. That's a huge, huge improvement.

And the reason why we could do this is because, like I said, this is a black box solution that a merchant doesn't need to care about, that business doesn't need to care about, but you feed it in a deal that is not incredibly good, and then you let it break the problem down into different steps, the output that you get out of it is going to be a deal that is addressing the problem that we had identified with that particular deal. And we can do this at scale. You don't need a human to look at this.

You don't need a human to say, all right, the, the text is bad or the image is bad. Let me find a new image for this deal. Yeah, I can do this automatically for you.

And we will discuss how we do that in a couple of slides in front. Alright, um, moving on. Um, we'll talk about another thing, uh, which is the onboarding of a merchant.

So I don't know if you, if you remember, but quite some time ago when single sign-on became a thing, it, it was a beautiful and such an elegant solution to a problem that both customers and businesses had. Businesses needed to get your email address, customers just needed to log in. The problem for customers was for every different website, they had to remember the login ID and the login password.

Uh, and for businesses, they will, they needed to get the right email address so that they can send them marketing emails or, or verify that it's an actual user. So single Signon was a beautiful solution like that, which shared information the way that it should. We were thinking how we could do that for our sellers, our merchants coming onto the platform.

Again, because these merchants, they have an online presence that we are not tapping into. If you want to start setting something on Groupon or any third party marketplace for that matter, you are expecting the center, the merchant to enter all that information all over again, which already exists somewhere on the internet. Why do that?

Why not automate the process for them? So what we did is we utilized something called appify, which is ABO Learning Platform, uh, RPAs, we will discuss that again. And we started using open AI to go scrape through all the information that we've got for the merchant online and then bring that onto Groupon.

And then the good part about this was that the merchant didn't have to figure out what's the right thing to use in Groupon. So as an example, what I mean by that is the merchant didn't have to worry about what's the right image to use on Groupon, because Groupon knows that images that are, let's say slightly more orange or that they have glass in the background, they sell better. I'm just saying.

So the AI was able to pick the right images based on our understanding of what sells and on, on what gets most clicks, and then we pick those images for the merchant. So what this did is two things. Uh, the one that I've mentioned over here is that we reduced onboarding time by over 70%.

This was a secondary metric. The primary metric is still from the last slide that we spoke about where some merchants didn't sell, some percentage of merchants didn't sell anything for the first 30 days. This would solve that problem as well.

If we go and scrape that information from the merchant's websites or from the merchant's social media, and we bring them onto to Groupon, and we use Groupon's understanding of what sells because we've got those metrics with us, we can put forth the best foot for the merchant that the merchant might not be able to do on their own. And I think that's why this, this feature was, was quite successful. Uh, we are currently onboarding 10,000 merchants, more than 10,000 merchants per quarter.

Um, and this has been running for the last four quarter now, uh, in Q4. Uh, the next slide talks about the tools and technologies. Now, I'm sure that a lot of you will be quite familiar with this, but, uh, it has to be complete.

Um, we, we use things to, to make our life easier. appify is one of these, or one of these tools. So appify will be a, a tool that has RPAs or robotic process automations where you can automate your tasks on that platform.

So if you want to scrape something that's cool, go to appify, um, find a bot that does exactly that you or that you wanna do. And, um, that task is taken care of. So let's say for, for us, it was scraping and appify took care of that.

Um, we also obviously employed large language models. We, we mostly used OpenAI, but if you are working with GCP, our, then you might as well use Gemini because it's the same integration. It makes things a lot easier.

What I would also like to talk about is how we utilize no-code and low-code solutions to get out MVPs and the prototypes that we spoke about really quickly. Because the only way you can understand whether a feature or a product feature is gonna work is to test it. Uh, and if you wait for a full development cycle, you are waiting for, you know, 4, 5, 6 sprint or you're waiting for a whole quarter to get the product feature out.

It doesn't leave you with enough time, um, to be able to iterate really quickly and, uh, and stay with the market trends. That's why we use something like Make or Any Den make is pretty much a no-code solution where you've got easy building blocks and you can build a, a pipeline. Any d is a bit more involved where you can write some amount of code, uh, to make your product features work.

Um, what I would, uh, go to is the last one. So when I said RPAs, robotic process automations is fantastic. One of the big wins in 24, I believe was when a company, I would name them, when a company was able to reduce their customer support, uh, cost by utilizing AI for it.

Uh, and then the next thing that I would talk about is start looking at explainable ai. Uh, again, the reason why I would say looking to explainable AI is because we do need to understand why a certain output is generated if we want to evolve that output as we go forward, right? Uh, we'll wrap up with, uh, talking about the stuff that we've talked about.

Um, the first obviously is gonna be cautious optimism that has been across this entire talk. Um, yes, AI can help us in amazing ways that we haven't even thought of, but we need to be cautious about the fact that for a lot of cases we don't understand why the output was created or that our expectations are too high. Um, from that particular, uh, technology, we need to understand how AI will align with business requirements and with customer requirements.

If you just build a, let, let's say you build the best AI tool, but if it is never used by your customers or by business, it's worthless, right? So we need to make sure that the alignment exists and those alignments can be done by either making sure all stakeholders are calls you democratize information in whichever form, confluence server notion, whatever. Um, the next one is gonna be measurable outcomes.

You need to make sure that you are measuring what the outcomes are gonna be like. So just as an example, if you, if you look at open AI as a black box, you are going to the only things that you can understand if as, as a response to an API call is going to be the number of tokens that generated the cost, that's pretty much all you're getting outta it, right? And that's not gonna be good enough.

That's what you need to understand that the metrics that you set for any feature need to be in line, again, with the business objectives and the customer pain points that we spoke about. So, uh, just as an example, we, we spoke about how we made deals a bit better. We want to say that if we try to improve the engagement, we are looking at the engagement metrics, but we also have check metrics alongside that.

So what check metrics do is we make sure that while it there you might see an improvement in one dimension. There is not a fall in, uh, another dimension, and that's what check metrics are for. So make sure that you have measurements that you want to make, and then all those measurements are actually measurable in the feature that you're building.

Uh, the next one that I would like, uh, the, the second last thing that I would like to talk about is gonna be cross cross-functional teams. A lot of, a lot of teams now are including data scientists and ML engineers, especially when ai, uh, features are involved. But you also might want to look at, and that that's quite obvious, but you also might want to look at things like designers who focus on, on ai, because some industries like marketplace industries are kind of okay with saying AI was behind this feature.

But there are some industries that are not okay with that as an example of finance. So if you're in finance, you don't, most, most companies don't want to say that this has been done by ai. People are quite uncomfortable with that still.

So build a very strong cross functioning team. You need to have your data scientists, you need to have your engineers, you need to have marketing, you need to have sales, and you need to have design. Make sure you get all of them in the same room and then you're working towards the same common goal.

Uh, and the last one was, is gonna be, it sounds cliched, uh, but continuous learning and adaptation. Um, that cycle of you launching something, learning from it and then making improvements to it is, is very important, especially in the age of ai because AI is moving quite fast now. And for us to be able to keep abreast with those changes, we need to adhere to this, this constant learning and adaptation cycle, uh, that's required.

Alright, with that, I think, uh, I will leave it to q and a. Um, uh, please pop in your questions. Thank you.