LinkedIn’s Karthik Ramgopal on Enhancing User Experience with AI

Hello and welcome to the latest edition of the Techstrong AI video series. I'm your host, Mike Bazar. Today we're with Karthik Koal, who is a distinguished engineer for LinkedIn.

And we're talking about, well, what exactly is LinkedIn doing with ai and how did that wind up improving the overall user experience? Karthik, welcome to the show. Oh, thank you.

Thank you. I am, I'm really excited to be here. I think a lot of organizations are kind of feeling their way around in terms of how to think about operationalizing ai and we're struggling to figure out what use cases make sense.

I think. So maybe if you could share how LinkedIn thought about this and, and, and how you decided to do what first that would be probably of great interest to a lot of folks. Awesome.

Well, so LinkedIn has been using AI for a long time. Like for example, one of the earliest users of AI in our product was the people you may know, recommendations where we recommend folks you should connect to right long ago. But when this generative AI moment came along about two or so years ago, again, like there was a lot of pressure to launch something with generative AI because everyone was doing it.

But we did not wanna build technology for technology's sake. We really wanted to think a lot about what product could we build with generative AI such that it could positively impact the lives of our members and customers. So we did a lot of internal brainstorming, user experience research studies, looked into uh, areas of high impact and came up with these suite of generative AI based products, which we've launched to our like members and customers.

And the blog talks about a particular category of them, which we've launched for our premium members on the flagship front. Were there things that you could do with generative AI that are different than say, traditional machine learning algorithms and predictive kind of ai? And is that changing the user experience and how do you think that will evolve?

So I don't think that, uh, anything is fundamentally different. It is just that it is a lot cheaper to do so with generative AI because we have a very powerful foundational model which is highly capable of handling a diverse set of tasks as opposed to a traditional approach to machine learning where you would train a model for a specific task with a specific set of imports. So your iteration cycles and the flexibility in product which you get is significantly higher with the generative ai.

Now what you need to be cautious about though, as we talk about in the blog, is that because we are still in the early days of generative AI models, uh, that can also be a false sense of promise or accomplishment with early results, which you see and the quality may not like, like bear out uh, when you actually like feed it in really large sets of data, right? Different experimental sets of data, et cetera. Try to see how the model is doing.

So It Takes quite a bit of time to get quality and the experience like Right. With generative ai, Are you building your own LLMs or are you customizing LLMs the and extending them or for all I know you might be using rag techniques to expose data or all of the above? Uh, all of the above.

But for the context of this blog, it was mostly using, uh, Azure OpenAI L lms and using What do you know now that you kinda wish you knew when you first started experimenting with all this stuff? Hmm. Uh, I think, uh, it is the quality bit which I spoke about earlier, which is that hey, it's really easy to get a prototype built out, but to get quality and the experience right, it's gonna take a long time.

Like lot of tuning on both the system side as well as on the AI side to deliver a good experience. I think right now most people are treating it almost like, uh, an expanded form of search. And yet when I look at the capabilities of the LLMs and the reasoning engines that are starting to emerge in there, how sophisticated do you think all this is gonna get, especially as the parameters get larger?

'cause it seems to me we can do more with prompts and we're at the beginning of this adventure. I would agree. We are at the beginning of this adventure.

I I think that we will soon see LLMs uh, and generative AI in general being used for various forms of like task automation. We will see the emergence of AI agents more advanced cognitive like memory architectures to power these LLMs, et cetera. Do you think we're gonna have maybe one master kind of agent that, you know, manages everything on our behalf or are we going to be trying to orchestrate a lot of very task specific agents?

'cause as I look around the landscape, I see lots of little agents everywhere and then other folks are starting to talk about, you know, one kind of uber agent, for lack of a better phrase. Um, how do you think that'll play out? So it's hard for me to predict, but based on current technology, I think we will have a mixture.

We will have specific agents which are customized for specific like tasks, and you'll have orchestrators amongst these agents. I think to draw a parallel to human life, uh, we have specialists, right? And we also have coordinators and generalists who work with these specialists or even us who in our day-to-day life, like work with various specialists like electricians, plumbers, or you know, other folks in order to get our, our jobs done.

It's gonna be very similar. Do you think we'll be able to reach beyond the, the actual sites themselves? And I'm saying that in this context, right?

Whether it's LinkedIn or anything else, I have to kind of show up and log in and see what's going on and I have to be there, but I should have maybe something that is my agent or avatar or whatever it may be that represents me that's kind of continuously engaged and then lets me know what's going on that's interesting to me and pulls me back in. Versus today I feel like, you know, short of an email, the only way I go back to that site is if, you know, somebody emails me a message or LinkedIn or I have a very specific problem. So I think, uh, again, hard to predict because right now having uh, an avatar of you engaged on your behalf is very computationally expensive, right?

And also it's really hard to get relevance, right? Because you as an individual may interact with LinkedIn or any product we use for that matter in like various ways. But you know, technology keeps improving, costs keep going down, but there's also the product experience aspect of it, right?

You gotta be very careful about what you do with the avatar as opposed to you because ultimately on a professional social networking platform like LinkedIn, we want real humans to be having real professional interactions. Uh, and we do not want a lot of AI talking to AI happening Right now. I think a lot of folks are trying to figure out whether or not they need to become prompt engineers.

And I wonder though, it seems to me maybe that the, the AI engines themselves will take care of a lot of that for us. So how smart do I have to be about engaging with the machines going forward? I mean I, I think the first rev is I have to have a fair amount of knowledge about how to structure a question, but I think that's getting less and less.

It is getting less and less. Even in the last two years we've seen a lot of improvements and offload. There are a lot of frameworks which offer a ML driven approach to prompt engineering itself.

And I expect this to evolve more and more. And there are also new techniques emerging for better and more predictable structured output generation, which is actually very common when you're working with these LLMs where you do not want freeform output generated, but you want output which is structured, say, representing a certain schema or serialized in a certain fashion, et cetera for programmatic interaction or processing. So both these things are gradually becoming more standardized and systematic.

Now I feel that prompt engineering as a skill is still gonna be there because someone still got to tell the LLM what to do, right in some shape or form, but how it looks in a few years, maybe very different. So my advice would be get started, keep up with the change and gradually adapt as the patterns change. How has any of this changed the way that your internal IT teams are structured?

And I'm asking the question because a lot of folks have data science teams over here and then there's an ML ops team and then there's a DevOps team and there's a bunch of governance and security people running around throwing a bunch of data engineers. It's a lot of things to manage to get this all right, it takes a village. Um, is there another way to think about the way we should structure our teams to really optimize for ai?

So I think that uh, as I said before, LinkedIn has been using AI for a long time and of course this generative AI revolution has certainly spread the net wider because it has like democratized access to ai. For example, a lot of our prompt engineering right now is done by apps engineers instead of AI engineers, right? Because you don't need AI skills for it per se.

So as with everything else, we are sort of continuously looking at the changing landscape and how we structure our teams internally and streamline operations, uh, you know, for the best efficiency possible. If you could fix something about all of ai, I mean we kind of what, what, I guess I'm asking you, what's your pet peeve right now? What's that thing that you just kinda have an itch for you and go boys, girls, I think we could do better than this.

So I think paying a lot more attention to the safety aspects of AI because one of the things about LLMs is that they tend to hallucinate a lot and that can result in inaccurate or harmful content which might cause all sorts of problems. So again, all of us are trying our best, like LinkedIn is really committed to responsible ai. I know a lot of other okay companies in this field as well, but, but you know, for everyone building applications about ais paying a lot more attention to this piece to ensure that we do not have inadvertently harmful effects.

Are you at all worried that we'll have something of a backlash against AI because the hallucinations are too much and people won't take it seriously or they'll be kind of disappointed? I mean, are we gonna have a bit of a trough of disillusionment? So I think if we do not pay enough attention and we do not curtail it enough, we will uh, see this sort of a backlash.

I think you may lose trust in what is being, uh, generated by AI, et cetera. But I also think that the benefits of AI in the way it can transform our lives is so large that all the stakeholders involved will try their best to like minimize these negative effects and ensure that AI becomes a positive, uh, force for humanity. Do we need to educate humanity about what it is exactly that AI does so they understand what it is that they're engaged with?

'cause sometimes to your point, they might put too much faith in the machine, which is, you know, confidently telling them one thing that might not be accurate. Yes, that is a great point. I think that should be a fundamental part of product design.

Even if you look at our own product, we have various sort of disclaimers saying generative AI is experimental, your results may vary, et cetera, because it is, that is the fact, right? So I think putting that sort of product disclaimers is really important. Educating people.

Also, you see a lot of uh, like AI videos, AI audios generated today. Having some sort of like watermark or other ways to differentiate that hey, this is actually artificial and it was created by AI is important to infuse a sense of trust and to avoid cases like impersonation or defrauding or things like that which could happen with such technologies. And I also feel that over a period of time, okay, governments and regulators will also start paying more attention like they already have.

I just expected to increase more and ensure that there is more regulation of how AI is used in general. It seems to me that a lot of folks don't really understand or appreciate the fact that the gen AI models are probabilistic and they keep applying them the use cases that require something that is deterministic in the sense that it's gotta be right a hundred percent of the time and it's gotta be the done the same way a hundred percent of the time. So do we need to just get savvier about our use cases?

We see we need to get savvyer about our use cases and there are also ways to constrain the non of these models to some extent. I spoke about the, uh, structured output generation, which is typically used for programmatic outputs, et cetera. That is one way to not eliminate but significantly constrain what the AI can do.

And there are also various sort of instructions you can give it in prompts, et cetera. But yeah, if you want a hundred percent perfect like determinism, then generative AI is probably probably not your friend and you're like better off writing an algorithm. So one of the techniques I've seen is that you generate code using AI and you just, you know, reduce the grunt work of you writing that code yourself.

But that code once written is deterministic and it does a particular task. You can write various sort of like tests around it similar to the way you do conventional software development and still achieve high levels of automation with a significant amount of determinism. Right?

Not as you remember to review that code before you hit that button. Right? Of course you gotta be really careful in reviewing that code.

Yeah, right. So ultimately, what is your best advice for your fellow engineers out there who are trying to build these models and they're heavily engaged with business users and they're trying to bring in developers and not everybody seems to speak the same language. So how do I, and I'm not talking about, you know, the difference between English and French.

I'm talking about, you know, just the nomenclature they use within their job functions is different. How do I kinda bring all those people together in a way that creates a more cohesive team? I think, uh, it starts with being realistic about the capabilities of ai.

I see both camps, I see people who are overly pessimistic as well as people who are overly optimistic. And usually, you know, the latter camp is a lot louder. You know, in the industry, especially when you look at AI appending all sorts of functions, the reality is somewhere in between.

So being realistic is extremely important. Playing with these models oneself in order to understand their capabilities, continuously looking at open source blog posts, publications, papers, et cetera, for techniques which others are using and trying them out for internal use cases, uh, is what I would do in order for, you know, an engineering team to build a good product and apply the AI win. All right folks, you heard it here.

Caution is always important, but you full steam ahead of the same time. You just gotta be careful about what you're doing and remember, all this stuff isn't going away, it's just gonna be a matter of how we fine tune it from here to get the most value out of it. Hey Karthik, thanks for being on the show.

Absolutely. Sure. Thank you so much.

All right. And thank you all for watching the latest episode of the Techstrong AI Leadership Series. We'll find this episode and others on our website.

We invite you to check them all out. Till then, we'll see you next time.