Bridging Quantitative and Qualitative Digital Experience Testing with AI | Digital CxO Summit

Hi everyone. Uh, I'm Ranjita Kumar. I'm the chief scientist at user testing and an associate professor of computer science at the University of Illinois in Urbana Champaign.

Uh, I've spent the last decade or so looking at how to leverage data mining and machine learning to optimize digital experience design. And so today I'm excited to share some of those learnings with you. I usually start out by motivating why digital experience design is important, but given that the summit is about digital transformation, it feels a little unnecessary.

Uh, so we can jump right to the more important question around how can we create good digital experiences. Now, I suspect that everyone here works at a company that uses one or both of these types of interfaces that you see on the screen to optimize design. Uh, the one on the left usually represents an analytics dashboard that presents quantitative analysis over high volume experience data captured during in the wild usage.

Uh, the one on the right usually represents a UX testing platform that presents qualitative analysis over rich feedback collected from a scripted test. Now, you can also see that I've listed surveys in both categories because you can actually find survey tool interfaces that look like both. So I thought it was only fair to list surveys as, uh, belonging to both categories, but I, I hope you instantly recognize these tools, uh, when, when you saw them on the slide.

Now, I also suspect that if your company is using both types of interfaces, that you all are using at least two different kinds of tools and that these different tools are being used by different parts of the organization. Um, and I think this observation is interesting because the goal of both sides here is to optimize the overall digital experience, um, and you can make a compelling case for how they could be used in concert to holistically improve design. But currently they, they are two separate worlds.

And the other interesting thing, uh, you know, is that I'm not the first person to ask this question, right? Uh, there's about why these worlds are different. There's definitely a race in the market to bring, um, quantitative and qualitative methods under one roof.

And one thing you're definitely noticing in the market is that the different types of experience data, uh, captured by quantitative and qualitative tools are converging. For example, uh, before analytics tools just captured interaction and clickstream data, but now a lot of them support video playback of a single session screen recording, and a lot of them call that session replay. And similarly, UX testing platforms are deploying survey capabilities where you can run tests with hundreds of participants.

But I think the big opportunity here is not in the race to capture all types of experience data that you can, but to reimagine how we do design optimization altogether to create new types of workflows. So I want you to imagine a future where your analytics platform monitors the key user flows on your website and detects fluctuations in performance metrics, such as drop off rates at different points in the flow. And if an anomaly is detected, it actually automatically triggers a UX test to be generated based on the context of the specific flow and the anomaly itself.

Then you can imagine it intercepts other users on that flow on your website to take the test, and you can actually learn why that drop off is occurring. So in this example, we're illustrating a future workflow that seamlessly feeds quantitative analysis, which identifies the what into qualitative analysis, which answers the question why. Now, similarly, you can also go the other direction.

Um, you can, you know, once you run the test, you can have, uh, results that are analyzed and insights that are summarized automatically for you. Um, but you still might have the problem of knowing, you know, which, which insights are important and how much do they matter. And here you can turn back to the large scale analytics data to help you contextualize the results from the small scale UX tests that you've run.

So, um, a lot of the components in this future vision I was talking about, right, like anomaly detection, automated test generation, automated insight summaries, insight summarization, all of that can be powered by machine learning. And I think that this is the great opportunity in this space, uh, leveraging machine learning to create new workflows that holistically optimize design by bridging these quantitative and qualitative worlds. And this is, you know, the future I've been personally working on, um, for the last decade or so, and we're not quite there yet.

Uh, you know, the solutions obviously are there in the market, but I wanted to share with you a few of my learnings in this space and, uh, what we've built at user testing toward this vision. So one thing I, uh, wanted to again throw out there as an observation is a few slides ago we looked at all the different types of user data, um, that was collected through these platforms, like user interactions, survey responses, think aloud, audio, and so on, right? Uh, but we never really talked about design data itself, and I think that's interesting because we are trying to optimize design.

And so it should strike you as a little odd that we didn't actually collect any design data except for maybe those screen capture videos where you're seeing the design as people interact with it. And it's interesting, right? Because a user that's watching that screen recording can understand that the person in the recording is searching for products or browsing through the search results and adding them to their cart, but a machine can't just automatically understand that from that data representation.

And so one of the, I think, key learnings or takeaways from my research, uh, for the last decade is that we should actually collect design data while collecting all of this other experience user experience data. And so I just wanted to show you one of the, um, the systems that we built in my research lab allows you to capture interaction and design data while someone is using a digital application. And I wanna point out to you here that by design data, I, um, mean both the pixels on the screen as well as the render time data structure that gets created that describes the UI elements and their layout on the screen.

And so we're actually building up this rich representation of design that looks at the pixels and the structure, um, of the design and projects user interaction on top of it so that you can, you know, actually think of it as having an aligned and it exploded view of a video where it shows you how somebody interacted with each screen and went from one screen to the next. And it turns out that if you collect data, design data in the structured way, it actually makes experience data more quantifiable and amenable to quantitative analysis. And so one of the other things my lab did was build a proof of concept experience testing platform where you could run usability studies, um, and aggregate that user interaction data like you would in an analytics dashboard and correlate that data with verbal and survey feedback.

So, you know, if you've ever run UX tests, normally what you do is watch videos and you don't engage with these quantitative dashboards. But we showed that if you capture the structure design data that you can basically, um, treat that you can actually, um, use that as a way to quantify that design data and, um, do quantitative analysis over it. So this is, um, this is where I actually transitioned to user testing because it was kind of at this point in, uh, my research journey that I was fortunate enough to connect with user testing.

And if you're not familiar with user testing, it's a remote UX testing platform that allows you to quickly get feedback on any digital user experience. So, so you can, um, easily launch tests and in a few hours get videos back that capture how participants interacted with the digital experience and what they thought of them. And so you can see we capture videos, um, where you get the think aloud data, you get the screen capture data, and you even get face data.

Um, so you can see what the participant, you can see the facial expressions of the participant. So when I joined user testing, and that was back in 2019, um, the, the main way of consuming the content produced by the tests that you ran on the platform was to sit down and sequentially and linearly watch all the videos. Now, many of our customers still find that very useful, and I think, you know, if you're doing qualitative coding, like that's the surest way to make sure you don't miss anything.

But we've also now introduced features that leverage my lab's design mining to techniques with, you know, cutting edge machine learning to increase the, the speed to insight. So what you're looking at right now is a, uh, feature that we launched two weeks ago in beta called AI Insight Summary. And it combines think aloud audio user interactions and design data to produce multimodal insight summaries for the test that you run.

And so one thing that's, um, I think a differentiator that we've created here in this space is while most people are taking audio data and, um, transcribing it and then providing it as input to large language models, what we're doing is actually, um, creating our own unique multimodal data represent data representation that combines the audio transcript and the behavioral data, which we convert into a behavioral transcript that combines, um, you know, the interactions on the screen as well as the design data. Because as I mentioned before, the design data actually gives the interaction and behavior context. So we actually produce what we call a behavioral transcript that basically, um, has timestamped lines that say things like, um, user clicked on search button, um, user filter through, uh, five options and chose the color red.

Um, so you are actually converting behavioral data into a natural language representation, and that allows us to blend that data with the verbal data, uh, the verbal audio data that we collect and create a multimodal transcript. And then we prompt the L l M to use that multimodal transcript and derive insights from it. Now, um, you can actually see some of these transcripts in the next screen here.

Um, because what we do, and this is very important when you're working with, uh, you know, anything that's probabilistic, but definitely large language models is that there is, um, you know, it is possible that a large language model can hallucinate and synthesize information that isn't actually there in the data, right? And as synthesize an insight that where there's not actually, um, evidence to back it up in the data. So what we do is we force, um, in the prompt for the large language model to provide evidence from the multimodal transcript to make sure that a user, when he or she looks at a a multimodal insight, can click into it and see, um, where this evidence is coming from.

And sometimes the evidence is coming from the behavioral data stream, sometimes it's coming from what people said. And what we try to do is we make it really easy for someone to verify quickly if they buy the synthesized insight or not, um, as well as actually go and watch the videos because, you know, I think the key here in this space is that at the end of the day, you know, the, the thing that matters the most is your customer's experience. Um, and so you really want to understand what you users and your customers are thinking and going through.

And so the role of AI here is not to replace what they're thinking, right? But to actually get you to those moments in the video where they're saying things that matter and the, and that you wanna capture and act upon. And so this AI insight summary feature, which I should say is in beta, is, um, attempting to quickly take you to those moments in the video and actually have you watch those clips to really understand what's going on.

Um, and you can see like we've also provided other types of visualizations where you can drill in and see aggregated views of different types of data, whether it's behavioral or verbal, um, to really pinpoint where interesting things are happening. And so, um, I guess I, I, I thought this would take me a lot longer to get through, but I think that the vision that I wanted to kind of put forth today was that there's kind of two takeaways here. I think, um, you know, when we talk about quantitative and qualitative methods, I, I think the future should be one where we're working together with both to holistically improve design and optimize design.

And, um, I think there's this really great opportunity to create new ML powered workflows in this space. And I guess I lied. There's the third takeaway is that what you wanna do to actually, um, create these new workflows is to leverage the design data that is also underlying all of this experience data to contextualize the experience data you're capturing.

Because that's what the machine learning models can also leverage to get you better insights, to aggregate information, to understand how to map from a quantitative world with large scale data to a qualitative world with, um, smaller but richer, uh, smaller scale, but richer data. And so, um, I think that's kind of my message for today.