AI Innovation with Gaurav Awadhwal at Atlassian Team ’24

This is techron tv. Hi, Mitch Ashley here at Atlassian, team 24 in Las Vegas, Nevada. Talking about ai, of course, you can't have a conversation without talking ai.

We're gonna jump kind of in the deep end of the pool and talk about building products around ai. I've a distinct pleasure of being, uh, joined by GU Al, and tell me how to say your last name. Awa.

Yeah, AWA. Thank you very much. Thanks.

Nice. Yeah. And you're a senior, senior engineering manager working in the AI ml space within Atlassian.

Yep. Great. When, introduce yourself, just a little bit about what you do and kind of work that you have going on.

Sure. Thanks. Thanks, rich.

Um, yeah, so I lead the AI platform team at Atlassian. So, um, my team is one of the many teams helping contribute to build out our portfolio of AI features, um, that we're launching under the Atlassian intelligence brand. And now, um, as of this event's announcement, uh, our new product, Atlassian Rvo, which is an AI powered product.

So, uh, yeah, my team helps build many of the underlying, um, AI tools that are used to help enable the other teams, uh, train, uh, harness the power of AI, and, uh, develop models and go through the whole ML and AI lifecycle. Um, so we can, uh, across the company really have a wide variety of different tailored AI solutions. Um, so yeah, I've been at the company about seven or eight years, um, working in the data ml, um, space.

Um, and it's been really exciting to go through this journey at Atlassian where, uh, we've been laddering up our ability to, uh, help our customers leverage their own data, um, in more productive ways, um, using tools like big data analytics, machine learning, and now most recently generative ai. And we're really excited about what we have to announce at this event. Very cool.

Well, I'd love to hear more about your journey, because every product, company, service company is, you know, you're supposed to be adding ai, generative AI to your product. You wanna do it in a useful meaning way, meaningful way, but I kind think of it as you don't put airbags on the car at the end of the assembly line right there. Yeah.

There, you've gotta do some work to get that into the design of what you're creating. And it's, tell us a little bit about your journey to, to how you got to where you are, and then let's talk about operating as kind of a platform team supporting a lot of AI initiatives in the organization. So start with your journey.

Sure. Um, yeah, so I've been Atlassian, um, over seven years. And, uh, in that time, um, the company was going through its cloud transformation very heavily.

So, um, when I was starting, we'd already launched our cloud products, but we were obviously trying to move to become cloud first and have, uh, a really great offering that fits support enterprises and we could use to build these richer features like ml ai, really advanced search features that wouldn't be possible, um, otherwise without these, um, the move to cloud. So, um, through my journey, I was able to work, help out, help build our first, um, analytics, um, solutions, help build our, uh, data lake, um, help build some of the recommended systems and things that we've built on top of that to help, uh, surface the most useful data. For example, um, our collaboration graph that, uh, surfaces, uh, who our customers, uh, users work with and who their most relevant collaborators are.

A, to show them, uh, show people within their sites, but also use that, for example, as a signal for search to surface only the most relevant personalized, um, data. So that's something that has been part of my journey. Um, and it always felt like we were laddering towards something like this.

Uh, I don't think anyone knew exactly when or exactly how generative AI would manifest or appear for us, but, um, it felt like we were, you know, building this pyramid with the foundation, of course, being a really secure, um, base of data, um, from our customers, making sure, of course, we respect their data and privacy. That's been a very, um, long running thread as well in my career here. Um, and then making that data useful and presenting personalization and, um, opportunities to actually leverage that in a way that delights customers, making it a product centric, um, and all of the new technologies that have appeared in that time to help slowly build and ladder us up to today.

It's a good way to think about it. I mean, I am thinking about your analytics experience, um, you know, solving problems by creating a collaboration graph, right? Yep.

As an end user would say, oh, that's a collaboration graph working, you know, there are technologies behind this in addition to the data. Same thing with generative ai. We all know LLM.

Okay, but what does that mean? You don't kind of start out there and there's not an LLM course and now you're off and running, right? I mean, you can, you can learn about things that way, but how did those stepping stones to where you are help inform or equip you and the people you work with?

I think that's a great question. So when it comes to, um, making data, um, useful for our customers, uh, we really want to, you know, collect signals about how they're using the products, um, who they work with, um, and also like the nuances in those signals. I, I think that's called to machine learning.

Hmm. Um, so we've been doing that kind of, um, behavioral data, uh, analysis over time and also building the underlying systems that can support the scale of Atlassian's customers. That's another challenge, which we, we have so many customers around the world in many regions.

We have, um, all of them having very sensitive data requirements. We need to isolate their data. Um, so having those systems built over time to help us gather these signals in a compliant, secure way, helping us, you know, maybe surface them back to the customers in their collaboration model, um, those has have helped us understand how to build these scalable systems.

And, and now those same signals and the same kind of pathways for datas data are also helping us, uh, feed our AI with relevant context, personalized, but securely, um, access data, um, and, you know, relevant search results that we have fine tuned over time, and we're continuing to work on and invest in those same, you know, building blocks. The, now the last layer is this generative AI that then extracts more meaning and understanding from it. Um, but yeah, working with a team that understands those building blocks, how to put them together, scale them out, how to do those things privately, we've just had to, um, invest in making that at a lot larger scale because we are developing so many new features, um, adding new capabilities to support all the additional development work.

Our many, many product teams who here we're developing AI features are working on. Um, that's been the newer challenge. So the building blocks have been there for a while.

I'm curious, 'cause, um, I imagine through your days doing data analytics, now moving into AI is Yeah. In the software DevOps world, we talk about feedback loops. Yeah.

Right. And so it's not just doing the work, it's learning from what you're doing and also what other people are trying to do with what you're creating. How do you build in those feedback loops?

Oh, that's great. So, you know, it's not a always a feature request that shows up nice and cleanly, right? Oh, we're looking, looking for more signals than that.

Yeah, absolutely. So, um, the data analytics are not just used for, um, improving personalization. They're also used, um, to help us build an understanding of how our customers use the product.

So we, we definitely take our customers Jack tickets, their, um, feature requests and their actual, you know, written feedback, but we also look at which features are people using, which are they, which are they not using? And it's actually very relevant to AI features, which is that we are really taking an approach to make sure we measure how those are being used, uh, where we're finding the quality of those experiences to be matching our customer's expectations, which are being most useful. There's a lot of experimentation going on right now, and that's another big part of our approach that's very important.

So yeah, data not only helps us, um, feed into models and provide back answers, but also we collect the appropriate level of, um, understanding from it to help improve the experience for our customers as well. Great. So this is not a trick question.

Yeah. There, there's no right answer. There's a proverbial, is AI a feature or is it a product?

I mean, does it, does you think AI kind of falls into one camper the other? Is this a little bit of both? Everywhere, it's early, we're still learning about generated ai.

Yeah, for sure. What's your view on that? I, I mean, maybe it's a bit of a dodge, but I think it's both.

Uh, we have, we believe that AI features will fundamentally transform our products, which is why we're investing so heavily, we think in a few years the way that users interact with our products will be fairly fundamentally different. Um, we're adding a lot of AI surface areas where a user, for example, might have had to manually enter and create a Jira query language query where they, in their brain would've had to understand this bespoke, but very powerful language that Atlassian has. Uh, but now we can use AI to help change that interface completely and have them use that as the interface.

Uh, so there's features coming that will slowly change the touch points, and we also have to get our users to adjust the people on the other side. And this is something I have experienced with as, um, a user of our own features, um, managing a team who are also onboarding into AI and trying to use it more heavily in our lives. Um, we're also experiencing, we on the other side are changing ourselves a bit, um, but at this, um, event this week, we've announced, um, Atlassian Rover, which is our first, um, fully, um, AI product.

And this is its own, you know, from scratch designed ai, um, from scratch AI led products that will a help you find, um, things using the power of machine learning. And, um, for, from data from across your organization, it'll help you learn by using AI to simplify and distill knowledge for you, and of course, help use AI to act. So yeah, the, the products that are completely AI are coming.

Uh, but yeah, we have so much, so much other surface area that will also be fundamentally changed my ai. Yep. And I, and I believe I understand right, robo will also bring in data from Google and other Yeah.

Um, yeah. Other, other data sources. So you can compliment that or with, along with what you have in Ian, that's A really exciting aspect of Rogo is that we think that, um, you know, Atlassian, we have 20 years of experience plus in understanding how companies build knowledge, and we want to extend that beyond our walls, um, especially in this AI era.

So at that first layer of rogo where we help, um, teams find the relevant information, we think it's important to help extend that to wherever they're working. So Rover will have connectors that will allow the dr your Google Drive, you know, other types of data sources that we will be announcing and sharing, um, and continue to extend in future, um, into your ecosystem in a single pane of glass of search that, you know, you can then, uh, very quickly collect the relevant data from if you're just trying to find something. But then the next steps are also gonna benefit from that, that when, when you need to distill the knowledge out of that in our knowledge cards, for example, which when you ask a question, uh, can give you a AI generated answer or, um, help give you, um, explanation of a topic, it can pull data from all those sources wherever that data exists.

Um, and eventually when you need to act on it through ai, again, that's pretty key. And all that experience will help you when you do those automated actions. That's right.

Yeah. Now you're not doing it for the first time and, you know, see what happens. You have a lot of experience of people using it in their workflow making decisions.

Yeah, exactly. So we, we believe we've learned a lot about how people and companies especially construct their, um, you know, decisions, how they go from knowledge to actions. And that's something, uh, unique about Atlassian, that we have this rich experience that we can leverage and make AI meaningful with because we want to use AI to help accelerate the productivity.

Like internally we've been testing these features and we've seen very significant improvements, um, in the time it takes, for example, onboarding a new team member, um, is a task that I've been doing a lot of on my team. And I think when people, rather than having to ask another person the first time, Hey, what does this acronym mean? But they can actually use a RO glossary, which is they can search an internal acronym name and it'll distill from multiple sources what it means as the first cut before they have to go ask one of their colleagues.

It saves a lot of time. So I think we've, we've had a lot of learning and how this works and, you know, seeing that transition between our products as well, for example, tho those definitions being visible on a Jira ticket when, uh, a developer is about to work on it and it references a confluence page with acronyms and, uh, made up terminology that say a product manager is documented somewhere else, um, we can really extend that because we know how people have used our products in the past and, uh, use knowledge products and workflow products, and we think we can extend that again to these additional data sources. Excellent.

Very good. You know, there's a couple of challenges everyone faces incorporating gen AI in particular. One of 'em is hallucinations and accuracy.

It's one thing when you say chat g pt, right? My bio, and yep, it's half right, you, it's another wind that's a procedure. I need to do some action, and it's gotta be accurate.

How do you, how do you implement something you know, you can control the level of accuracy? That's A great question. Look, that's something we're very con concerned with, um, when we're developing.

And, and of course, you know, we have to, um, continually improve that. So we are always, there is always some, you know, expectation that AI generated answers may have some occasional challenges with them. So one of the first things we do is we make it very clear to users wherever we use ai, um, it's gonna be very clear that that was an AI generator response, because we think that that's a really key part of it.

So, um, it's fairly transparent. That's really, um, important aspect. The second is, again, as I mentioned, we're taking a very data centric approach by first of all, leveraging data and using that as input to the LA language models.

Um, we believe that they're far less likely to hallucinate. We have a lot of data to prove that when you generate with a really rich context, with the relevant data pulled in, it's much less likely to hallucinate an answer because it has actual, um, material data to fall from. So that's why we focus so much effort on the teamwork graph, which is what we call, um, our model of wherever, um, our customer's data is and all the rich data sources we have by having that far less likely to hallucinate.

And finally, we also do a lot of internal benchmarking validation work. We do a lot of technical work to improve the accuracy of our usage of the models, any search algorithms we use, any additional models, we use a lot of technical work to continually iterate and improve this process. And, uh, we also, you know, really believe in trust and safety around this and have a very, um, considered approach to AI because of that.

And you, I have to believe there's a vast difference between building your models, building your language models around data that you have of actual people doing things, versus going out to the internet and scraping all the stuff that's out there and who knows what, what validity it has or relevance to what you're doing. But you do have kind of a domain specific, maybe not in every case for everybody, but it's people doing these kinds of work, these kind of workflows. Is that, that's gotta make this easier to tune in the accuracy or, or is it that big of a factor?

Am I making too much of that? Well, the language models themselves will contain, um, you know, to understand natural language, have got diverse data sources, um, but I think it's about the craft and the art of using them correctly so you can make them behave in the way that will be far less likely to do that. We also do stay abreast with the latest model developments because things are progressing very quickly and there are additional controls you can put in place that will help, um, mitigate that check that the responses are correct, help, um, detect any kind of, uh, unexpected behaviors.

So we also do that. So yeah, for for sure, um, what we call grounding on domain data, that's very important. And we do that in our products.

We do that in a very, uh, a security conscious way that we only use a teamwork graph that your company has. We don't use it to train the models, we only use it to, uh, provide context to the models, but in that process, we ground it so that they're less likely to hallucinate. That's also personalized, so it's not gonna show you documents even within your company, you don't have, uh, permission to.

So data leakage is another concern that is very, um, valid in the world of AI right now and something we took very seriously. Um, but yeah, we can mitigate some of this effect of the language models themselves having a wide kind of, um, in internal, uh, knowledge base. Um, and only you fall back on that rarely when it's necessary, um, and have it very transparent to use is when AI is being used.

Okay, great. You know, and you, you touched on another really important, I'll call it a challenge of when you're offering a service, excuse me, but you want domain specific data knowledge in it. Um, how you protect that.

You know, there's things like retrieval, augmented generation Yeah. Things like that, that help you not let my data go into someone else's model. Correct.

Are you using technologies like that or other techniques? How do you, how do you balance that challenge? Yeah, so Rvo search and all the function, all the features that search, um, your content or retrieve from the teamwork graph in Rvo or any of the Atlassian intelligence features inherently respect permissions.

Wherever the data is, even if it's on one of these external sources like Google Drive, we actually use each individual and each company's permissions. So by making sure that the data before it can even enter the AI system has been filtered to only the appropriate content, we ensure that there can't be any data leakage. It's, it's basically that simple.

Um, and a lot of our effort has gone into that layer to have this retrieval, augmented generation piece, first of all, secure, that's table stakes for us, you know, uh, respects company and per user based emissions, and then also be very relevant. So we're continuously working on the algorithms that underpin this, and that's some of the work, my team and some of the teams, um, that I work with, um, are really rapidly iterating on right now, which is how can we make the quality of the search results, all this, um, knowledge rie extremely relevant to our customers, to our users. So it can both help them find what's relevant to them when they need it from all these sources, but then also very importantly, help when using these more advanced AI capabilities have the AI focus in on the most valid and important pieces of content.

Kind of a new version of the garbage in garbage out, right? Secure. Exactly right.

Secure and secure out. Yeah, it all starts with that access control. Both security and relevance are two things that we think are very critical, and I think we as Atlassian believe that, you know, we need to have a very considered approach of, So not asking for any, you know, insights into product or anything like that.

Just speaking generally technically, what, what are some of the next problem areas challenges that you might be interested in working on? I think I'm really excited to see how, um, our customers define agents in robo and, um, themselves. Go ahead and, uh, customize the product because there is some element of creativity now, so it'll be really interesting to see how they use it and get feedback and see where the most, the highest value comes out of which parts of the product surface area they really, um, drill in on.

I think we can also, we're also on the technology side seeing this, um, move to having, uh, more like task specific models, so that can do really narrow specialized actions, and I think we'll see that come in, um, and we're working in that area, so that's really exciting to see that we can do more of these advanced capabilities on how, um, how well the AI can actually know the nuances of every type of action or across workflows in, in our products. So I think both learning from our customers and, uh, you know, really leveraging the depth of our products are two areas I'm really excited in. But I think what's, what I find really cool here is that if you ask any individual who works on AI features, that'll give you a different answer, which probably tells you that there's so much opportunity here.

We've basically, we've just, you know, unlocked the door really on something huge, and I think that's what I'm most interested in, which is how can we, um, especially because my team's role is helping support this diversity of use cases, um, I'm really excited to see how do we stay abreast of all the potential, you know, alleys and, um, you know, directions that AI can go in. How do we continue to that securely and scale it out? Um, because yeah, I think one year's time there'll be aspects of this that we can't even predict sitting here today, though, Even a year from now.

Yeah. That'd be difficult to do. That's it.

Yeah. It's, it's amazing, you know, innovation comes from a lot of sources. Yeah.

You're describing one is that here's all the interesting things that we bring to the table. Yeah. Another one is what you were mentioning around, let's see what people do with our products.

Yeah. Um, you know, I've had the, well, you're not supposed to do that with our product. Well, but there's an interest.

Let's find out why they're doing that, what problem they're trying to solve. There may be an opportunity there or a new kind of use of film we hadn't never thought of. So yeah, we're kinda limited by our own Yeah.

Distillation of the problem set of in ways people might solve using our technology. Be open to other things. That's right.

And we've already seen, seen just by releasing, um, the AI platform tools within Atlassian to our many developer teams, the proliferation of ideas, like some of the things that are now full on product features, you know, came from individual engineering innovation projects. So I think this is gonna be a real grassroots thing. There's good ideas come from anywhere.

We're really handing out a very, you know, interesting capability. Some things will stick, some things won't. And what, and just the speed of iteration I'm seeing is, you know, it's very, very exciting.

That's what I would say is really novel about ai. Excellent. Well, congratulations on the great work that's led to several innovations Yeah.

Coming out in, uh, uh, in products today and thank you capabilities tomorrow. And keep up with the good work I tied it to see what you and your team and the rest of Atlassian where you go next. Yeah.

Thanks Mitch, on that uncharted path. We don't know what it is yet. We think we might know, but we'll see where you end up.

That's great. Thanks Vince. Yeah, I'm excited as well.

Thank you. Good talking with you. Nice.

Great. Rob is, you know, this is living the dream here, helping create the AI that, uh, we're using and learning from and in the technology. And you always wonder if you haven't started around on your own AI journey, how far you might be behind.

So if it's, so much of it is experiential too, and getting some hands-on knowledge. So dive right in, learn and join with the rest of us that we're learning. So thank you gr good talking with you.

Thank you. We will see you again. We've got another, uh, set of interviews coming up, so don't go too far.

We'd be glad to, uh, share some more great information from amazing people here at, uh, Atlassian, team 24.