What you Need to Know About Microsoft Copilot AI | RSAC Virtual 2024

So my name is Naomi Moneypenny. Yes, it is my real name. Just get that question out of the way.

So that trusted identity, uh, is not quite the same as what you see on the movies. Uh, but, uh, very boringly got the name from my father. No idea where, uh, the James Bond references came into that.

And no James Bond jokes, I'm sorry. I do not have those in. During this presentation.

I have the pleasure of leading product development for parts of Microsoft 365. So Microsoft 365, there's a lot of stuff inside of the Microsoft world. Microsoft 365 is the part that you interact with as a, an end user.

So it's the word, it's the office, it's a PowerPoint, those kinds of applications, uh, along with things as Mark mentioned, like SharePoint and other pieces too. So lots of goodness there on the end user side. And that's what I'm gonna primarily talk about since that's the part where my team as an engineering product leader, I have to go through and think about these different areas and how we're incorporating obviously security as we go through this too.

Now, my plan for today is just to tell you just a little bit about Microsoft Copilot, because I'm not assuming that everybody even knows what Microsoft Copilot is. So forgive me if it's a little bit of, uh, you know, recap for some folks. And then we're gonna go through just a little bit about how copilot actually works and all of the extensibility pieces there.

And so there's a whole bunch of pieces that go into this and I thought it was really useful for you all to be able to identify this and be able to understand like how we're approaching some of those areas and some of those technical challenges that we have to deal with every single day. So first off, I have a much better video than it will explain than I can do. But, uh, it will help me to explain what copilot is.

Well Straight to the top, never going down. Don't wait for the drop stay still. This the motto.

Brand new pants for the auto foot to the ground. Full throttle, big energy for the nightlight lotto. Y'all talk gots never dis quiet life.

F*****g Seinfeld, black ball full of songs, all of them bombs. Something like a minefield. Got boom, get it, get it.

Tickets running out quick. It it, never get it out. Said it said it never had a shout in.

They said, I couldn't ever do it. Okay, how's this fit? Week worth of work in a minute.

Machine wire oil, you high stay efficient to do this for I do this different. Hold up boy, a minute I ain't finished. Okay, flows the same, the same, the same, the same.

Now paved the way the snakes and fakes are chased out back to back to back. I changed the pace now had the stack, the cash until the banks out straight to the top. Never going down.

Don't wait for the drop. Uh, all good. So hopefully that gives you just a little bit of insight into what Microsoft Co-pilot is.

When we talk about this, this is going across all of the applications that people look at every single day. So we have copilot for security, we have copilot for developers, we have copilot inside of GitHub, we have copilot in those office applications that you use every day. We have copilot inside of Bing and Edge to tell you a little bit about web and work results too.

So it's just a great way to be able to see all of these pieces coming together and then help having those copilot experiences assisting you at every kind of role at every kind of user. And so it's really interesting world that we live in because of that, right? Because we have to build on top of that.

We don't wanna think about copilot as just being a large language model. We've talked so much today about all of the latest, you know, information that you have, all of these generative AI capabilities coming to life. But we wanna make sure that all of these things are informed, of course, by the customer's data.

And so we get to use things like the Microsoft graph behind the scenes, which actually helps to store a lot of, all of that good information too. So we wanna make sure as part of that security compliance approach that's it's here, that helps us to understand just a little bit more about all of these pieces. And so the ra, our ra ai part of this is the most important I would say.

So when you think about these models that we're interacting with EV every single day as an end user, we wanna make sure that there's no bias in those models, right? We wanna make sure as much as we can, we've removed any of those complications in there. And so from a product development perspective, this is something that we really have to look at very, very carefully.

We take these incredibly seriously inside of Microsoft, looking at all of these different issues and then making sure that the models and how we produce content out of that becomes I even more important Microsoft. So I wanna go through this. So a couple things that I wanted to, to go through because I wanted to show you just a little bit about how people interact with copilot and that will help you to understand just a little bit about how much they, they love the service.

And because within that we love the service, it means that we also get a whole bunch of new vectors for that, right? Because people and end users are unpredictable creatures. So there's a study that we did, and I'm gonna go through these very quickly, but there was a study that we did that helps us to understand the impact of copilot.

When end users get to use a model, like a large language model for the first time, and you expose all of that amazing creation and copilot ness with them, uh, you get to see how they're thinking about it. So a couple of things that came up as part of this, 77% of the people who used copilot don't want to go back with working without it. So as we think about the rise of the machines and all of these pieces, that's, uh, another big stat that we think is important.

We also see all that people saw that they were more productive, right? This was about creativity and they thought about the quality of their work as part of this too. And they were also looking at how they can think about faster and fast and speed on the, on the specific tasks that they were doing.

As we looked at this as kind of, you know, catching up on things like missed meetings, these are very obvious things and Q'S cases that we want to use for our copilot adaption. Making sure that we're able to recap a meeting with the correct, uh, the synthesis, the correct summary, that's part of it. And then thinking through how we're actually helping to explain all of the resources that are connected into that.

And so some of that we're invoking that through that Microsoft graph. That graph is basically the connectivity layer behind the scenes. It knows about your identity, it knows about how your meetings, the tasks, all of the pieces that are connected together.

And so we wanna give you a synthesis that's important to you. And then of course, make sure that we're helping you to get to the resources that created that synthesis in the first place. Lots of things here around lost less time on email, less time on searching, which we all appreciate.

And then we even ask people as like, Hey, do you wanna choose co-pilot over having a weekly free lunch? Uh, and so this is a good one. I thought it was really interesting to see, uh, that folks actually picked that too.

So lots and lots of goodness out of those co-pilot experiences. So what I wanted to go through now is just to tell you a little bit about what we're doing from a developer productivity perspective, because this is a piece where I think it becomes even more, uh, clear what we have to do in terms of copilot and the other pieces too. So a couple things here.

First off is to look at kind of that developer experience, and we are looking at how we can increase that developer experience with things like copilot inside of GitHub. I've got some examples that are in the deck. And so when you get the deck afterwards, you'll be able to check these out.

But it's really interesting to think through, like how does that developer experience continue to evolve when we're looking at how we can incorporate these into things like the GitHub community, there is also a plugin that's available inside of GitHub right now that helps you to get some of those security best practices directly in the product as you're using it. And we think this is a really important way to help create that, that sensitivity, create that understanding that there's code snippets out there. We wanna make sure that those things are not being incorporated as part of your code as well.

So lots of, lots of goodness as part of that. Now when we see people using those developer productivity tools, it's pretty amazing, right? I for one, it's definitely something inside of my own engineering team that we observe is to make sure that folks are able to go faster with that co-pilot experience and then making sure that they're becoming more productive and that people are seeing the benefits of doing all of that as well.

So lots and lots of goodness here. Now I wanna walk through just a little bit more on how copilot actually works behind the scenes, because I think you all will be the most interested to see this. And then think about, you know, how that grounding happens and how all those data models interact with each other.

And in as much as I'm able to show you that, I'm gonna try to, to do a little bit of that today. So first off is just thinking through like, you know, what is that, that favorite canvas that you might be looking at from a copilot experience? So we're looking at Microsoft 365.

In this case, we wanna look at how apps like teams and Word and PowerPoint, et cetera are being used here. Or it could be the Microsoft 365 chat that's on Bing, which is basically helping a secure experience there directly inside of this. So we're able to use essentially those utterances and they reach copilot.

The app is then helping to copi is then interacting with copilot, and then we're using obviously our web sockets as part of that too, and everything's running securely as part of that. So then we're doing a bunch of pre-processing in here. So that copilot, pre-processing, uh, helps us to do the grounding essentially here.

It helps to improve that specificity of the prompt itself. And then it's helping you to get the answers that you need as part of that, that data that's coming from copilot here is coming from an authenticated user. So we're making sure that we're scoping down to the documents and to the data that they already have access to as part of this.

And then we're using that through the Microsoft 365 role-based, uh, access controls. So the thinking through, like how we're using that grounding information, we're essentially reaching out to that graph, that graph that has all of those signal data, all of that edges in there, and then thinking through that customer and tenant data and then essentially using that to ground it. And then we are also able to supplement that with web data.

Again, that's being secured as part of this too. There's a couple things that I wanna draw out explicitly here too, because this is an interesting world, right? When we are in this world of end user data and all the things that have to come as part of this.

So we wanna make sure that we have, you know, administration, co-pilots as part of this too. Uh, and so making sure that that user experience is helping us to understand like the query preference. We can actually look at like, are you interacting with co-pilot on the web from the intent of the user versus co-pilot for work, right?

And we ground things and we do things a little bit differently as part of that too. And then as part of the, the data flows that are going in here, right? We wanna make sure that they're actually coming in through that Microsoft 365 boundary too.

And then assuring if there's any plugins that are coming. So many of you all are with software companies, right? You have an ISV, you wanna be able to bring in that data as part of it too, and we wanna make sure that there's actually explicit admin consent that's happening as part of copilot to access the services and how it gets to be controlled.

So it's a number of different areas that come together here. So when we think into stage four now, um, this is really around the retrieval of all of that information. And so this is our retrieval augmentation generation.

It allows co-pilots to get exactly the right type of information. So we make sure that we're inputting that into the LLM and then we're combining that user data and helping us to use that information retrieval. So if we need things like a knowledge base article to help improve the prompt, this is the stage that we're bringing that in as well.

And then we can get ready around a couple of times, right? Sometimes the query's not clear, sometimes we wanna make sure we're improving as part of this too. Uh, and then we can think about however many times we need to run those different stages to improve the post-processing as part of that.

So I know there's a lot of a lot of things on this diagram, but I'm trying to break it down step by step. You're tracking with me so far. Just wanna check.

Yes. Got some head nods? Yes.

All right, thank you. Thank you. Just wanna make sure, uh, and then we wanna think about post-processing here.

So this is where, you know, responsible AI really comes into this. And so responsible ai, we wanna make sure that we're, you know, we're not looking at involving any humans here at Microsoft. This is customer data, right?

And so we wanna make sure that the customer protections are always there as part of this, but we wanna make sure also that we're, you know, calling out to the Microsoft graph for additional security, the compliance reviews, privacy reviews, uh, and all of the command generation that happens as part of this too. So we wanna think about, you know, this is where the governance piece comes into this. So things like Microsoft Purview that help you to put in data labels on your data, right?

Making sure that end users are actually doing that compliance as part of it too. Uh, and then all of the traditional ways I would say we have of securing information in this area too. So this is kind of a cool thing, right?

And you think about this because we wanna use copilot to use things like sensitivity labels or something marked, you know, confidential, highly confidential of that. Goodness too. We wanna make sure that they're doing that and actually checking for all of that auditing, e-discovery, all of these, uh, these issues that come up as part of this as well.

So this is the part where we're able to do a bunch of these checks and helping to make sure that we can actually, uh, inform all of that too. So obviously into the next stage here, uh, you can see here that user prompt, um, is being sent back essentially to the user. And then we have those citation links that are coming in as part of this too.

Uh, those citation links allow us to be able to check and make sure that we understand where that information is coming from. We always wanna make sure we're giving an explanation essentially to the user. Seems very simple thing to do, but we want people to be able to establish trust with the system so they're able to go through and authenticate what they've seen as part of this too.

And then as last stage here, um, hopefully then co-pilot is then giving you kind of a recommended response, right? It's gonna command back to the applications where the user is actually seeing all of this. And, you know, make sure that you can assess that suggested response here.

So you can see here, like it's a pretty sophisticated, you know, process and orchestration. I have it in the simplest diagram that I could put together on this, but hopefully you see how all of this is coming to life, uh, as part of this and making sure that we're contextually basing all of this information too, and then ensuring that we have basically the, the best of the web and any third party extensions that you have in this part as well, and that it kind of really helps you to put all of this together. Now, let's go into a couple of other areas here.

So core component of copilot is obviously making sure we have the Microsoft graph. If you're familiar with the Microsoft graph, could you put your hand up? It would really help me.

Thank you. Uh, not a few folks. Okay, so, um, so the Microsoft graph is basically that universal program, uh, programmatic, a programmatically building model that helps to look at all of those data and intelligence services inside of Microsoft 365 I mentioned earlier, essentially those signals and edges that are in there.

So every time you send an email, every time you have a meeting request, all the people who are on that meeting request the things that were attached as part of that meeting, how that document traveled, how that document was created, all of those signals and edges as contained inside of the Microsoft graph. And then is we make that surface available also for third party data, uh, for third party applications to run on top of and allows that extensibility as part of that too. So then we are able to use that, all of that amazing graph data to actually ensure that we're helping to leverage all of this inside of copilot.

So Microsoft Graph is basically providing that data. We're then using things like search over the top of the data and then semantic fabric is also ringing relevance here to make sure that we have a semantic representation of that data. And then all of these components essentially are used inside of our rag model as well.

So with copilot, we're on top of all of this goodness here. Uh, we are using a rag model, so we don't train the LLM on the user's data, right? This is a really tricky thing for like, how do we think about improving the products and services, but we wanna make sure that the customer data is always the customers, right?

We're not using that to actually service and try to improve our models. And so it becomes very tricky of like, how do we understand this, right? We've gotta use a bunch of techniques as part of this as well.

And so when we wanna make sure that we're actually using that LLM to help generate the response, of course we've got some limitations on things like prompt sizes, et cetera, that can come in as well. Uh, we can always reason across all of those, those pieces in the tenant data, but we wanna make sure that we have some semantic understanding here to help us. So we have things like chat history, right?

You know, LLMs are stateless, right? They don't learn from your data. The chat history might have like all of that conversational context in here.

So we think about, you know, all of the different ways that you're creating that and how we're storing that, and then helping you give basically a bunch of consistent and responsible responses as part of this too. And then thinking through, you know, how we're already citing sources here, making sure that default prompt is coming up is responsible as well. Uh, and then ensuring that we don't serve any kind of harmful, uh, content as part of that too.

All right? So that is a little bit on the how copilot works. Keeping tracking with me.

Still just checking. I know it's a lot, but I'm just trying to give you the, the grounding on everything that we need to do. So let's talk about some of the experiences inside of here.

So how do we think about this from an experience perspective? And hopefully this brings it all to life as well. So I have some little quick demo videos here.

So in this case, um, this is inside of Microsoft Word. Um, there's an extensive document here. You wrote this white paper up.

How many times have you had this happen? You wrote up your security audit, you wrote up your postmortem, and then people are like, turn it into a PowerPoint deck, right? How many times has that happened?

So a lot I know. Uh, and so we've always had that experience, right? So we wanna think about copilot basically doing this for us automatically.

And so we can ask copilot to basically go process that document, uh, make sure that it's actually outlining what we're generating and make sure that we, we like what it's doing and then it's creating things like speaker notes as part of this too. And then we can keep iterating basically in natural language, right? As part of this to help you to describe a lot of the information that you want.

So I want a specific slide, whatever it's able to serve that up for me as well, or to do things like, you know, reformat the content that we have too. So lots and lots of goodness as part of this as well. So then you think about, okay, that's cool.

How do we then think about co-pilot to driving that app? And so making sure that we actually build the content with the data that we have. So we have to think about, you know, how we demystify that just a little bit too.

So we have the current state of the doc, essentially, we've got that content that's being created and then we've got a safe execution flow to help produce that final content here too. But this kind of represents a whole series of problems, right? This is a big challenge for all of us as well as folks who like myself, who are product makers.

And we have to think about the language that's being used here. It's pretty low level, right? And it's verbose as part of this too.

Of course our LLMs are prone to hallucination. We wanna think about things that might be like a fabrication might seem like it's right on the surface, but it may not be. Uh, and then we wanna make sure that we're not opening up too much, uh, surface area here for unsafe actions and, you know, challenging that when we're actually calling internal APIs as part of this too.

So we wanna make sure that we're separating each of those concerns. We wanna make sure that we have instant understanding, which is what the LLM is really good at, but we wanna make sure that fulfillment is actually happening by our APIs, which is what they're really good at as well. So we bring all of that together to help us to make sure we understand, you know, the commanding piece that's shown here, uh, thinking about that complex chaining and then we're bringing back that information as succinctly as possible.

We wanna think through like a program synthesis and into LLMs and then make sure that it's abstracted into the code that we wind up actually then generating as part of all of this. Now inside of office, we actually then run this right into a specific language. So we have a domain specific language as part of office and we dynamically construct these prompts and we can translate basically that natural user, natural language user intent here into a verifiable DSL.

And that DSL is easily authored and creates and generates that consistent code for execution too. And then the data is flowing essentially through here. So we use things like an entity classifier as part of this to help determine the types of entities and the queries associated with.

And then we of course are determining if the existing document content is required in order to fulfill the user's request as part of this too. So lots and lots of goodness as part of this. Um, so I wanted to go through a couple areas here.

So first off, generating that prompt itself for the LLM. So the rules that the LLM needs to follow here. So we have a kind of a syntax guide there, essentially for that domain specific language.

We have a lot of code samples that are happening in here for that prompt library that are based on those associated entities and they're coming through and we wanna make sure that the prompt is as small as possible and as targeted as possible. And then we bring in the document context and then also the user query here as well. So lots of lots of pieces coming together.

And then finally, when we are looking at this kind of the output of that LLM, right? We wanna make sure that we have uniformity right? We want, if you ask the same question inside of PowerPoint, inside of Word, et cetera, all of those pieces are coming together.

We want to ensure that we have compact syntax here. Make sure that we're not, you know, minimizing we basically that we are minimizing the amount of LLM queries that are coming as part of this too. Make sure that we can actually consume all of that by the DSL as suppose an input and an output.

And then we wanna make sure that we have document context and awareness of part of this too, so we can easily map those document entity identifiers back to the properties too. So lots and lots of goodness as as here. So a couple of, just couple quick examples as part of this too.

So we wanna make sure all of this code is then transpired obviously into our APIs, it goes into our JavaScript and then it's get executed here. And we do this by helping to control that permissible statements. So we wanna make sure that we're disallowing things like certain and file actions we wanna do very rigorous syntax, uh, checking here, typing validation as part of this too.

And then a scoped DSL as I mentioned, that helps you to make sure that certain assumptions that hold things like autocorrecting, you know, buggy code that might have been generated by the LLM here too. So lots of, lots of pieces coming together. So couple things right within this and the output of all of that goodness is that hey, that code is then generated by the DSL.

We think about it as being generated as very robust, it's verified and it's safe that we'll actually pre complete that task and then we're able to go up a level and think about how to customize this too. Alright, that was just a little walkthrough on that. Still tracking.

Yes, I know it's a lot, but I'm trying to bring you around all the different layers here. And then I'm, the last eight minutes I will finish up and think about how you can actually then extend and think about some of this customization too with copilot. So on this piece of it, like we wanna make sure that you know, you as customers, you as partners, you think about, you know, you can augment copilot skills here, right?

We wanna think about people bringing all of that together and we wanna bring and make sure that we're able to access knowledge from inside an organization. So whether it's a developer inside of a a customer or it's a third party software, uh, vendor that's bringing in. And we wanna make sure that you're able to do that without, you know, compromising anything on the organization's security, governance and compliance standards too.

So we wanna think about how we extend this, right? First off, we wanna look at how we're extending with skills with plugins. We wanna expand knowledge essentially with things like connectors.

And so if you have a source of knowledge, we're able to bring that in and there's lots of different ways to be able to construct that too. So those plugins are kind of a runtime interface essentially, right? They're coming in from a third party service.

Uh, and then we're able to bring that in easily as part of this too. And then you can think through here's that data flow just to show a little bit about how, what's going on behind the scenes here, uh, and how we're thinking through these different areas and how it's bringing that directly back into the graph. And then you're able to use those Microsoft graph connectors, uh, to bring in additional information.

So it helps to kind of usefully extend that knowledge base of the organization and ensure that we're actually helping to ground that company's co-pilot in organizational results as well as basically extending out those data sources too. So on the last piece of this, to think about this from a low-code perspective, we have things that are for essentially business users. You have things like business, like the co-pilot studio that allows you to create an LLM or a custom set of prompts as part of this and useful, uh, useful, uh, interactions across different sources of data.

And then on the proco part of this, we have obviously Visual Studio and you have all of the goodness we do for copilot inside of GitHub as well. And just to wrap it all up, there is an architecture and a data flow diagram here to help with where we think about Microsoft copilot for security and how all of the different interfaces that it's interacting with as well. So lots of goodness in here, there's lots of documentation available here as well.

So the go to pieces. So I've given you a whirlwind of how copilot works, right? How we've gone into the depth of the applications, how we help to generate secure that information that's inside of an organization.

Think about how you do extensibility inside of an organization as well as maybe if you're working for a software company and able to bring that data into the Microsoft graph. And then obviously how we can think about that data flow as part of this too. So I would be remiss if I didn't finish off with just a couple of things here.

First off, there is a Microsoft co-pilot for security. Uh, we announced about a month ago or so, but these are plugins again, they're available, uh, across a number of different surfaces. Those co-pilot plugins are really helping you to be able to create those security boundaries as you want and use the best of the stack, if you will, inside of those interactions too.

So it's a great piece from just an admin side as well as making sure that you're using it every single day, uh, with the privileges that you want inside of the company too. And then the last piece here is just the newest announcements. Uh, we just had these today, so it's a very fresh slide that came in.

I would encourage you to go to that Microsoft blog. Uh, we have a whole bunch of of areas here that we've expanded out, uh, and it's really helpful to understand like that whole stack. So we've got things like Microsoft Purview that's doing things around permissions and policies and role-based access control and sensitivity labels, all the way up to how we think about copilot for security and how we're influencing that as well.

So lots and lots of goodness as part of this, but hopefully you can see the challenge that we have both from a product development perspective, which is where I have to raise and rationale every day, uh, in addition to how we're able to use and mobilize all of the creativity and talents that we have, uh, from folks across the world as part of this too. So with that, thank you so very much.