Accelerate AI Innovation in the Cloud | The Six Five Summit

Welcome back to the six five Summit 2024, and as hopefully, you know, it's all about ai, the two flavors of ai, which is a continuing building out the infrastructure for ai, but as we've seen increasingly in 2024 enterprises and consumers getting benefit from it. And I can't think of anybody, uh, other than Microsoft Azure opening up our, our track opener for cloud infrastructure for the summit. Introduce Amanda and Mike.

Great to see you. Thanks for having us. Great to be hear.

Absolutely. Yeah. You know, I've been tracking, uh, uh, Microsoft and stuff that you did, uh, for years before.

It was Azure and it's, uh, I guess it was Microsoft server before then. But it's amazing to see, uh, what you have been able, uh, to build. And I think, uh, imagine that Gen AI is, is has been the talk it seems like for 18 months.

You know, AI wasn't just invented 18 months ago. It's been around for a long time, but boy, this generative AI thing has really, uh, heated everything up. And I, I'm curious, how have you seen generative AI adoption evolve, and what are some of the common use cases you've seen your customers, um, uh, adopt and, and maybe we'll start off with you, Mike.

Yeah, absolutely. Well, thanks. You know, I'm a, actually a fairly recent addition to the Microsoft team.

So over those 18 months that you're talking about, that first year I was watching really as from the outside and, uh, really admiring all the great things that were happening here. And for the past six months or so, I've been able to have a front row seat. So obviously an exciting time period.

Um, things have come so far in the last year, and probably the biggest thing that we're seeing is this shift from more of an experimentation phase into a production phase. So all that energy and excitement that we saw over the last year of people really starting to understand what this can do for them and how they might, uh, actually take advantage of it, and now really translating into things that are delivering real value, that puts a different set of requirements onto the customers, it actually creates a different mandate for us as a, a vendor that's really working to build out this tech stack as well. So I'd say over the last year, what we've really been doing is learning, you know, learning from our customers and also learning from the services that we run ourselves, whether that's teams or Bing or GitHub, copilot, all of these are things that help us, uh, become smarter and then actually build that innovation back into the customers, uh, solutions that we can actually bring out for them.

So, you know, we kind of see this, uh, impact around production apps, uh, in really across the board for our customers application strategy. So that's new applications and people that are building in intelligence and using data in a different way on new, uh, applications for their business, as well as taking existing apps and looking for ways to modernize those and modernize those faster, um, in order to, to, uh, infuse them with, uh, more intelligence. And we actually see many customers sort of leapfrogging their modernization strategies and saying, I used to be modernizing just to get to a new infrastructure or a new architecture, but now I wanna modernize really for the benefit of ai.

Oh, that's great. Amanda, did you have anything to add to that? I'm sure you do.

Well, you know, unlike Mike, I've been at Microsoft now for 23 years, so I've seen a, a pretty incredible, uh, transformation of the company, uh, during that time in many, many different dimensions. And in some ways, kind of the first era of the, of the cloud was really about replacing on-premises lab hardware. But I think in some senses, this generative AI kind of, uh, period is really a really finding almost the killer app for the cloud, right?

Yeah. It's how can we take the power of the cloud and actually, you know, turn it, transform it into intelligence that we can bring to all of our scenarios. And what's interesting about it from our customer's perspective, and, and we've certainly experienced this ourselves as we've been building out our own solutions, is there's really five vectors of kind of maturity that have to happen to be able to enact this, right?

The first is obviously the de DevOps journey, which I think everybody has been on for the last, you know, 10, 15 years. The next is moving not just to from to cloud adoption, but to actually cloud native architectures. And then the next one is platform engineering, thinking about how DevOps kind of works in conjunction with cloud native architectures.

Then the other thing that you really need to enable, uh, intelligent applications is data unification and integration, because you also need to be able to reason over the data streams that you're getting from your customers. And then finally, you know, intelligent apps and, and building with AI is in and of itself its own kind of, um, maturity model in some senses where folks really start early with experimentation. They might start with very simple scenarios, and then over time, once they realize the power of it, they start to integrate it into more and more of their user experience and even a lot of their backend operations.

And so when we think about use cases, uh, some of, you know, the most common use cases that we're seeing is obviously summarization and q and a. And this is obviously very often used for support scenarios, um, data-driven decisions, like how can I actually make sure that, you know, we're giving a particular customer the right treatment so that they don't churn and they're retained, or, you know, they're getting the offer that they need personalization so that, you know, uh, your users are feeling more engaged with the products that you're building. And then lastly, automation.

And we're seeing, you know, most of the customers really explore the first and second very rapidly. Um, a few are exploring the third, and, and, you know, a much smaller, uh, percentage are actually really thinking about this as how they're going to power automation of their businesses. But I think that, you know, just like the gen AI kind of wave has really seen an incredible amount of innovation just over the last year.

Um, I think we're going to see our customers really rapidly evolve over this coming year. Yeah, we're seeing a lot of those similar, um, um, applications, uh, in, in, in our research, and I'm so glad you hit on the data with a lot of the enterprise that we run into is once they realize what it takes to do the data, especially when you're crossing the streams between, you know, ERP data and, you know, customer service data or something like that, uh, it, it seems to get, uh, uh, hairy. So, uh, Amanda, I'm gonna keep you on the spot here.

What are some of the key learnings that, that you've taken, uh, from your journey in gen ai, gen AI that, that you might be able to, your customers might be able to apply and learn from? Yeah, I mean, you know, our philosophy is always like, how can we take our learnings and actually build it into our developer tools and platforms so that our customers can also kind of get those, get those same, uh, benefits from what we've learned? And, you know, we've been working on, uh, GitHub co-pilot, which is kind of, you know, one of the most popular gen AI tools that actually helps developers build, you know, write code more quickly and learn about their code basis more quickly.

And we've been taking our learnings from building that, and we've actually been kind of incorporating that into our tools and our platforms. And some of the things that we've learned is really the, the, the inner, inner and outer loop. Like when you think about, um, developer cycle, uh, there's the inner loop where the developer does the edit, build, debug kind of process, and then there's the outer loop where they collaborate with the rest of their team and they do continuous integration and things like that.

What we're seeing is that when, when you move to build features with ai, that actually changes a bit. What you need to do is you need to evaluate how good is the model at, um, at accomplishing the goal that you want it to, to accomplish. And then you need to actually do cost comparisons to understand which model you should be employing, and you need to actually look at the prompt evaluation, uh, to be able to kind of have confidence that it's going to stay grounded in the data that it should be grounded in and not, you know, suggest things that don't make sense for the conversation, um, or are off topic.

And so it kind of starts with really just changing the way that developers actually work. And what we're seeing with that is, it used to be that there were machine learning scientists, data scientists and developers. And what we're seeing emerge is this idea of, of an AI engineer who really understands the models and the capabilities, but also understands prompt evaluation and how to incorporate that into the application architecture, um, to not just deliver great user experiences, but also, you know, great cost and performance and security and quality and everything that we do as application developers.

And so the next thing that comes there is, um, you know, not only do we need to make sure that our application platform really supports what they need so that we can have fantastic cost optimizations and things like that, um, so that they can do, you know, uh, manage the tokens that they need to give to each of the different applications or, you know, uh, be able to kind of take, if there's a computation that comes back from the, from the large language model as an example, that they can actually do that in a safe way without, uh, risking the other tenants in their multi-tenant application. Um, so we need to build a lot of those intrinsics, um, so that developers can really build the, the applications that they want to. And then lastly, it impacts kind of how you think about platform engineering.

Uh, what kinds of, of, you know, governance and policies do you need to enforce across your application state as they start to build with ai? So one of the first things that we started to do inside of Microsoft is we have responsible AI requirements, and we had to make sure that, you know, we were actually applying those filters and those, um, those threat models in a sense, uh, to kind of figure out, you know, what, what did we need to check for before we, we shipped things to our customers? Now this is great stuff, and, um, so many learnings so quickly, and you're learning even inside of, of Microsoft.

And then ultimately you need to tee up the tools and services that make, uh, developers even more, uh, productive and efficient. And ultimately those enterprises do all the magic that generative ai, uh, uh, promises. So how are you building, uh, the right set of tools and, and capabilities to help developers, uh, build gen AI apps?

And Amanda I'll, or Amanda, I or Mike, I'll put you both on the spot here. Yeah. Well, Amanda, Amanda's the expert, as you can see, especially from all the great things that we've learned internally.

And maybe I'll just kind of outline a couple of things that we think about. Um, you know, I think with every shift that we have in applications and, you know, across our careers, we've seen so many, you know, there's so many different things that you have to think about. You have to think about end to end, what changes and how, uh, organizations can really absorb a shift like that.

Um, AI is really an interesting one because there's a greater affinity between the technology and, and the business outcome. So one of the things that we have to really think through is how we help customers identify a clear ROI or a business value for what they're doing. You know, when there's so much excitement in the market about a new technology, there's often this, you know, energy to do something fast, but we also want to make sure that the impact is there and that the value comes through.

So Have, we're thinking through, through mean. Yeah. I mean, after a while, that tops down board of directors c-suite pressure down actually has to come up with something tangible right?

In the end, and valuable. You gotta nail that first one, you know, in POC, otherwise it throws everything off and trust is lost. Yeah, That's actually something that we saw with the cloud generation as well.

There was a lot of mandate to move fast and, you know, move to the cloud really quickly. And I, and then the burden really, uh, ends up on the developers and the IT leaders to really do it right? And to, you know, to convert that into a, a really tangible, um, strategy with value.

I think we're even seeing that even more so within the ai, uh, market right now. So number one, that's one thing that we're looking at is really how do we establish, you know, whether it's just a, a conversation or a model or even templates that can actually help, uh, organizations move quickly and start to build some of the, um, most in demand patterns and, uh, that they need within their applications. But the second thing is really in the tooling.

I think part of what we're looking at strategically here is how do we help the, uh, largest number of developers come, uh, with the skills that they have and work within the tools that they love and actually be productive, be efficient, and build the kinds of services that their business needs. And then the, another area is really looking at how do we remove some of that friction between the development and deployment processes such that we're building applications that are secure, that are scalable, that have the performance, that have the reliability that they need to be in production, but we're eliminating that burden from the developer while also ensuring that app, that application meets the needs of what an AI application or an intelligent application really needs from a scalability standpoint without eroding things like cost. Again, all of that's about protecting value and making sure that the business gets the most outta that, um, uh, AI application.

But Amanda, you have some, uh, great ideas about how we're putting that into motion and, uh, can dive a little bit deeper. Yeah, I mean, I think one of the things that we see is, you know, you start to tiptoe into Gen ai and then pretty quickly you're, you're neck deep, um, incorporating it into a bunch of different, different applications. But what we see in terms of kind of the adoption model of a bunch of customers is that, you know, they really start with generating the ideas of all of the different use cases that they think are going to be valuable for their organization.

And, you know, a lot of them will end up being customer facing, but a lot of them are really more about, uh, driving back office efficiency. Um, so, you know, the first ca first first step is really just evaluating the use cases. Then they start to do proof of concepts with internal apps to see, you know, to, to Mike's point that ROI, like, what is the return, you know, how effective can it be in terms of improving efficiency?

Then they might move once they reach success there to external apps and exposing it to customers. Um, and then moving to production and, and, you know, obviously that comes with a whole host of, of additional requirements as you make it available to more and more customers. But then eventually you have to get back to the point that Mike was talking about, which is, what is the value that you actually think you're providing to your customer and to your business?

And to do that, you actually need to build in a, a loop for continuous improvement. Um, so you need to actually have data feed from, from the intelligent app scenario, um, and then you need that to actually feed back into, you know, the, the, uh, the application that you're building. Um, and then, you know, we wanna make sure that from our perspective, that we're really enabling the developers and the businesses to move through those different adoption stages.

So we wanna kind of, you know, help them explore these different use cases and then give them the tooling and the developer experiences that allow them to really, you know, start easily and explore and then, you know, iterate very effectively and deploy quickly, but then, you know, be able to get plugins like, uh, that allow them to do experimentation in production so they can actually see the statistically significant results of their AI scenario that they're trying to apply it to. Um, and then, you know, to that point, then, then it starts to become this question of how do they achieve the cost optimizations as they scale the application to more and more users. Now, I appreciate that and, uh, great conversations about Microsoft Tools and its customers and your journey, but, you know, Mar Microsoft has always been a partner company, right?

And, you know, we've seen this, uh, which ended up being true Enterprises wanted a combination of, of closed and open models, and we've seen what you've been doing, uh, with different types of model providers. What about on the application, uh, space? Who are your partners with on that?

And maybe Mike will, uh, start with you? Yeah, absolutely. So we see just a rich opportunity to work with a, a big ecosystem of partners, um, around this, uh, overall AI tool chain.

And our goal is to give developers the flexibility to work with the tools that they need to, um, and then make it incredibly easy for them to, uh, bring those tools into something that's integrated and feels very natural for them. So what we've actually been doing is working with a, a growing set of partners, um, think of them as AI ISVs or AI tool providers that, um, really provide discreet functionality. And a lot of this functionality is actually really widely adopted by that, uh, particular AI engineer that Amanda outlined earlier.

So we're seeing a lot of affinity for some great technologies out there. And in fact, at our Microsoft Build event, we announced some new and expanded relationships with a set of core providers. Um, a lot of these are coming together in what we would consider to be an AI stack, um, focused on very specific use cases like building a, a rag enabled application.

Um, so just at Build we announced new relationships with Arise and Lang Chain Llama Index, um, expanded relationships with hugging face and Pine Cone. And all of these are great services. They're, um, in use within the, uh, type of, uh, AI engineer that we think is on the front lines of building these services.

But beyond those relationships, one of the things we really want to do is make it incredibly easy for developers to get access to those services. So within each of them, we're building unique integrations. Some of those are direct to Azure, some of those are through GitHub.

Um, and, uh, we're also looking at ways that we actually embed those technologies into the templates I mentioned earlier, so that the building blocks for the services and applications that, uh, developers are building, um, actually have, uh, prebuilt and integrated access to those services. Um, our goal here is really for the right services for a small set to give developers a first party experience, um, so that they feel like they're operating within an integrated environment. And in the end, it's about really helping developers use the things that they love and the skills that bring the skills that they have and be incredibly effective at building, um, the, the applications that they need to.

Yeah. Amanda, how about, uh, how about the, on the, I don't wanna say the nuts and bolts side, but, um, you know, any, any adders from you? Yeah, I mean, I think, I think to Mike's point, that that point around having it be as integrated as possible is a really critical dimension of it.

Because, you know, if, if developers, you know, developers oftentimes have to interface with a huge number of disparate tools. And every time they switch tools, um, that actually impacts their focus and concentration and it, and what we see is that it takes about 23 minutes every time they do a context switch to regain that focus and concentration. And so what we're really trying to do is to make it so that all of these great, uh, you know, tool and platform, uh, vendors are able to integrate into the developer experience where developers hang out, right?

Which is in Visual Studio Code in, you know, visual Studio in, um, in GitHub. And so one of the things that we announced at Build is that we're making GitHub copilot extensible. Yes.

And so we can take basically all of these different types of extensibility, uh, scenarios for all of these different ISVs and incorporate their experience directly into getting copilot. So if you think about, you know, the pair programmer experience that GitHub copilot provides to you now, it has, you know, Docker knowledge, now it has Pine Cone knowledge now, it actually could have knowledge about the specific frameworks that are, that are internal to your organization, um, and give you, you know, better insights as a developer, how you should be building that application that's going to kind of adhere to the best practices of what it means to use Pine Cone or what it means to use ate Yes. Or, you know, whatever.

So, you know, our goal here is to make sure that ultimately that developers who use our platform, uh, which is, which is an open platform, allows you to build any, any programming language for any platform target, uh, you know, but it is, is open and modular and an extensible by design. And so what that means is you can build, bring the best of breed tools for any scenario or job to be done and incorporate that into your developer experience. Yeah, I really do like the meet your developers where they are, right?

You can start with a Microsoft experience and then go from there. Or you can start, uh, with a Docker experience or Pine Cone and go from there. I, I like that.

It's almost like, how, how could you say no? Right. So, uh, early on in the discussion, uh, with Azure and ai, we heard a lot about Azure AI services, essentially your app platform.

Mm-Hmm. We also heard about Azure Open ai, and I'm curious, what is your strategy for making Azure the go-to platform, uh, for developers building apps? And by the way, I think your initial value prop was, was really good, but maybe just reiterate, you know, what your strategy is and, and what you're doing.

Yeah, I mean, I think it really starts with, first of all, making Azure a great place to build, to build AI itself. And obviously OpenAI was built on top of the Azure infrastructure. You know, we have many, many ISVs that are building their own ip, which is, which is itself an AI model that are using the same core Azure infrastructure to build that, that, uh, technology.

Um, but then on top of it, we wanna kind of make sure that, you know, Azure really provides a great way to explore the breadth of all of the different models that are out there. Yeah. So we have, for example, um, with Azure, Azure AI studio, like the model, uh, model as a service kind of model, uh, catalog in a sense so that you can explore different models and actually swap them out so that it's really easy to compare the performance versus cost, um, of one model versus another, including not just models that are served up by, uh, by Microsoft on Azure, but also client models that you might actually, uh, like the recently announced, uh, fee model, um, that you might wanna have run locally for cost optimization purposes if it's still kind of meeting your goals.

And so with that, we have to kind of make sure that we have the evaluation loop really well supported. Um, but then beyond that, we wanna make sure that not only are, do you have the AI models being served up by Azure and allow you to have cost optimizations for your solution, um, but also that the application model is also primed to host the, the unique requirements that are really coming in, uh, with the, in the era of intelligent apps and ai. So, um, so we have to kind of make sure that, you know, we have support so that they can get the scale and the reliability that they need.

And one of the things that's so interesting and cha challenging in some senses about working with AI is it's not, it's not discreet and it's not as predictable as, you know, our our previous era of, uh, of technology. And so a lot of times what you have to do is you're basically working in a stochastic world and you need to kind of create, you know, guardrails, but also you, you sometimes need to dynamically reve, you know, which model you direct traffic to based on the prompt that's coming in from the user. And so being able to do that with, for example, our, uh, gen AI gateway and Azure, uh, API management, uh, really helps with that.

Um, and then also, you know, be able to kind of do things like extract, uh, you know, extract and store document data, have, uh, have memory and history, um, so that you can provide more context to the model as you're trying to come up with the right response, um, you know, provide summarization capabilities. Um, a lot of this kind of really is about the support for, uh, the interactive, you know, human interface of these applications. But, uh, but you know, as I mentioned, a lot of the scenarios are also back office scenarios to improve workflow processes.

And for that you also need to have data ingestion and data, data cleanup tasks where, you know, large language models can be incredibly helpful for that. No, that's great. And this has been a great conversation and, uh, we're coming up to the end.

So we have time for one more question here. We talked about the incredible past related to developers and Microsoft server, by the way, I was, I worked for a company that was your first Windows nt, uh, and Windows server, uh, OEM. But I want to talk about, uh, the future, uh, here on, on what's, you know, not looking for you to give us your entire roadmap, but, but what should we expect in the future?

And maybe we'll start off with you, Mike. Yeah. So I think, uh, you know, if we look in the immediate future, what we really see is this idea of just intelligence in applications.

We think that's just gonna be table stakes. You know, that's going to be the norm for most applications. In fact, you know, in general, the conversation is about how do I get, you know, the vast majority or all of my applications to have some sort of intelligence built into it, whether that's a new app or modernizing an existing app.

And I think there's just gonna be this massive wave of AI applications that are then coming in, and we have to start thinking about the infrastructure that underpins all of that, and ensuring that it really provides all the optimal, uh, requirements and qualities that those applications need. So the immediate future is really about helping everyone move to production and do it at scale. And that means we're thinking about how that dev process continues to evolve and become more sophisticated while also becoming more simplified, um, how we remove that friction that can exist between developers so that they can just continue to focus on building their services, and also making sure that the new requirements for these applications, whether that's around security or performance, or getting a better lens on things like cost as these applications really scale and production, that all of those are managed and they really have what they need, such that the outcome for the business is one, um, that meets their requirements.

But that's just the beginning. I mean, we have some great things coming and, um, I'll let Amanda talk a little bit around what's down the road. Yeah, I mean, I think in a sense, I think this is the year that Gen AI is crossing the chasm, right?

For the last year, uh, you know, it's been a lot of kind of bleeding edge folks incorporating it into their solutions. Yeah. But I think this year what we're going to see is that, that, uh, you know, folks who have not taken a look at it are going to take a look at it, and they're going to get started very quickly with, you know, default templates and things like that that we have, we are making available in GitHub and, and targeting Azure.

Um, but, but also what we're going to see is sophistication, more sophistication of the folks who are already building with, with generative AI and more, um, standardization of best practices across the entire industry. And so we're going to see, you know, in many cases, uh, you know, uh, different best practices like experimentation become really solidified, um, or, you know, evaluation loops, uh, just become kind of normal, you know, part of your, your developer stack. No, this is great.

Uh, Amanda, Mike, I really want to thank you for the opener for the cloud infrastructure track here. Um, I, I love the, his, I'm a student of history and not that the past always repeats itself, in fact, many times an inflection point it doesn't. But, uh, I think we all have a really good idea of what you're doing, uh, at Azure.

And by the way, if you wanna know where to start, hit the Azure Developer Template Gallery. Uh, there's a lot of great stuff, uh, in there to do it. So Amanda, Mike, thank you so much.

Thank you. Thank you, pat. Yes, this is Pat Morehead.

That was the track opener for the cloud infrastructure. It's all things AI six five, summit 2024. We appreciate you tuning in, take care.

And hey, you can view all of the amazing, uh, uh, videos and tracks that we have out there. Um, and they'll be available on YouTube in a couple weeks too. Take care.