The AI Gateway Pattern in DevOps: Streamlining AI Integration and Governance with Jack Tysoe at AIE 2024

Okay, so this is, um, a, a talk about a new paradigm that's been coming up quite a lot, which is the, uh, the AI gateway pattern. So this is, um, a new product set of probably quite a few pre-existing products, but also quite a new, a few new business ventures, um, which is obviously, um, a system by which we can take an existing kind of ai, uh, API proxy or some kind of proxy and we can enhance large language models and other AI usage through this, uh, this portal. And it also allows kind of easier reuse, um, and better security for the users, uh, better cost efficiency and a bunch of other stuff which we'll get into.

So I'm Jack Tyso. Uh, yeah, I am a staff field engineer at cog. So that, um, essentially means that, um, in my actual day to day, I am somewhere in between services and bit of programming.

So, um, Kong as an enterprise, uh, in terms of headcount is still kind of small, medium-ish. So we tend to kind of float between these roles depending on what the customer actually needs us to do for something more advanced with sort of inter custom code solutions. And we'll also help them build automation and a bunch of other stuff around their API usage and consumption.

Um, on the left side, of course it goes the other way, which is more towards the, the actual installation and then enablement for these customers to actually get things to work and use our product to the best that they need. So basically I make condu whatever they want it to, which of course then met in the middle, which is why I've been, um, working on this AI gateway initiative and this AI gateway product that we have launched. Um, so mixing the skills together basically over, um, just being an enthusiast for this and really getting into it as it's kind of launched.

Um, I've kind of ended up working on this quite predominantly. Um, so like saying, uh, just a really quick one about me. I used to work in publishing, um, IT stuff so super early, um, BizTalk and uh, camel type integration services.

And from there went into, uh, API management and I think we were pretty early, especially in my business, um, my previous employer in terms of actually getting Kubernetes and sort of investigating this, uh, containerized sort of guest like, boom, that has happened. So we're gonna break this down into kind of what happened at the beginning and then we're gonna talk about what it is like now and the problem that we are facing in terms of what AI gateways are trying to solve. So that obviously then leads basically into the solution, which is gonna be a bunch of different things that we can do to augment and massively enhance large language model usage as it sort of grows and scales.

Um, and then obviously I'll look at just a tiny bit about what's next and there's still gaps and this is still evolving rapidly, so we'll be able to sort of look a bit into that. So this is the beginning. So even just a decade ago, um, there were some with quite good foresight.

So we had, um, the idea of this digital assistant that would be around, uh, somehow wherever you are, and it would sort of be able to, you know, heuristically, try and figure out what you were trying to command it to do. Um, usually from a sort of stencil set of commands kind of in the early days. So voice recognition, feeding into sort of, um, vectorizing what you've said and trying to turn that into commands that actually works.

We also have of course, um, the boom in sort of, uh, SaaS services that we're paying for, um, including entertainment and those kind of things fed quite a lot into how, uh, we structured and productized the analytics of like massive amounts of data in order to, well, I mean there's been some benefits to this and of course, uh, slight effect is come with quite a lot of upselling of various things, but the industry is what it is. And so where are we now pretty much here? So, uh, the open, I guess OpenAI kind of, uh, there was generative sort of language stuff before.

I guess OpenAI is the most prominent in this field for starting this kind of boom, this revolution, but it's happening and, um, the ability to basically now ask a computer, like in language, in your language to do something or change something is obviously hugely beneficial and comes with, you know, a lot of risk and a lot of uncertainty around this. So when we look at, uh, an initial solution, um, what would kind of happen is this came out and companies had, uh, an API gateway, and so they, uh, would try and, or would successfully, um, proxy these models through their existing infrastructure. So what this can actually do is, of course we can proxy directly to an AI or, or some LLM provider or some kind of chat model, for example.

Um, we could authenticate the client or we could pass through authentication from that client and we can collect pretty standard HTP metrics, so like request counts and latencies, but what missing is some kind of authentication like mediation layer, so it could authenticate the language model on behalf of a user. And then the actual authorization happens kind of as part of your existing gateway, um, technology. And so you may already have these systems in place, but the AI sort of backends aren't, uh, obviously aware of that, that, and you have this, um, chicken egg problem where there isn't in a standard API gateway any detail or any context about like, uh, language model and what's happening in one of those requests out of the box.

So there's no LLM aware format, uh, transformation. So we can't fill in defaults or block certain things or do certain things without sort of custom scripting. Um, same with con like I said, contextual settings per model.

We can't kind of provide a single model per endpoint, so we can't just productize a large language model specifically very easily, especially the ones that let the user, um, set their inference parameters and their target model in the request bodies. And obviously there's no data or cost protection, there's no, um, contextual awareness of how, um, LLMs and subscriptions are actually charged. It's all sort of built around the, you know, you are build per billion requests or per latency, that kind of thing, which most LM kind of AI services aren't really done like that.

It's missing a lot more, which should get into. So obviously the problem is that LMS are becoming more than just an API that you would put on a gateway and monitor and watch and let people use for the obvious reasons. But we'll obviously dig into that a little bit.

It's more than just a, like another internal product service thing that people can use as well. So providing or enabling an LLM for a company or for customer use, it carries like large unpredictable risks. So first of all, cost, like I said, LMS usually build by token, not per call or latency, depending on kind of what the operation is.

Um, and like I said, traditional API gateways aren't naturally aware of this yet. Um, we're missing governance, so we can't do, like I said, access control per model if the a, uh, API gateway just proxies sort of directly traffic to whatever you set the target, like open ai co here. Um, and it also means that with no actual language processing or like any kind of, uh, content moderation stuff in there, uh, LMS do carry the potential to be massively misused.

We also, uh, given that a lot of them have quite long running requests, now the async mode's come out, it's a bit different, but back when you would wait seconds to minute for an inference, um, update on like a, a long hat chat history that obviously is holding sockets open, which traditional gateways are trying to get through like thousands, thousands of TPS, um, this sort of thing isn't really designed for that for the most part and obviously user experience. So if you just go, yeah, I've set up a a bunch of, um, cloud providers, we're gonna try some out and you can go, um, user experience differs wildly per, per model or even, you know, per, um, provider and or per platform. And that obviously with different user experiences and purposes and nothing to kind of arbitrate some simple, uh, code from a developer to wherever they want to talk to.

IE likes maybe the cheapest or the best uh, configured model with none of that, the user experience kind of suffers because you're writing a lot of code to sort of balance between these different formats yourself and it's going on the client side. And so obviously a new product category here is gone. So the traditional, uh, proxying, uh, of language model, uh, let's say like in the legacy sense when something like this.

So in terms of something that's automated, you would have a repository and it would control the deployments of your APIs and, um, the actual gateway itself potentially. So in my example, I would, uh, if I instantiate like a new gateway so that we can start running some stuff, I set some data playing values that we're gonna require, uh, in here and I get the right, lemme choose the right image there, that's the one. So, um, somehow helm, repeatable helm situation that can set you up a gateway and can just wait for commands from a control plane, which I can't already have initiated, um, as like a a a con cloud stuff.

But this is something that I have here and we just need to join a new proxy to it. You would have your API definitions inside the files and in here I've just got basic, uh, like I said basic proxying to uh, open AI mushroom co here just for proving where I'm going with this. So if I set that to commit, Uh, so say, uh, make a new gateway and then we'll push that up and open for request on this.

So someone will be saying, you know, when a proxy, and we'll be starting this up. So a typical water automation process is gonna launch to deploy the APIs and the data planes, but I think it's just gonna d what we've changed here so far. Okay, so he wants to create one data plane for com dp, this is what would be created if this is merged.

So obviously some administrator is controlling this and is gonna accept that. And from here we get a release after that has executed. So what we have now is, uh, it's typical DevOps workflow where we have a point in time 1 0 7 and a change log on what's actually happening, which we can then just revert to whenever we want.

So when this is actually activated, uh, I'll go and set that to deploy. It doesn't take long and set that deployment up. So that should have gone, looks like it.

And the control plane should create my simple routes to do that. Okay, let's check this out. Uh, yeah, so we have this, so like I said, what's the, what's the initial problem?

What are we trying to solve with AI gateways? Um, so we're bringing with this, um, the typical use case as in we have Python, SDK and we're trying to do something with this. So the obvious thing here is that, okay, so we can now set the proxy, uh, the base ERL of open ai as long as it's 'cause we're assuming this is compatible, um, to the cong proxy and then the root path that we have on there.

And then we can send basically in what we want to do here. So this, you know, completely typical proxy, um, I go and execute that, here's our inference response. That obviously works as expected, um, an user would expect that to, but then um, we have a second like open AI compatible so we can use, uh, swap the base RL for maestro and then swap the model and we could make a reasonable assumption that is also gonna work.

I need to source the right key. Okay? So what we're also doing here on the client side is we are, without any kind of transformation happening in the gateway, we're delegating these API keys, which can probably do quite a lot of damage or a lot of stuff, okay?

Um, and the users are setting those into their, uh, SDKs and into their programs. So obviously what an AR gateway here is trying to solve is off mediation for one thing and for a second one, for example, if I use my third route, which was the cohere, the issue that we get in this one is that we get, yeah, so the message needs to be one taken along. So this has a completely different input format.

So from here you are now ODing or redeveloping something that you're trying to do, um, with different language models which isolates certain providers and it, it eats time and it's cho So like I said, a new product category here is born, which is the AI gateway. So yeah, a gateway is, um, in our cases where you would hit the typical API gateway and you would hit a set of transformation plugins that are gonna handle and arbitrate everything needed to call that language model for you. And you just send in basically the messages or the inference requests that you require.

It's an emerging co product category that simplifies and, uh, standardizes consumption of LLM. Um, this can run across models, across providers and we set consistent input and output formats such that all models could be consumed the same way, um, and try and stay compatible with re like reasonable amount of sdk. Um, this can obviously do like, uh, LLM contextual logging reporting and some security and it can offer other, because we have this SDK now that's built, we can then also offer that as like a no-code AI enhancement to existing APIs.

So given that the entire framework and structure is there and ready, we can just utilize that. So now if we're actually bringing all this together, we can install our AI gateway and I can show you what we can do here. So what I'm gonna do here is, is set up, um, basically some login plugins so we can see what's going on and I'm just gonna set up some flat routes for, um, the different providers in this example.

I'm running co here again so that we can see what this is doing with the different formats. So now back to Python SDK, where we're able to do here is just call out the gateway and then one on open ai. And we don't even, in theory we don't need the model parameter here because what we've got is all of this is set inside of, uh, the AI proxy.

So what we've basically done just with like sim, a simple configuration in that you can do in our open source gateway is that we've produced, uh, a large language model including all the, and everything that's required to get to it. Um, we have produced that as a product that sits on this route, on this HP path. So what we can do here is then run a standard suite of um, you know, API gateway security and monitoring that kind of stuff on top of this.

And then the AI proxy is just gonna handle and arbitrate that for you. So if you actually execute against this, you see like we get some more contextual information, but we're also therefore because it's um, actually, um, contextually aware of what we're doing here and we have transformers for each of the stuff. We can also then obviously capture, um, wherever you are, just gimme a sec Here.

So we can then capture AI proxy meta, um, including logging payloads, um, if you want to optionally and the token usage and that kind of stuff. So obviously what we're able to do here is for each provider we would have this adapter that does this and for each supportive model we'll be able to kind of make a best effort to actually calculate or otherwise retrieve this data and then you end up with something that's actually showing what people are doing. That was me earlier.

Um, in terms of like which providers are most popular, what's happening, who's using the most models, token usage over time. So you can kind of sum that and you can even see like right down to who's calling what and who's doing what if you so wish to do that. Um, and of course the other benefit, as I kind of alluded to quite a lot is that it's quite easy then to just uh, swap this here for, for example, K here and you'll see this time we've executed against the command model and we've got a completely different, uh, chat response, but this is of course like the same code and everything's the same.

We can just change the endpoint or we can just park a different model. Here we have like, uh, many different ways to kind of isolate. Now because it's contextually available in the gateway, we can now route this appropriately.

So this is where I think the kind of the AI gateway system as a whole, this is probably one of the biggest benefits is that, um, keeping up to date with it as long as you do is that um, in future obviously, um, the developers are gonna be adding uh, support for these things and that's kind of handled by either you know, the product you're using or the product you're purchasing. And in that case, um, it kind of takes the complexity away from the development of um, LM integrations. And so from here, uh, we can basically just now build, you know, features, features expand on this.

So what we would be doing is we've seen single SDK and multi LLM, we can also do something with access control at the gateway. So what you would get here is the ability to, like I said, use the um, gateways, uh, built-in auth systems that you probably already have set and already have working and we can use that to arbitrate um, kind of roles and scopes when trying to call different language models. So if we were to pre provision something that's quite expensive, you can obviously then have roles that can only access that specific thing And that would be done for example, like in this here.

So if I set like API key for use of one, what I've got here is I've got the equivalent in Kong of this with a consumer. So that should I think without that should be protected. Lemme just check um, I'm in the right place.

Yeah, so this becomes unauthorized With the correct key I can pass through. But then of course like I said, you can actually now fine grain protect different endpoints, different models, um, just different instances of uh, whatever you want to pretty much. So in this case what I'll do is I'll protect like open AI specifically this model and I can do that with like an ACL.

So say the access control list you require, uh, you acquire open AI to do that and what AI gateway's obviously allowed you to do is pretty much just like that is then you get 4 0 3 because the credentials are correct, but you can't conce that. So then you would just attach the roles to that or you could do it through your, uh, identity provider or kind of any means that uh, works for you. Like I said, if we kind of now like continue sort of rapid features, you can do things like, because we're able to scrape usage data and analytics from each request, what we can now do is something like token base rate limited.

So in this case, um, I'm just gonna sync everything for the rest of this actually, let me resurface Again. I would normally commit this to gi, but for timing just let me just settle that through. So in this case, if we just execute a short prompt with like a token limit of 10, you see like the first one that goes through, but then the tokens are counted and after that we're B with a 4, 2 9.

So that can also be set per user or per client and you can kind of actually find grain control in terms of cost or in terms of like, you know, compute usage, who is actually allowed to do what. We have a centralized governance with uh, LLM security tools. So both things are included in kind of open source tools that can be quite simple.

Like in this case what we've got here is, um, when I center to this one, there's a hidden system command, which basically pre pens an existing chat history or some kind of context. And in this case it was to only reply in French, but obviously you can see that could be used to kind of like pretext a whole conversation of setting certain limits or things that shouldn't be said or competitors that shouldn't be um, referenced, that kind of thing. You can also protect it even kind of further.

So you can actually set a complete prompt template including the entire chat history and your developer could just fill that in because we're in control of the formats and everything that's happening to actually transmit this data to and from different providers. We are able as an AI gateway to do this. And then you can also block things we don't want.

So if you have a programming bot that you know, uh, the team uses Python, so that's fine to use, but then if you try and use kind of language that isn't supported, you would just get badge requested and then I guess you could optionally tell the client what's gone wrong there. Um, depending on like how much you want to kind of arbitrate or hide, but it's also the up and coming, um, I think there's gonna be a boom of this, a kind of um, maybe like prompt sanitizers or especially category grading. So in this case, if you wanted to use something like as your content safety, you could do that with a safe prompt and it will park it on the way out to check that there isn't any kind of like hate levels or violence levels, that kind of thing.

As you would see in this example, you would get bad requested and then again optionally we'll tell you as the client why you have been blocked. So this will work across any provider, you could just change the path but header send this in. Um, and as your doesn't care because we've transformed this into some kind of strengthening understand and the backend doesn't care because it doesn't know that there's a con in between this.

Again, like I said, we can also audit who is saying what to which LLM. So we've kind of seen that when I was authenticating with uh, API key, I key, um, using the that obviously then when we're authenticating against um, the gateway using the built-in um, API key auth and then um, proxying to the LLM backend, that also obviously records who that was contextually. So you know, you can store the token metadata and emails and certain things and you can actually dashboard out who is doing, who's doing what at what time.

And then there's this final remaining missing piece, which to me I think is probably kind of like I I feel like this is the feature and this is where we're going here. So this gets me most excited is again, the no code in AI enhancements of existing APIs. So what we're able to do, if you look at a little technical level is um, it's in here on, on the way out we're able to um, set some system context and a prompt for, uh, any configured like any supported large language model provider and model to you.

You kind of, you're physically telling it in readable language to to do something or act on something in the request or block something. So what we get in this case is just something simple like for anytime you see a city, can you just enhance it and add the country that goes next to it. And when you would send that, the conversation history is on the right.

Kind of ruined this I know, but you'll see here what my echo server got in the back here is that the gateway was, um, handed this response back as the body from the um, open AI large language model and that obviously was then, uh, decorated with the country that that came with. So this obviously can then be used for things like auditing. So if you want to have your self-trained model that is operating on specific audit rules for documents, you can then obviously like park those on the way out and you can, like I said, physically ask this model to change something in the document or change something and do something in much the same way.

You can do the same. Now we're in control of the request and the response is, again, I'll call my echo server, but what you basically get is in this case, while we wait for that, basically telling it to return me an object that contains like some kind of transformation instructions and convert to XML just for fun. So what we get in this case is I tell it, I've told the gateway, I've told the language model, sorry, to set my response code to 400 to set the content type X application XML and then actually convert what I sent in or what would've come back from the echo server into X ml as well.

So when you would normally be writing sort of custom transformations or auditing rules and systems, that kind of stuff, I really feel like in this case as this kind of grows and gets uh, sort of less latent, um, you were gonna see like a, a boom in this kind of thing where you would sort of, you might just log in and then just say, I'm gonna proxy this route and the URL is here and I need this security, something like that and it will just kind of do it for you and to me that's pretty exciting. So like I said, yeah, what we get in here is that because we control the formats and we have the SDK uh, at hand, what you can do with an AR gateway in this case. Um, is it kind?

Well, I mean it would enable this functionality to be able to park requests either on the way out or on the way back and do something. So that can be used optionally or that you can do that kind of forcefully for compliance reasons as well. And finally, what's next?

Um, obviously product sizing, LLMs and using some kind of portal to provision them out. Um, you can then produce actual detailed reports of usage and what's going on. If you, um, keeping up to date with the industry, I'm sure all the players are also gonna be keeping up with that.

And as stuff changes like we get multimodal, that kind of stuff coming in, I think there's gonna be like a lot more transformation things that we can do, compression stuff like that. And obviously for me the most sign again coming up is a load balancing models. So some kind of intelligent system now that you have the context inside to be able to say like, uh, this, um, particular set of, um, vectors like is faster processed against this kind of model and record that sort of information.

And then intelligently proxy to the either the lowest cost or the least latency. And what's going up for us is rate limiting semantic caching and of course support for more lms. I think that wraps it up and I know I talked about quite a lot, uh, in a short amount of time.

I hope that made, um, I hope that made a lot of sense and you can kind of see where this industry is sort of going and where this new product category is going. Um, I, yeah, I, well thank you all for watching.