Won’t You Take Me To Functiontown? With Craig Jellick at AIE 2024

Hi, I am Craig Jelic, uh, director of Engineering at Acorn Labs. And this talk is all about function, calling in LLMs. So let's start off by just level setting with what function calling is.

Uh, this definition is pulled from open AI's documentation, and basically what it's saying is you can describe functions to the LLM and it can tell you when to call those functions. You can give it the response to calling those functions and it can use that to further the conversation or help derive answers. Very high level, but just wanted to level set on what function calling is in LLMs.

To highlight this with a somewhat contrived, simple example, uh, we can use getting the weather. So the user might say, can you tell me the current temperature in Phoenix, Arizona, by the way, I have a tool called Get Current Weather, and it accepts a parameter called location. That alarm goes, sure, why don't you call, get current weather with the location parameter set to Phoenix, Arizona and give me the result.

User then, or the system representing the user then makes that call and says, okay, I've made that call. The result was sunny 110 degrees, no chance of rain. LM says, great.

And since the original question you asked the LM was, can you tell me the current temperature in Phoenix? It says, the current temperature in Phoenix was 110 degrees Fahrenheit, and that's the entirety of the interaction. Now again, that's somewhat contrived, but it gives you sort of a feel for the flow.

Next, I want to walk through what that looks like from an API perspective. So here we're looking at the OpenAI, uh, chat completion, API. Uh, throughout my examples here, I'll generally be using OpenAI as the reference point.

Uh, there's a couple reasons for that. Uh, the first is it has basically the best, um, function calling that, that we've seen. Um, secondly, a lot of the open source models, really the vast majority of them you'll notice actually support the open ai uh, specification.

So if you have code that is working against the open ais API spec, uh, generally you'll, you'll be able to find open source models that support the same API specification. Now, that's not to say they all do an equally good job out of it at it. Um, many of them miss nuances of the API or just implement things slightly, slightly differently.

You also have to take into account the large non-op source proprietary models. Uh, those generally may or may not have an open AI compatibility layer. Uh, once you get to a certain size, a certain, you know, level of users, you don't necessarily have to follow suit.

With all that said, let's look at what this API looks like. First I tell, uh, I set the system message. You help the user figure out whether that's just letting the LLM know what its job is.

And then I send that user messages, that mu user message that says, what's the weather in Phoenix? Now, this next part is particularly important. It's where we tell the LLM about the, uh, functions that are available.

You can see get current weathers there. It has a description, it has its parameters, and those have descriptions as well. Uh, defining these properly and giving good descriptions is incredibly important because that's how the LM is going to decide what to call when and how.

So if you give a poor description or you have a messed up parameter, the LMS not going to be able to help you out, in particular in your descriptions anywhere. You can supply things like, you know, very short examples for parameters, that sort of thing. Those things will help the LM out a a lot as well.

So next, in this interaction, uh, the l LM returns this assistant message that says, can you call get current weather with these arguments? And, uh, here's the idea for this call ID for this call. That's important because when I finally say, yep, I called it, here's the answer, I supply back the same id, and in that way the LLM can match questions to answers or function calls to function call returns.

So that's what we're looking at here. We've kind of gone all the way through, can you tell me the weather? Sure, call this function.

Here's the result back. And finally we, we end with the l LM telling us what the temperature was. Okay?

So that is a very basic setup, gives you an idea for the flow of function calling, but going beyond those simple examples, what is the power of function calling? Well, you can do all sorts of interesting things. Uh, with function calling LLMs.

You can integrate with web services, local clients and applications, database queries can even drive a web browser or integrate with other LLMs and their capabilities. And what I wanna do next is actually walk you through examples of all of these to do. So I'm gonna use GPT script.

So this is an open source we've, we've developed here at Acorn Labs. And the idea is basically it's a new scripting language to automate your interactions with LLMs. Uh, it's largely natural language, so it's super easy to use.

It's kind of like if you took, you know, what you would do with, uh, chat GPT, but then were able to script entire workflows with it. Uh, in, in addition to that, you can kind of weave in and out, you know, calls to tradit to traditional languages like Python, JavaScript, and you know, we even make it easy to integrate with those external, uh, APIs as well. Um, it's open source free to use, just download and start using it.

The real power here is the LLMs and their function calling. So we're really just enabling that. There's plenty of other, uh, projects out there that enable LLMs in a similar way.

I'm using this one because I'm most familiar with it and, uh, can really give you good examples, uh, quickly and easily with it. Okay, so what I wanna do next is actually switch to live examples here. So as I mentioned, the first example was going to be around web service calls.

So if I just list the files we're looking at here, we have a pets dot g PT and a pet store m um, I am going to go ahead and first show you this GPT script file and you can see it's very simple. So this first line here is basically saying, Hey, there's an API running at this endpoint. Can you learn that?

So GPT script can go out to that kind of learn the API turn the API into functions and, uh, then make those functions available to the LLM. And then this is the question I'm asking. List all the pets.

What are the names of the dogs? I'm going to launch this. Uh, it seems like sometimes this one takes a minute to run, so I'm going to kick that off.

And while it's running, uh, we will take a quick look at that schema and I can explain some things there. Okay, so let's open up petstore yaml. So you may be wondering why the example is Pet Store.

This is Open API's canonical example for, uh, you know, what their specification can do. So open API is a, you know, standard specification for, here's how I describe my rest web service. And in their documentation, you'll see this pet store example all over the place.

We went ahead and implemented this, um, spec, put it on the public internet to make these, uh, you know, example integrations. Super simple for folks to run. But you can see here it maps very well to function calling in, uh, LLMs.

So first thing is paths, right? So you can hit slash pets and the method you can call on it is get that becomes basically the function name. Get pets, the summary becomes the description.

And parameters map directly to parameters. So you know, anyone who's writing a good open API specification for their rest API is already doing everything they need to do to integrate that API with ai. Uh, just to elaborate on this example a bit more, we see you can create PEs via post and you can list individual pets.

Uh, so that's sort of at a high level what's going on. Let's check back over to see, oh yes, it returned already. So let's walk through what we're seeing here on the screen.

Again, the, uh, question I ask is list all the pets and what are the names of the dogs? And, you know, GPT script went out, grabbed that schema, transformed into functions, provided those functions to the LM to call. We can see the LLM told us to list pets.

This is a response that was returned and then we returned to the LLM, but you can see there's all kinds of pets in here, dog, cats, fish, everything. So the uh, LM actually processed those results and was able to, without any additional, you know, knowledge or information extract just the dogs from that and tell us the dogs. And we got the out output output.

The names of the dogs are Rex, buddy, spike and Rusty. Super powerful. Again, kind of a contrived simple example, but you can imagine how powerful this can be.

When we were developing this, we integrated with a, uh, with the Spotify API and that API is huge. It has tons of things, you know, tons of methods you can call. We didn't need to learn it, we didn't need to actually know in depth that API, we just needed to have the LLM basically learn it.

And then we can say in natural language, here's what I wanna do, how do I do it? Can you tell me how to do it? And worked like a charm.

So that's the, uh, web service example. Next, we'll go to ACL I. In this case it's going to be the GitHub CLI, which most, uh, developers should be familiar with.

So if I just show you what this GPT script looks like, it's very simple. Helps you with the GitHub, CLI, uh, and then the instruction that we give to the LM is use the GitHub CLI GH to accomplish the user's request. The function that we make available to it is execute GH commands.

And you could tell the syntax is obviously not, uh, the JSON you would pass directly to, uh, uh, the L-L-M-G-P script itself takes, you know, takes care of translating that. So the name of the function becomes execute GitHub commands description, very simple, execute supplied GH command, and then the parameter is command. And here's the description of that.

Uh, you can see an example of including a good example so that the LM ex, uh, understands exactly how to pass that parameter. Uh, in this case, I just wanna make sure that it's not giving me the GH command as well because I'm doing that. Uh, when I make make the call, you may have noticed here, there's this context line learn GitHub, CLI.

What that does is, you know, I wanna make sure the LM knows as much about the CLI as possible so it can form good commands. So context just says, take whatever the output of that call is and prepend it to the messages going back and forth between, um, the LLM and the user. So in this case, I'm just outputting the help.

So, uh, as the first example here, we will ask the script to clone the repo GPT script. Um, and you can see that's the help command going by. Um, and then it went ahead and did it.

The repository has been created. Let's see if that actually is true. Yes, the repository is there.

And then, you know, I can ask it a completely different question. Uh, let's say, what are the first three issues open in that same repository? So we'll do that again, it's getting the help.

Um, you can see it formed the command. Let me see if I can find where it formed the command. Uh, so yes, this is just the response.

It's actually not showing. I'd have to go into debug to show more details there. Um, one thing I'll say is this returns super quickly.

Uh, that's partially because GPG script knows how to cache responses. So just to kind of show you it in real time, we'll disable cache here, rerun the command, and let's see what we have going on there. Um, now you can actually see sort of how long it actually takes to make these requests and we can see the requests as well.

So if I scroll back up, execute GitHub command, here's the, you know, parameter for command. It runs it and it's probably, yes, it's already done. Uh, going through the output there, super powerful.

You can do this with any local CLI and, you know, have basically this very easy way where you don't have to learn these CLI in detail and you can make things more and more complex. So this next example is going to be driving a browser, uh, using an LLM. So if we just look here quickly at what this GPT script looks like, it's pretty straightforward, um, we're going to look at a tweet and then get the sentiments expressed in that tweet, uh, perform the following in order, get the text of the tweet referenced by the URL, then analyze sentiment sentiments.

But you know, uh, it's not easy to use the Twitter API so we can do so through the browser. And I want to show you very quickly, uh, what the browser tool looks like. So this tools, GPT script AI is um, sort of our tools hub.

Uh, you know, community members can register tools here we have tools we support ourselves, one of which is the browser tool. So if you come here, you can see all these different examples of using it. Um, you know, we've gone as far as even being able to create emails and in Outlook using this tool.

Uh, if I go to the tool itself, you can see it's actually a collection of tools. So you have these low level functions, browse, get page contents, login, click fill, enter, scroll, all these things. So you can kind of give high level instructions and the LM will know it has all these things available to it.

We'll figure out what to do. So jumping back to our example, we say perform the follow actions in order, get the text of the tweet referenced by URL, uh, then analyze the sentiments. So let's go ahead and run that.

Um, find the right command here. Here we go. Alright.

Right. So this is gonna figure out, hey, I need to open the browser. Um, it's gonna find this tweet, it's me tweeting about being supportive of my son's haircut.

Um, it downloaded that information and then it went ahead and did the sentiment analysis of it. So says it was positive humor, humorous and supportive. Um, nature of it was personal reflection and comparison.

Super powerful. You can imagine scripting more and more complex interactions with the browser. Um, and we're very excited for this tool and just continuing to enhance it, uh, as time goes on.

Alright, so next, uh, I mentioned you can use this to script database queries. Uh, one thing I'll say is this example is actually fairly more complicated than just a database query. Uh, but let me show you what it is.

So I'm asking the LM to download this sample. This is all using SQ l light. So very popular, very well known, uh, database solution, asking it to download their sample database, um, extract it, uh, inspect the schema, uh, and then run a query that tells me, uh, to find the artist with the most number of albums.

So you could see there from that, I don't really know anything about how to talk to SQL Light. I'm not giving it any particulars of how to form the query, but beyond that, I'm asking it to download, extract, uh, run this, the, these commands. Um, so it's all very complicated.

So let's run this one. Just go back in history here. And you can see it's using these kind of built in tools, you know, so we have predefined tools for, as you know, downloading and exec and commands, removing files, that sort of thing.

So, you know, called an unzip command. It called this command. Here we're now running SQL commands.

That was to get the schema, um, so it could learn the schema. And now you can see it's formed the query and is executing it. So if we just wait a second for the answer here, We can see it went ahead and came up with the answer.

I pre-checked this, it is correct. Uh, the artist with the most number of albums in the database is Iron Maiden with 21 albums. So again, if we, uh, I'll just kind of go through the previous tool calls.

Uh, you can see it's doing tool call remove. Before that it was doing exec to, uh, call the, this is how it actually executed the MA command to uh, do the, do the query itself. Um, and then I want to, it's going to, I think search in my, uh, terminal here is being a little bit funky.

So I'm gonna go ahead and skip over that, exit that out, and we'll move on to our next example. Uh, anything else I wanted to say about this one? No, that is, you know, at the end of the day, uh, what is interesting about, interesting about this, I was able to learn the schema form a query, execute the query, give me the answer all while simply, you know, all I asked from it was this simple planning list question, uh, form and run a SQL query SQL query to find the artist with the most number of albums.

Last example I mentioned is talking to other LLMs. Um, so for this example, what we're going to do is have a GPT script that actually will as part of what is doing, generate images. So this example we've actually built a whole app around, so I'm going to go ahead and bring up the app to show it to you.

ai. And earlier today I prompted this with a story I just said, uh, a story about a man who uses a rubber duck, talks to a rubber duck to solve his problem. And let me jump back to here.

It went ahead and generated this story. We'll come over here. So it generated all the text, but then it sent that text to the image generation, uh, API and it generated an image for the story.

You can see I asked it to do three pages. So it kind of kept that same look and feel. Uh, the look and feel is actually a little bit different, but you can see that the character sort of looks similar across these and that was part of the prompt engineering to get this to work properly.

Last one, kind of hyper realistic here, uh, but you can see the theme across the entire story. So how did we do that? Mm, this storybook app is here and we can open up the, uh, storybook GPT file and look at that a bit.

So this is very complicated. Let's just scroll this through this and see, you know, there's a lot of text here to get this, to do this perfectly. It took a, it took a lot of prompt engineering and there's lots of tools available.

Uh, and really it's an entire application kind of leveraging to a high degree, uh, LLMs. So this top level pool is kind of, sort of the overall of what we're asking it to do. So, you know, generate a story.

This is supposed to be sort of children appropriate stories. So we're telling it that, um, create the necessary, you know, files so that the story gets persisted. And we could show it in the web app from the prompt call story writer to write the story.

Uh, if then review the story if it's not appropriate, give up and then for every page in the story, do what is necessary to illustrate it. And so these sub tools, we have one tool that's a story writer. Then we have this illustrator tool.

Um, and it's very interesting, it's making use of this image generation tool we've written. Uh, but it's prompted even further, you know, 'cause we wanted to get a very consistent, uh, experience generating these images. But if we look at the image gen image generation tool, we'll see that, uh, let me jump back to it here.

It's really just a wrapper around open AI's image generation capabilities. So here's a few examples. You can do very sim simple things with it.

If you go to the GitHub repo, you can see it's really just, you know, a simple python script to talk to the image generation API. So yeah, that's super interesting, uh, 'cause it shows just how complicated these things can get. You know, we're creating files, we're talking to other LLM capabilities, we're coming up with a story and we're stitching it all together into this one cohesive application.

So that covers the examples I wanted to share with you. The next thing I wanted to do was just talk about sort of our experience with function calling and, uh, a benchmark that we're developing to evaluate various LMS function calling capabilities. If I jump back to my browser here, this isn't public yet.

Uh, it will be public either by the time this recording goes live or shortly thereafter. Um, but this function calling benchmark, uh, basically runs a series of tests to emulate these complex function calling scenarios that we've seen through our usage of the capabilities in GPT script. There's other good function calling benchmarks out there.

One in particular the Berkeley function calling benchmark. Um, really does a good job of hitting a wide array of use cases. Um, this kind of picks up where that leaves off and extends into what we think are more complex use cases.

So in particular areas we're focusing on is, is, uh, one big one is chained function calling. So that's when the output of one function becomes the input to the next that you could see that in our examples, how important that is, kind of like maintaining that flow from one function call to the next. Um, and that's kind of, you know, where we see some models doing better than others.

So the basic, this is kind of very straightforward, these are sort of, you know, all models essentially pass these. The prompt is call funke with one and respond with the result of the call. You can see we, uh, describe sort of the functions available.

We then define our expected function calls and our framework. We'll kind of match up what the LM is sending back to us and make sure it's giving us the calls we want. And then finally we use an LLM to judge the final answer.

So in this case we use open AI's GPT-4 as the judge. It'll see what the LLM under test has responded with and then decide if it meets this criteria. Again, the basic, uh, the basic test cases pretty much everyone passes negative is, you know, we felt that was important.

A negative test case is when we expect the LM to recognize that it can't complete the task. It was asked to do with the most basic example again, poly fun A with fu. Why can't it do that?

Well, the parameter to funk A is an integer and obviously food is a string. So we're expecting the LM to tell us, one, we're expecting it to have no function calls and we're expecting it to tell us, uh, something went wrong or I can't complete that operation. Those are the sorts of things we're testing for, because that's kind of the more complex situations we see these things getting into.

If we jump ahead here, you'll see we represent the semantic test cases. All of those generic ones I just showed you, fun, a funk B that's not really indicative of indicative of the real world. So, uh, these semantic semantic test cases are more indicative of sort of real world function calling.

Again, start simple, get the weather in location x assert the response, but if I scroll all the way to the bottom, they, these things get sort of increasingly complex. This one retrieve the list of potential sales leads and for each, add them to our system. Then send an initial contact email to each lead.

Finally, assign a sales rep to each lead. Uh, you know, this is complicated for a couple reasons. Basically it, it's sequentially chained.

So each, each item in that sales lead response, you need to call add to system. You need to call, send initial email. You need to call assign sales rep to, uh, and in addition to that, the response is, you know, this basically CSV style, new line delimited set of, uh, uh, set of contacts or leads.

So the LM needs to be able to parse that and then iteratively go through and make these appropriate calls. So you, you can imagine this is much more complicated than those simple examples. And this is really where we see the best of breed models separating themselves from the rest of the pack.

In general, sort of, you know, the lower down this list you go, the more separation you see most can do these ones. And it just kind of slowly dwindles or pretty rapidly dwindles to the point where, you know, these types of calls, um, only the best models can perform. I'm not going to get into specific results in this talk.

We want to just vet the test cases a bit more to make sure that, uh, we're being fair, the test cases are fair and then, you know, just work through any other issues before we share out these, uh, results. But we think it'll be very promising, will be a good way to help you evaluate the best models for your job. 'cause it may be the simple models that just do simple function calling may be perfectly acceptable for you.

Uh, and they're generally going to going to be cheaper, potentially even faster. So, you know, everything's a trade off. Okay, with that said, the last thing I wanna do is just jump back to our presentation here and give you a few more details.

So once again, GPT script itself is open source. Uh, you can try it out. It's a great way to learn more about function.

Calling in, uh, uh, you know, any LLM, um, you know, if you want to kind of learn more, you can follow GPD script AI on uh, x. Um, I build the cloud is our chief architect and GPD script is sort of his invention. If you're interested in that benchmark when it is made public, it'll be here and we'll have an announcement log, kind of going through all those results, um, in more detail, kind of really getting into the nitty gritty of which models are are the best.

Um, and you can again follow GPT script or myself on Twitter and we'll certainly, uh, uh, announce it when, when we make this available, uh, for everyone to read and learn about. With that said, uh, that concludes my presentation. I hope you learned a lot about function calling and lms, and I appreciate your time.