Improve Your Enterprise LLM Code With Fine-Tuning with Shomron Jacob at AIE 2024

Uh, hey everyone. My name is Sharan. I'm the head of Applied Machine Learning, uh, and Platform at Iterate.

Today, I'm here to share our insights around how you can improve your large language model called Projects with Better Data and fine tuning. Uh, I'm delighted to share insights on this, uh, because this is something that we have applied internally here at Iterate, and it has worked really well. So, uh, today we'll explore strategies to enhance the accuracy and performance of large language models, particularly for complex tasks and specialized coding tasks.

Right? Uh, that's something that we do all the time. So a little bit of introduction around, uh, how large language models work.

So everyone's aware of large language models, uh, such as GPD three, GPD four. Uh, we have a new one as well. Uh, the GPD, uh, four oh is, that's what you call it.

Um, and they have shown remarkable capabilities in generating code. Uh, I have myself used these models to generate code, and they work fabulously if the task is simple. But when the task is complex, that's when you start to see issues, right?

So that's what we are covering in today's, uh, session. Uh, they can automate repetitive tasks, and especially in coding, they can assist you in debugging, which is very helpful. Um, they can also create applications from scratch using natural language processing, but when the application requirement gets, uh, a little bit complex, that's when a lot of issues starts to happen, right?

And that's what we'll cover. Uh, so these model usually struggle with intricate code generation and structures. Uh, they struggle with specialized libraries.

Uh, if you're working with such, uh, at your end, um, they can struggle with complex application demos as well. Um, and then one of those significant issues which everyone's aware of is hallucinations. It happens a lot.

It happens all the time, um, where the model is either generating irrelevant code or irrelevant information, uh, and then you just can't differentiate, right? You just don't know if it's right or wrong. So it's a, it's a problem that everyone's aware of and the entire AI community is working towards solving it.

Correct. So we will cover that session as well, correct? Uh, a little bit around, uh, what I just talked about.

A small example is if you ask a language model to generate a simple hello World program, it'll work really well 99% of the time. Uh, but if you ask it to integrate a complex library like TensorFlow, uh, you will see it starts to make mistakes and it starts to add things which are irrelevant and things like that. So that's what we'll be covering, correct?

Uh, that's exactly, uh, what we have done internally at ADE to see how we can solve that. And that's what I'll talk about a little later. Uh, conventional fine tuning issues, uh, again, fine tuning is one of the ways of, uh, improving your model on your data that you have, that you're trying to use the model for.

And one of the most popular ways of doing that is fine tuning where you use question output pair. Correct. So it's one of the traditional fine tuning approaches where you show it a question, you, and then you show the answer for it and you fine tune it.

Correct. Um, so for example, you might provide a prompt, like generate a function to sort an error, and then you will show a code that will do it. And that's how the model learns from, correct?

Uh, it'll look at the code, it'll look at the prompter, and it'll say, okay, I know how to do this next time. So while this can work for simpler tasks, uh, it often leads to hallucinations for very, uh, complex tasks. Uh, correct.

So for, in my example, if you, it's, if it's an area to sort, um, or if it's an area to add to numbers, it'll work. But you go one level down and you start asking the same model to integrate some sort of a TensorFlow function call in there, it'll start to not do it well. Correct.

So this is very normal. So for example, uh, you can try this as well, uh, if you ask it to, uh, a language model to generate, uh, rest API call with lask, uh, before fine tuning the output that you see versus after fine tuning the output that you see, which it'll differ a lot. Correct.

That's the reason why, uh, fine tuning happens because you can see the stack defenses before, uh, uh, in the model output before it does it and after it does it. Correct. So what can be done differently?

Uh, what can we do different at, uh, improving this problem? Correct? Uh, and that's what I'm about to cover, where we have applied certain data representation strategies added rate, which has improved, uh, these models at generating complex, uh, code, correct?

And it, it's way better than the regular, uh, models that we have. Uh, so, uh, this is what we call it as a data set representation strategy, correct? Uh, we have tried, uh, a strategy where you break down the training data into smaller descriptive components, uh, then you use those smaller descriptive components, uh, to fine tune the model again, and it's done in a specific way, which is what I'll cover as well.

Uh, and then that particular model then perform really well at generating the code, uh, as opposed to the regular model. Correct? So, uh, first step that we propose is, uh, breaking it down into, uh, training data.

Uh, I mean, breaking down your training data into smaller descriptive components and structuring them sequentially into complex code. Correct? It's a, it's, the sequential structuring part is also very important.

Uh, so people who code would know that once you're writing the code, uh, for example, a Python code, you can, you have to first import before you can start using that library, correct? You cannot use the library and then import. So sequentially structuring the code is very important when you fine tune as well on, on certain code, uh, libraries.

Correct. And that's something that we have tried and it has worked immensely, uh, at rate and the model are doing really well. So, uh, break down your data into smaller chunks, then sequentially structure it and then you fine tune it.

So that's what, uh, I'll, I'll talk about in a second. Correct. So, um, what I'm trying to say is, instead of large, a single large block of code, we can train the model on individual function classes, library specific amounts, et cetera.

Correct. So you can structure it in a way where you are picking out the things that you want the model to launch on. Correct.

So it can be a library specific command, it can be a function call, it can be a, a task that you wanna train it on. So you can pick and choose, uh, when you're fine tuning, but depending on the use case and the, the, the problem you're solving, uh, that's how you can structure it. Correct.

So for example, uh, let's say we want to train it on, uh, an individual function as your, let's say your first component is training on individual function. So here you can, let's assume we are defining a function and we are returning the additional of two numbers. Correct?

So that's your step one. Then let's say we are training on class functions, uh, definitions next, sorry. So you can define a class, you can add that function underneath, and you can say, here, this is the class that I want you to learn from.

Correct. Uh, component three can be, uh, training on specific library commands. Correct.

So you can, for example, you can say from flask, import flak, correct. So in that order, you have broken down your training data into smaller descriptive components. Then you have strategically, uh, aligned the data in the right flow that you wanted to learn in.

And then you start training it one by one. It's still in the, in the, in the question answer format, because that's how it learns. But you still, you still, you're just doing it in the smaller chunks and in the right order.

Right? So the, like I said, constructing sequentially would be the next step. So you will then combine these components to generate the comple, uh, complex code.

The example that we took before where your component one is, uh, function call component two is training on the class. Component three is the import. Correct.

Uh, and then when you put all these together and fine tune, and then you say, here generate a flask cap, which, uh, with an endpoint to add two numbers, is that simple. So after fine tuning, you should actually see the output sequentially coming out too. You should see the import flask, uh, set up on the top.

Then you should see a small set of, uh, code in the middle, which is having an endpoint where you can send two numbers in and it can give you the additional result back and then the, the class definition around it. And that's how you would write the code yourself as well. Correct.

So that's, that's how we do it, uh, as humans. So that's how you're telling the language model to do it as well. Correct.

So if you follow this, uh, same pattern, which we have tried internally, uh, it has worked really well. And for complex codes, especially like if you're targeting a certain library, if you're targeting certain, uh, code generator functionalities, it will make a big difference as compared to regular code. Correct.

Uh, what are the benefits of doing this? Uh, improved alignment with target libraries. Like I said, uh, elimination of random permutations, another one big one.

And, uh, it has a better understanding of code structures and dependencies. 'cause you, it has learned, it has learned from your code structure, it has learned from the way you have, uh, broken down your training data into smaller descriptive components. So it has learned from it to say, I know exactly how you want it to, uh, how you want me to generate the code.

Correct. So, uh, it goes in that order and learns from it. Correct.

So, so that becomes very important, uh, for us to do. Correct. Uh, benefits of enhanced fine tuning.

Um, again, there are several benefits. I'm just listing out a few here. Uh, fine tuning large language models with our enhanced data representation.

Uh, strategy yields several benefits. Like I said. Uh, we have already tried this on a bunch of complex libraries and has worked really well.

Uh, one of the things is that it significantly improves code generation accuracy by reducing hallucinations, uh, with respect to that particular, uh, uh, code, uh, generation, uh, step. Correct. For example, uh, in, uh, we have used our, uh, fine finetune model to generate link chain code to generate vertex a code.

Uh, and then we compared that with regular models outside, like charge BD three, charge BD four, and we could see certain differences where Chad three, BBD four are not able to generate the complex version of the same, uh, query while the fine tuned models can correct. So that's a big difference here. Uh, lang uh, it, uh, language models trained with this approach, uh, can produce code more aligned with the intended library because you have taught it like that.

Uh, and then it eliminates permutations like we talked about. Correct. Uh, and they also become more reliable and useful outputs.

Correct. Uh, it improves the language model, uh, performance overall. It becomes better at providing you the accurate answers and you actually also start trusting it in the longer run instead of figuring out if the answer is correct or not.

Correct. Uh, real world use cases, uh, again, something that we have tried internally and applied to real world use cases. So, uh, again, based on experiences, what we are sharing it with you, uh, uh, and then again, you can feel free to try it yourself and you'll see a stock difference.

So we've applied the structured dataset representation strategy to fine tune, uh, code generators on new library. Like what XAI lang chain, like I talked about, uh, it has significantly improved the code quality and, uh, the generated code. Correct.

It, uh, it becomes very, very, uh, powerful to integrate these libraries together if the fine tuning was done correctly and sequentially. Correct. Uh, for example, ang chain integration, uh, before fine tuning, we saw random and incorrect usage of ang chain commands.

After fine tuning, we see accurate and structured usage of lang tuning command. Correct. It's not randomly bringing in the same commands everywhere.

It's not randomly throwing in, uh, parameters that we don't know where to use. It's not just randomly doing those things. Correct.

It's actually doing it in the right order, in the right structure, and it's doing it up to the point that we need nothing more, nothing less. Correct. Uh, importance of data representation, uh, in the real world applications, accurate data representation is very crucial.

Correct. Uh, it's, it's a different thing where you're not applying your solution in the real world, but when you're going to the real world, it's very important to have accurate data representation. Correct.

And, and it, it ensures the model understands the intricacies of the libraries and generates code that is both functional and efficient. Correct. You don't want it to generate random stuff and you don't want it to generate incorrect code and you don't want it to generate in random model as well when you're really, when you're applying in the, in the real world for, and the classic example here is, uh, charge B three, charge B four, A lot of people are using, uh, OpenAI to generate code that they can then use in their, uh, pipelines.

Correct. Nothing wrong. And 99% of the time they do it really well, it comes out structured and it's ready to go.

You can use it as it is. So it's almost like think of stack overflow where you would, uh, go find answers to your problems and then construct the code yourself. Uh, one step ahead, charge brief was doing it for you, so it constructs the whole thing and gives it to you and say, here, go use it.

Correct. So, uh, very similar to that, but, uh, it struggles that complex codes, that's what, uh, we are saying that in certain areas, then the fine tuning it in the right way will help you get there. Right.

Uh, and this is, this happens a lot, uh, and, uh, like I said, newer, uh, libraries such as Lang chain, vertex ai, uh, these models are yet to learn from them. So you'll see a ton of difference between what, uh, uh, open air models generate for these libraries versus what you can generate with a fine tune model. Correct.

Uh, practical strategies, uh, for enhanced performance. Correct. This is something that we apply, uh, added rate, uh, works really well for us.

Uh, I would recommend looking into it. Uh, you might be adding a few things in between. You might remove a few things in between.

But, uh, overall one to four, uh, if, uh, it's for fine tuning a model on complex task, these really, these work well for us for sure, but again, you might alternate here and there. Uh, so the first step, uh, is data prep. Uh, like we talked about, collect diverse and representative examples of code for the target library.

Right. That's becomes very important because if you do not do a right data prep for your model training, then it's not going to work. Right.

It's just, and in the right order, like we said, you have to, uh, put it in descriptive, uh, smaller descriptive chunks, then you have to put it in the right order, and then you have to go fine tune new model that that's the only way it'll learn from, uh, it and then give you the right answers. Correct. Uh, so breaking down these, uh, examples, examples, when I mean examples is when you're doing your data collection and representation of it, uh, breaking them down into examples which are smaller, manageable components is very important.

Putting it in the right order is very important. And structuring it is also very important. Correct.

That whole setup then becomes the, the input to your, uh, model. So doing that is very, very crucial in this step. Correct.

Training is the next step. Assuming you have done the right data preparation, uh, fine tune the model on these components sequentially. The, the reason I stress on the word sequentially is because you as a, as, as a coder yourself, you know the importance of importing the library first and then using it.

Correct. So, you know, it, you cannot do the other way around. So in the same way, you have to, uh, fine tune the models on the components that you have defined in your data preparation sequentially.

Correct. You have to go in the right sequence for it to know that, okay, I also need to follow the same sequence. Correct.

Uh, ensure that each training step builds on the previous one to maintain logical flow of the code. And again, this is a very practical example for anyone who, whoever writes code knows that you have to go in the right order, you have to maintain the flow of the code. You have to use the, the, the step that you have defined previously to build on top of that.

Uh, and that's how you write code. That's normal process of doing it unless someone does a different lee, where I don't know if that's even possible, but you have to write it in a way that you can build on what you've written before. So the same thing needs to be taught to the model where you have to ensure that each, each train training step builds on the previous one to maintain the logical soft code.

And that's the only way it'll make sense in the, in the long run. Correct. Uh, third step is evaluation.

Very, very crucial. When you are fine tuning, uh, your, uh, model on your diverse data that you've collected, um, not always. It'll work right out of the go.

So you need to evaluate your, uh, model that has been fine tuned and trained, and then you have to, uh, go back to the training process again if you see discrepancies. Correct. So continuously evaluate the models output to ensure that they align with the target library's usage.

Correct. That's evaluation is super important that you just, you just cannot find you a model and just randomly go use it. It just doesn't work like that.

So you have to continuously evaluate. Correct. Uh, and you can use automated testing frameworks to validate the generated code.

Um, you, there are multiple ways of evaluating. There are, you can use so many open source testing frameworks out there to validate the code that was generated by a model versus the code that is, uh, technically supposed to be generated. Correct.

That makes more sense. And based on the evaluation, you can see the difference and you can go back and maybe redo a little bit of your data prep, or maybe you can go back and redo a little bit of training again, and then you can come back to the valuation step. Correct.

Uh, optimization, uh, is the next step. Let's assume that your data prep was correct, your training was correct. Your evaluation was correct.

Now you're gonna come into optimization. Correct. Because now you know that what you have done before has worked.

Now let's figure out how to optimize this. Correct. So you can, uh, the very first step that we recommend is your leverage consumer grade GPUs to train a smaller model effectively.

Correct. Uh, you can totally, uh, rely on, uh, consumer grade GPUs to train and it's, it's usually a one-time cost, uh, when you're doing this, but it works really well. Correct.

And then you can optimize training parameters to balance performance and resource usage. Sorry. Uh, you can work around how you can optimize those training parameters and how you can balance the performance and resource usage of that particular model.

Uh, 'cause that's when now you have coming, now you're coming down to the level of let's optimize my model so it can do more, use less and work effectively. Correct. Because your data prep was correct, your training was correct, your evaluation PA was good and it passed.

So now let's optimize it for production usage. Right? So that's why you're coming into that.

Uh, example, classic example here is, let's say you are training a model to generate machine learning pipeline using TensorFlow. Very classic example here, right? So your component one here should be the import statements and data loading, assuming that's what you want to do.

Component two will be your model definition and compilation. And component three would be training and evaluation. And if you write this yourself, you will do it in the same model.

You will first import, then you will define the model and it's compilation. Then you will do the training, and then you'll evaluate same order in which you will go. Correct.

So, uh, that's a classic example of how this can be done, correct. So, uh, these things, um, sound, uh, somewhat complex, but if you, uh, break it down into, uh, the right, uh, not even the right, the smaller steps in between, uh, they work really well. Correct.

It's not something that you can be, uh, uh, uh, you can not do, uh, over a period of time. And then again, once you define this pipeline correct, where the prep training, valuation optimization can work for one model, you can apply it in multiple, uh, ways to multiple models. You can apply it to multiple data sets.

Uh, it usually works well correct. Now, again, I'm sure there are, there, there will come time where this won't work and would require some sort of enhancements. And then we can work towards enhancing this pipeline to level two.

But internally at the thread, this has worked really well, uh, for our custom models that we are using to generate link chain code to generate what exec code, uh, to integrate into different libraries together. Uh, and again, this is, we have done this on, uh, open source models. So the one time cost to fine tuning was all that we looked into.

Uh, after that we can just use it as we go. Uh, and then the next steps we are moving towards is, uh, using our proprietary code to fine tune a model so then everyone internally can use the code to build on top of, uh, on our platform next, correct. So that's our, our, our vision there as well.

Uh, but yeah, uh, that's something that we have applied internally. I hope it works for you. Give it a shot.

And, uh, hopefully, uh, you can give us some feedback around if and if it didn't work for you, what did you do different. But, uh, other than that, uh, thank you for your time, uh, today. Uh, this is all, uh, uh, something that we are working towards and I hope this strategy comes handy for everyone who's trying to find, tune their own models, uh, on, on code generations, uh, internally.

Thank you.