Revolutionizing NASCAR Strategy with SmartPitbox with David Ellison at AIE 2024

Hello, I am Dr. David Ellison, and I am here to present, uh, revolutionizing NASCAR strategy with Smart Pit Box. Um, just introduced myself a little bit.

I am chief data scientist for Lenovo, and as the chief data scientist, we do a number of things. We help end-to-end AI solutions, uh, with companies. Uh, like this NASCAR project.

Uh, we lend out hardware and we do ML perf benchmark testing, uh, which is the leading AI benchmark. So we do a lot of activities at the AI Innovation Center here at Lenovo. Um, and let's discuss one of them with this NASCAR strategy, uh, with smart pit box.

Um, so what we're trying to do here is you see our Lenovo branded car there. Um, we are sponsoring a team called Richard Cirus Racing. Um, they are local to North Carolina where our headquarters is, and, um, we are helping them measure the exact amount of time it takes to fill up the, the car.

Um, so we have a live feed from overhead cameras. Um, the resolution at 30 frames per second, and the point here is to measure how long it takes to fill up the car. I'm gonna show you a video in a little bit of how this happens.

There are no fuel gauges on a car, on a, on, on a nascar. Um, so you need, you need to, uh, figure out how much fuel there is, uh, based on how long the gas tank has been attached for. Um, and right now they're just doing a kind of a one Mississippi, two Mississippi and just doing a ge a, you know, um, a general rule of thumb of how much fuel's in the car.

Um, they wanted something more precise. Why? Because if you are, uh, putting too much fuel in the car, you're not only waiting too long 'cause you're holding the car back when it could be on the track.

Um, but you're also, the car's also riding heavy. That's if you put too much fuel in, but too little fuel in. Well, the results are obvious there.

Um, in that case, the, uh, the car doesn't get to where it needs to go. It, it, it runs outta gas before the next, um, pit stop or, uh, you know, before the finish line. So, uh, figuring this out and getting it precise is very important.

It can shave precious, uh, seconds or milliseconds off the race time. Um, so we wanted to set up something where there was a no lag inference for the race and engineers and pit crew to review. And they can use the strategic information for mileage consideration so they can really plan out how much time is left, how much fuel they have in the car, so how many laps they can get, and to try to use that strategic positioning to leap ahead of other cars you see in, in a race, in, in a NASCAR race.

Um, you know, pulling ahead of cars might be exciting on the track. It's an exciting thing. And the reason why it's exciting, because it's very hard to do, it is very hard to do to, to pull ahead of other cars 'cause they're blocking you or the, the, the, you're, you're hitting a curve and, and it's a little bit out of control.

There's all types of reasons why it's very difficult to track. The easiest way to pull ahead of cars is to do it in the pit stop, to make your pit stop's a little shorter. Then when you pull out your one or two positions ahead of them, um, that's the very easiest way to gain position in a NASCAR race.

So let's look at one of these pit stops here. You hear you're coming in, see the fuels attached, uh, as they're doing the one side of the car and then they has to back away and there he connects and the car is let go. Um, so that's, that's see it a whole, everything happens in about, um, seven seconds.

Um, the whole pit stop can take about five, five to seven seconds depending on how much gas they need to put in the car. Uh, one of the rate limiting steps is the amount of gas they're putting it in the car. Um, so let's look at this.

Uh, so we took all the raw data and all the 2023 season data, um, stored that on Azure blob container. Um, there, there was al types of problems with the data. There's, this isn't a file format, this is resolution.

And the car deliveries or the, the, um, the advertising that's on the car changes every time. So it becomes very difficult to track and, and, um, and deal with. Um, but we took it to the cloud and we had to choose which AI to use.

Do we use a classification model, a detection model, or both to solve our problem? Um, in this case, we, we chose to do an object detector to detect, um, because the, uh, deliveries of the car were, uh, confusing the classifier or making that less, less accurate. Um, we defend object detection.

The difference between a classification and object detection. If you're not familiar with ai, um, computer vision and AI is classification tells you like, is this a dog or a cat? It tells you, is it connected or is it not connected?

An object detector tells you, um, you know, what draws a box around what you're looking at. This connector is connected to the car, and so it knows to look at that particular area on the, um, on the connection. And we need to be able to handle live inferencing at 30 frames per second with exceptional accuracy.

Um, you know, even just getting a couple frames off can mess up these calculations. So getting a very high accuracy is important and we have to choose models that do that. Um, data labeling, um, we need to choose an efficient and quick annotation tool.

Um, and you'll see the tool that we chose in the, uh, next round. And then we also used, uh, managed data and developing AI in a single software. We used, uh, software called leco.

Uh, LECO stands for Lenovo Intelligent Computing Orchestration. Um, it is a Lenovo, um, software that we use to develop AI models. And I'll show you here, here's the login.

Um, so web portal, and there you can choose. Here we go. You can choose run, uh, Jupyter Notebooks or the data labeling CVA software.

And you see there we're labeling. Um, you can, how, how quick it is to label, uh, data sets in that. Um, if we run it again, you'll see that, you know, after running the CVA, you can, you can select, you can select the classifier of whether it's plugged or unplugged, and you can even do that every once every couple frames.

So you can, uh, select this frame and then, uh, select a frame 10 frames later and it'll just interpolate what, what is that, that was connected that whole time. So I really say in the efficiency and, um, and, and labeling software, you may have used it if you, if you're using, um, computer vision software, other places, um, CVA is a very effective tool for doing the data labeling. And we had to do, uh, hundreds of, of pit stops.

Uh, we, there's about, uh, for, you know, there's a, there's a, uh, race every weekend except in the dead of winter, and there's about four to five pit stops per race. Um, so that turns into hundreds of, of, um, of videos to label. Um, and, you know, you wanna be able to do that in as efficient manner as possible, and that's what we, we show here in, in this video.

Um, so we did experiment with various classification models, um, but found that their accuracy was not, um, to be, to be desired. And we theorize, uh, based using on a saliency map. Um, saliency map shows you what, um, what your computer vision algorithm is looking at it colors in the region.

So you're looking at a picture of a car with a fuel gauge. That car, if it's looking more at the car versus the fuel, the fuel can, um, it'll, the, the car will light up and that's what sale. So we are finding with the classification models that the, um, that the car, the, the, the car was being looked at too much and not the gas tank.

Um, so because of that, um, we had to go with a more precise model on object detector and that we used a Yolo V eight, uh, CLS architecture to train the dataset. Um, this is a classic YOLO stands for you only look once. Um, it's a, it's a, a classic model out there, um, and it is very robust.

Um, we ended up getting very high accuracy. Um, and you know, as we talked about, you have problems with data imbalance. Um, most of the time the gas can is not connected.

You know, most of the time the car is not there or it's not connected. Um, so, um, you have an in or a class, you have a class imbalance between the time it's connected and the time it's not connected. Um, so you can do that.

The way that we did that, um, is through two mechanisms. We did, uh, oversampling, which means, um, so you have lots of data where it's not connected for, um, so that, that's the, the majority class, the minority class is when it is connected. Um, so you can, well oversampling you can select that minority class and, and reuse that data multiple times to get a balance between the two, uh, data sets.

That is, um, what we were looking for. Um, and that helps you train, you know, a classifier or an architecture much more effectively. You can also do, and what we also did was weighted classification, which means that you penalize the model more for, um, not using or for, for making wrong decisions on the minority class.

So the one the, in this case, when the data you, we would penalize the model more if it made a decision that the, um, minority class was, um, was wrong. We, we, we penalize the model more, which may means it, it, it adjusts itself and, and comes up with a much more accurate, um, overall model. Um, and then we, uh, inference on future races to ensure the act results with no model drift.

Model drift is something that you always have to detect in your models. Uh, it would be wonderful in data science if you create a model once and it would just work forever. Um, but that doesn't seem to be the case.

Um, things change, seasons change, uh, out in the track. Um, you have overcast days, uh, rainy overcast days in the spring. You have bright summer days, um, that are clear and, and have a lot of, uh, shine and feedback on the, during the summer.

And then in the winter it gets a little bit more cloudy again and OA little bit o more overcast. So, um, there can be changes based on the weather. Um, and based on, because these are all, these are NASCAR races, they're all happening outside, they're all happening at different pit positions.

They're happening with all different, uh, camera placements and on all different types of tracks. So things change over time, and we had to be able to ver uh, verify, um, that ability to handle the inference data and do it on live feed data with no output lag, had to, we had to hit it. Um, you know, at that 30 frames per second was the target.

Um, so here we were, you also using our leco software, um, to run, uh, the notebook on the cluster. Um, so even though the model itself will eventually be running on the edge, um, where, where it won't need a cluster, we want to be able to train the model on a cluster. Um, and so this allows us to, um, train that model on a cluster, allows us to, um, leverage all the resources we have and apply all the Nvidia specific, uh, transformations to the data in order to get it to run in the most efficient manner possible.

And we're gonna talk about those adjustments that we made to make it run in the most efficient manner possible in a little bit. Um, but this, this, uh, leco software enables us to, to really get in there and use, uh, Junos. But honestly, if you have a small enough a model and you're, uh, training it, you, you know, you could do this on your own personal computer.

This just allowed us to leverage the resources that we have at Lenovo with servers and all that. Rather, everything is speed up. But you know, you could, you could in theory do a lot of the same stuff just on the using Jupyter Notebook on your, uh, local computer.

Um, as long as you had a relatively powerful, uh, GPU to do the, the fine tuning and the retraining of the model, it was gonna take a little bit longer. This takes a few seconds to train as you saw in the video. Um, it might take an an hour or two on your personal laptop.

Um, so there's nothing. Um, and then, you know, hearkening back also to the data labeling that CVAT is a, uh, tool that is available online if you wanted to do a project like this. Um, there is nothing stopping you.

Uh, we like to think our software simplifies things a little bit and makes it a little bit easier. Um, but there's nothing here that you couldn't do yourself. Um, Yolo VV eight is a public open source model.

Uh, CVA as an open source model, Jupyter Notebooks is obviously open source. Um, all this is is things that you could do that you don't have to, you don't have to spend a cent in order to do it. Uh, you just need the data.

And I get admittedly, getting that data is a little bit hard. Um, so, uh, here are the results. With YOLO V eight.

4% validation accuracy. Um, this was at 70 frames per second, um, at, at a 1280 by 1280 re input. Um, we were able to reduce that to a 22 millisecond inference per frame on a six 40 by six 40 input.

That was enough for 30 frames per second. Um, and you'll see at that 94% accuracy. Um, you know, we get a, a wonderful looking confusion matrix.

Um, you know, they're, they're in the, in the center on the, the, the diagonal there. 65% of those, uh, images were misclassified. And when you're talking about a frame 30 frames per second, um, that's, those seven frames that are misclassified are a rounding error, um, in the overall time it takes to, to calculate that fuel connection for.

Um, so the, uh, model is found to be robust across changing car deliveries, weather conditions and lighting conditions. Um, as you might imagine it, you know, also these NASCARs not only happen in different weather conditions, but also some happen during the night and you have overhead lighting versus daylight. Um, you have all different types of, um, situations here.

And the model consistently performed inferencing, uh, for that 30 frames per second and live rings for multiple cars. This is now being used for multiple cars and multiple teams. Um, so it's, it's a very exciting use case right now.

Um, and as I mentioned, it's all running at the edge. It's all running on one of our edge devices, um, that is sitting on a track. Uh, that is because the cloud inferencing situation, you just can't do that on, on a race racetrack there.

The, the connection to the cloud is not good enough. You can barely, if you've been at a sporting event and tried to use your cell phone, um, you realize that with everybody trying to use a cell phone, those, those towers get overloaded. Um, and so your connections to the cloud, uh, get overloaded and then everything turns into really slow latency.

So you have to do this on-prem, and that is, uh, what we designed and what we set up for is to, um, do everything on-prem at as close to the data as possible because, uh, moving these, you know, 1280 by 1280 videos to the cloud inferencing and getting back to the, the, uh, the team, that just doesn't happen fast enough. Um, so let's look at some of the things, um, that we did in order to speed it up. Um, we're using the tensor RT model with Triton Inference server.

Um, and, you know, we convert the models from PyTorch to Tensor rt. And what that does is it reduces the computational overhead by using layer fusions. What layer fusion is, is it takes operations and it sticks 'em together and makes 'em faster.

So instead of adding and then, um, you know, multiplying and then adding, it just does one operation that multiplying adds. Um, and so it just really speeds up the whole process. Uh, quantization is another, another fancy word I know.

Um, but that's just tain changing the precision. Um, normally things are stored in either floating, floating point 32 or 16. Um, and so you move, you move that floating point 32 down to floating point 16.

Um, that's just how the number is represented as a floating point, which means something with a decimal point that has, um, 16 bits in it. Um, 32, you take it, 32 move down to 16, or you use it to an N eight, um, which is the integer eight. Um, so this, this difference in precision allows you to speed things up.

You lose a little bit of accuracy and maybe, you know, two of those seven frames that were misidentified are because of quantitation, but it does speed up your accurate, your, your, um, your, uh, your, your inferencing by over two x and then deployment. Um, efficient use of of GPUs running in multiple workloads in parallel. Um, Triton allows you to use, uh, multiple infra, you know, do multiple inferences all at the same time, use multiple models because we can't dedicate an entire computer to running one model.

We are running lots of different models and, you know, we're doing extra models out there where we're looking at doing, um, models for, uh, the angle, uh, the angle of the can, um, you know, not just the time's connected for, but look at the angle of the cam, look at, um, you know, how long it takes to change tires, how long it takes to jack up the car. Um, so we're running all different types of models to speed up, speed this up, and to deliver, you know, the best value to our customer, uh, possible. So let's look at it.

You see in the upper left hand corner, it's plugged and unplugged and it's certainty. And you see it, it's 100% certain that it's unplugged and that's a hundred percent certain that it is plugged. So you see that it's a very accurate model and, and very precise, uh, given the timing.

And this is being done in real time. So customer uh, success story is, uh, sending information to the race engineers and strategist the current fuel level to make important decisions. Um, it helps avoid, um, downtime, it helps avoid making mistakes in the filling.

Um, and then, um, it also has to potentially help the team during cautions. Uh, cautions are, when they slow down the race and have a pace car slow down the race, that's a great time to pull into the pit and fill up as much as you can because people aren't racing at full speed. Um, and that allows you to avoid extra pit stops, um, as a hosted solution of multiple pit stops.

It can be inferred in parallel and it provides results in real time dealing with multiple feeds. Uh, the overall is at Talladega 2024 and other races. Um, we almost won, uh, the car spun out at last, uh, at the last, uh, last few LA laps, but, um, we were able to gain two to four positions at every pit entry.

Um, so that is an amazing increase in, uh, or value added to the, to the customer is getting two to four positions for free. Um, just 'cause you're doing more efficient fuel fueling and usage, um, really delivers value to the customer. Um, and, you know, they'll be using this as a competitor, as a competitive advantage for them for a long time from now.

So with that, I would like to thank you for, for listening. Um, it was an exciting project. I loved working on it.

Um, and you know, I love the fact that we get to work on these exciting use cases at Lenovo.