Robbie Dyer – Cost-Efficient ML Model Training Using GitLab CICD in the Cloud

I'm happy to introduce our next speaker, Robbie Dyer, a Devops consultant advocate at Eficode. Robbie's talk's about cost efficient ML model training using GitLab CI CD in the cloud. Data scientists are not usually the first group that comes to mind when thinking about Devops and cloud native practices.

Yet they create software and oftentimes consume large amounts of cloud services. And this talk, Robbies shares his experience, results and learnings from his journey to enable data scientists to do machine learning using Devops and cloud data principles. There's a lot of great detail in this talk.

Let's dive in. Hi, my name is Robbie Ryer, I'm a Devops consultant based in Copenhagen, Denmark. K.

and I started my career as a web developer before moving on to managing cloud infrastructure. And I'm progressing to bridging the gap between the two as a Devops engineer. I've now been a part of Eficode in Copenhagen for nearly a year.

com and see if we could be of any help with any of your cloud or Devops needs those. We've got some fantastic training, which I highly recommend. Today, I'm going to be sharing my experiences with creating machine learning model training pipelines, using GitLab CI CD.

I've been working with some data scientists and engineers at a client on this for roughly the last six months. We've had great results from the work that was done was whilst the system we've created is still far from ideal and finished. We've already learned many lessons progressed fairly far, and I hope this story might inspire some others to take the same journey.

The reported outcomes of this project were pretty positive from all sides. I personally felt like I learned a lot. I always enjoy being pushed outside of my comfort zone and keeps work interesting.

The data science and engineering teams are both been very happy with the increased ease at which they're able to go about their work. The heavily decreased training times have been reported as another great benefit and also people love learning new skills. And that's really proved that Devops practices do belong in data science.

We've actually even had one of the members of the data science team move over to the Devops team to help create more cross-team collaboration and further push Devops practices across more areas of the organization. The business was happy by progress on modernizing the platform and with the increased increased ability for the data scientists to innovate and drive away from that data. The theme of this event pairs rather well with my talk today.

I certainly wasn't sure I belonged in the world of data science at first. Prior to starting this project, my knowledge of machine learning and data science was limited to a couple of neural network and machine learning courses done at university. But I'd never covered any projects such as this in the wild.

As a Devops engineer from a software engineering background, I was more at home with developers, so this new environment was certainly a little out of my comfort range. So let's start by briefly going over the ways things were and why this project came about . Previously, the data scientists were relying on their development workstations, compute resources for most ML training tasks.

These workstations were running as VMs in the cloud, but it was still far from ideal setup. I've also heard rumors of live systems being hosted on these workstations and that certainly wasn't going to fly going forward. The previous set up had rather a large number of downsides.

The workstation VMs were having to be left on 24/7 while the models were trained overnight or over the weekend. This added cost both an inefficient use of their time and added frustration for them, and I'm sure didn't do anything good for their productivity. Given the workstations generally weren't particularly powerful for obvious cost reasons.

This also meant that nothing moved particularly quickly and iteration of models take much longer. Also wasn't possible for individuals to work on multiple models at the same time, and I'm sure it didn't do anything good for the responsiveness of their workstations whilst they were training. Another issue was that there was no single place to look when issues arise and relevant logs persisted, persistent.

There was no audit trail either. So as you can imagine, the desire from the business was to make things more efficient and keep the data scientists happy and productive. There was a strong requirement for training to happen faster, which in turn would allow faster development of new models.

And we needed a standardized pipeline and set of tools which would allow new libraries to be experimented with easily. They also wanted some better tracking of operational expenditure, and there was also a desire for more cooperation between the data science, data engineering and the platform team; ie: Devops, so that further developments to the platform could be driven by the right people. The requirements certainly were the easiest parts of the project to pin down for me.

I had very little knowledge of what was being done and therefore what we'd need. As with all projects like this, the requirements certainly changed a fair amount as we experimented and progressed, but we had to start somewhere. So we knew we need some storage for artifacts as we were on Azure, Azure storage was an easy one here with AWS, theres's S3 or GCP, that's the cloud storage.

So other cloud providers have offerings as well as. Database and stream access was needed. So we needed some form of secret management.

Luckily, GitLab CI CD has variables built in, and we also did look at using Azure key vault in combination with GitLab CI CD variables and service principles. We needed a good range of VM sizes for different tasks. We knew we wanted to do both CPU and GPU based training and we weren't sure what instance sizes would be best.

Therefore, we needed to be pretty easy to experiment with different sizes, virtual machines, and especially in the initial stages. We also needed a Docker image registry as we needed store our images we were already using Azure container registry for other teams. So that was another easy win.

I believe AWS and GCP have their own versions of those as well. We needed some form of monitoring and logging for diagnosing issues. And we also wanted performance metrics to make sure we were getting the maximum utilization of the infrastructure we provisioned.

As the data been used in these models was sensitive and personally identifiable information, European data laws are pretty strict. So we would limited self managed systems as the client wasn't keen on this data being past third party parties or any external systems. So the obvious choices for orchestration in this instance was either GitLab CI CD or Jenkins.

Jenkins does provide additional flexibility and plug ins, but the requirements in this case were already met by GitLab CI CD and the overhead of configuring and managing Jenkins was just undesirable. GitLab was also being used organization wide for repository storage, and GitLab was also being used by other development teams. So these were further attributes that supported choosing this option.

Having chosen GitLab CI CD as the pipeline to the next that was to decide how to achieve the compute requirements we needed. GitLab runner executers can be run either using Docker, Docker machine, or Kubernetes. Azure Kubernetes service didn't support node pulls of zero at the time.

This would be required creating node pulls of expensive virtual machines with specific taints for each type of job, and we wouldn't have been able to scale them down to zero when they're not in use or not automatically, which, as you can imagine, is not particularly efficient use of money. We were already using Docker based runners, along with Docker based excutors for cpde deployment pipelines. This obviously wouldn't have worked for the compute requirements.

We have ML training, but it mean with minimal effort, we were hopeful we'd be able to create new instance based executors as using Docker machine and the Azure driver. Using Kubernetes would have added additional difficulty and testing new instance types, as well as we can quickly add new instances in the same way you can with a machine, and it would have required creating node pulls and taints, which is not particularly desirable. We did have a slight concern that Docker machine has been moved to maintenance mode, but GitLab have forked this and I believe are still continuing to support it.

But I suspect we will eventually move away from Docker machine as the executer and move likely to Kubernetes. But it was a very good starting point. As you can see from this diagram, we've got virtual machine with multiple GitLab runner Docker containers, each configured to use different Azure virtual machine as executors.

These will be created and destroyed as necessary using the Docker machine and the Azure driver. So when jobs are created and start running, it will create a Docker machine executor VM, and when the job's finished and the time that's been exceeded, it will then destroy it. Configuring the GitLab runners to use the Docker machine executors turned out to be a fairly big task compared to what I was expecting.

The GPU virtual machines where the biggest challenge. With the added GPU comes added requirements for specific drivers to enable its use. We knew we needed to be able to use both Tensorflow and PyTorch torch.

We experimented with the N series of virtual machines on Azure. They use the Nvidia Tesla V 100 is the easiest method I found of getting things to work as we wanted was to use the Azure marketplace images which helpfully Nvidia provide themselves. I did attempt to provisions myself using Ubuntu, but I didn't fare well with my initial attempts and it seemed like a silly thing to waste time on this point to achieve the desired options for training during the PAC, we needed at least for runners to different GPU powered executors, and the same number of CPU powered executors.

This would hopefully give us enough flexibility to experiment with different sizes for each model without constant changes to the configuration. With the run of registration, I'll start by saying the documentation is pretty good, only so much of the runner configuration can be specified in the complex config template and the rest needs to be specified during the registration. We quickly found that we were hitting limits on the number of runners which could be created on each virtual machine that were hosting these containers.

We believe this is almost certainly because we were using the same machines as a based excuse us for some deployment pipelines and the issues appear to be related to Disk IO. We were getting large queue debts and we found running more than about six jobs on a node, even using premium desks, which are supposedly able to support 600 jobs, we were getting pretty poor performance and jobs would often time out. Auto scaling using either scale sets or probably more likely switching to Kubernetes likely things that we'll look at in the near future.

But for the initial stages, we went with a dedicated runner virtual machine just for the of training jobs. Another thing to take into account at this point is Docker pruning, certainly looking at or you quickly run out of space. Just quickly touch on some of the tools which we use throughout this project and hopefully give you a clearer picture of how we went about prerequisites and they did succeed.

We heavily used Terraform for infrastructure management. We've had some issues with Azure, the Terraform compatibility is definitely lacking the web console. So but by and large, we've been able to work around these issues and not use click ups.

I know Azure's hard at work catching up on these differences, but I suspect with the speed new cloud features are released, I won't expect complete parity anytime soon as those used for configuration management and helped simplify and automate the process of registering all the different GitLab runners. We used ElasticSearch for logging and we used Prometheus and Azure log analytics for performance metrics and letting. This was used due to the sometimes short lived executer instances so Prometheus wasn't suitable and we had to just use the built-in Azure on monitoring and log analytics.

GitLab Prometheus integration is certainly something we'll be looking into in the near future, but we didn't manage to try that at this stage. Dockerizing jobs was probably the area which I felt by far the most uncomfortable at first, but actually it turned out to be one of the areas where my help was required the least. I hope you can excuse the S rather than the Z arising, but I'm British, so I can't help that.

I was lucky that few members of both the data engineering and data science teams had already been experimenting with Dockerizing some of their work. So with a little assistance, they were able to get things working fairly quickly. I'd suggest, starting with a good set of example doc files, which can then be easily copied and adapted as required.

This helps with ensuring best practices are followed and often say as many issues and questions from arising in the first place. As we were being fairly security conscious, or I certainly was myself, one of my primary concern I have is the source of base images which were being used by the teams, Docker hubs full of perfectly good base images. But it's also known there are some bad apples on there too.

Having a local set of known safe images stored on your registry is definitely advisable. That can also help ensure that the base images being used only contain the required components and thus the smallest possible multi-stage build. Definitely something which should be considered too, to this can save time space, essentially make things more secure.

The end result. Was that both the data science and engineering teams were easily able to create, train and deploy new models, we saw significant reductions in training time models. They were up to 20 times faster.

We were able to train models both using Kubernetes for less intensive training jobs and Docker machine executors, both from the GitLab CI CD the pipelines. Issues have been reported much less since this project went through, I suspect was largely done so much, the standardization of the work which was being done. Many of the data scientists and engineers have actually since progressed significantly with their cloud and Devops utilization.

We've seen a massive increase in the number of teams asking about monitoring for their models and services. We've also seen our usage of GitLab CI CD mature significantly. We've had cleaner pipelines and better uses of all the other resources available on the platform during that pipelines.

I'll start by admitting the title of this talk is perhaps a little misleading. The aim was never to reduce the operating costs too much, but to ensure that the unnecessary expenditure was reduced whilst also improving the platform and tooling provided to the teams. The infrastructure cost savings of not running workstations 24/7 more than overshadowed by any training jobs which are running in terms of total cost savings.

I don't personally have any figures to hand, and it's certainly not my or my expertize. I think it's fairly safe to say that the time saved by for the data scientists and battling to get insights is probably more than offsetting the additional infrastructure costs for the expense of draining instances. And the next step that the data scientists are actually undertaking is to increase their knowledge and awareness of which insights are actually creating value for the business and why they should focus our efforts on next.

Some specific steps, one, big one was the VM name character limi. I believe it's 64 characters I wasted a not insignificant amount of time debugging why Docker machine was failing to provide an infrastructure and this turned out to be the culprit. We also experienced issues with Docker machine failing to delete virtual machines after the timeout had been exceeded and jobs had finished running.

And I'm unsure if this is an Azure issue, an Azure Docker machine, or Docker machine issue, but wherever it came from, it was worth keeping an eye on as having expensive machines running for hours without any jobs to run is not recommended. Availability of instances was a bit of an issue on the way. This has been exacerbated by covid-19 with more companies moving to the cloud.

But we actually experienced availability issues on Azure before this, especially with GPU based instances in the Amsterdam region. GPU based instances were quite often unavailable in many sizes and even for other instanced types, we had issues occasionally although far less regularly. Spot instances, probably won't work for us unless you've got very fast training jobs.

The risk of your instance being being unavailable and two minutes notice interrupting long running jobs is probably undesirable. Reserved instances, unless you're running a large number of jobs 24/7, the credits don't get saved up when you're not using them and you can use them in parallel, so it's probably poorly suited. And the other thing to bear in mind is that Disk IO is also constrained by the VM.

So it's not just the desk. So making larger disks won't necessarily help at a certain point for performance. The maturity of teams is going to be a very large factor in whether projects like this will succeed.

Data science is not currently known for its universal acceptance of Devops practices. I don't think that's right. And I believe any team can derive huge benefits from adopting Devops practices and utilizing the cloud.

I was very lucky to have a few members of both the data science and data engineering teams keenly involved from the start. Without buy in from the team and business, Projects like this have no hope of succeeding. It will be slow and disruptive at first, working in an agile manner with data scientists and engineers committed to the project from the start is vital and I wouldn't have got far without this.

Regular meetings and communication with the team is something that really helped me personally. As with any area outside of your comfort zone, it's sometimes difficult to fully understand what needs to be done without regular opportunities to ask questions and communication really is king. Thanks for watching my talk, I hope it's provided some inspiration, hopefully some useful information too for anyone embarking on this journey.

If you've got any questions or comments, feel free to reach out to me on Twitter, email or feel free to connect on LinkedIn.