AI Tools and Code Development with Focused Labs’ Austin Vance

This is Textron tv. Hey guys, thanks for the throw. We're here with Austin Vance, who, CEO for Focus Labs.

And we're talking today about, well, exactly where does AI fit in our DevOps workflows, because well, developers have all kinds of AI tools, but it's not quite clear how the rest of the workflow is gonna work out. Austin, welcome to show. Hey, thanks for having me.

At least on paper right now, it looks like we're giving developers more AI tools to generate more code than ever, and hopefully that's a good thing. But all this stuff is now starting to cascade through our pipelines, and it's not clear to me that our DevOps workflows are really set up for that. So do we need more AI to help manage all the ai?

I think there's probably a, a couple places AI fits, but yeah, I think we probably need some AI to help us manage that ai. We have to figure out how to operate and deploy, and we have to figure out how to build, um, automated testing paths for AI applications. And, and then of course, scale is gonna gonna matter a ton as we scale out all of our H one hundreds across all of our data centers.

So how should I be thinking about this if I am the manager of the DevOps team? Um, you know, in my mind at least, I can imagine a world where there are humans who are managing or orchestrating things, and then there's a lot of AI agents that are managing specific tasks within that workflow that I'm trying to weave into this. But how do I think about that and how might I actually execute that At, at a highest level, AI agents that are, you know, deployed as consumer applications, I think will operate and run just like any other, uh, microservice or service.

So service oriented software, um, you'll have to scale. You know, you'll have your humans there building terraform scripts writing CDK deploying applications that have access to GPU clusters. I think where we can get really interesting is how do we start applying these AI workloads or these a work workflows into the continuous integration pipeline to even do automated continuous integration, um, battle testing stories, exploratory testing, or maybe even release note writing.

So we, we'll start to see that kind of stuff come out in, in our CI pipelines. They'll get super robust over time. Mm-Hmm.

As a, you know, there's a large overlap between security. We, we all know, like large overlap between security infrastructure and infrastructure and DevOps and how we, how we manage our data, give access to certain parts of our infrastructure and make sure our applications will only have access to what they're supposed to, is also gonna matter a ton as agents crawl through our cloud infrastructure or our data centers. A lot of what you described today is, shall we say, a bit messy.

Um, will it get cleaner as we go forward? Because we'll have these AI agents that might, at the moment that we're actually trying to do something surface an alert that says, Hey, just so you know, this particular piece of code here that you just inserted has 22 known vulnerabilities and 15 dependencies, and, but it'll just be part of our workflow. It'll be in, it won't just be, become something we hear about, you know, three weeks later when I've long since forgotten what I was working on in the first place.

Yeah, think of it like a sonar cube plus plus where you have matic complexity, dependency checking, vulnerability scanning all even more proactive than than an installed, you know, an install appliance in a Jenkins pipeline. So you get all of this stuff, AI will start to add AI can, and we'll, we'll likely start to add a lot of that like cl cleanliness. I don't think it's gonna get, I think it's gonna get massier before it gets cleaner.

I will say like, there's a lot of, there's a lot of experimentation happening in how AI can apply, and we're definitely at the, we're at, at, you know, the upward trend of the hype cycle where AI is this panacea of solving all of our problems. When we realize what it can and can't do and what the limitations of, of the current language models are, we'll end up applying them in much cleaner ways. But for now, I think we're gonna try and apply AI in every place.

You know, can it scan my code for vulnerabilities and does it actually find them or does it make them up? Can it look through, uh, my software and optimize code if it's automatically optimizing something, does it introduce more bugs than it create, than it than it solves? Or does it make code harder to maintain for a developer in the future?

And we'll, we'll start to iterate on that as we, as we play with ai. So it doesn't sound like to me that you're especially concerned that AI will replace software engineers anytime soon. So what becomes of a software engineer?

What is the job? Boy, it's a good question. I I don't think it's gonna replace us anytime soon.

I think that, uh, just like the compiler didn't replace a software engineer, uh, or the cloud didn't replace an infrastructure engineer, the AI will become a, a new interface in which we interact, interact with the, the construction of computer programs. And so whether that's instructing an AI to write significant amounts of software for us, or using it to check bugs, a human in the loop will always be necessary to, to drive the, the product vision and the logical constraints. I think AI can, can help implement, you know, the same way memory management helps implement a, uh, a feature in a more efficient way.

AI can make us more efficient at implementing our visions, but software engineers and product people will always be necessary to help, you know, scope down product to what is usable and necessary. I feel like, you know, for every one software engineer in a DevOps workflow, there's probably, um, making a number of 10 developers, um, totally. Is there gonna be, um, some rebalancing of that dynamic?

Uh, because AI will enable one software engineer to manage X number more of workflows created by X number of developers, and the math is just gonna change in a way that'll be, um, compelling and interesting. Yeah, the, I wonder if the math was a product of like the, of the world investing deeply in software ideas without a lot of, with a lot of, without a lot of pressure on return. So for a long time, a lot of companies had a lot of software developers, so they could put a lot of product into market.

And as we've seen, you know, with x reducing software developers with, uh, Google, with, you know, all these big tech companies reducing software development capacity while also continuing the same product output. I wonder if that 10 to one is actually really even in, in the old days, is like a actually, you know, a four to one. And then I, I think that DevOps will become even more important because SLAs, SLOs and scale will matter a lot.

Observability and traceability, the management of infrastructure and the uptime of that infrastructure will become the, the key piece in the interface for how we interact with customers. And the product itself can be built, will be start to be built out by ai, like you said, and even that four could become two to one or something like that. A lot of folks I talked to are also scratching their heads about, well, where does ML ops fit within the context of DevOps?

And is that gonna converge as well? And has that become part of the, uh, revamp pipeline as we start to figure out how to embed AI models into the applications that we're trying to run through a continuous delivery pipeline somewhere? It's a good question.

I think ML ops, cloud ops and, uh, data center management, like think true infrastructure management will all, I think start to blend a little bit more. There'll be, we see a lot of customers talking a lot about how they can move their continuous workloads back into, you know, capitalized infrastructure. So their own data centers versus a, a cloud infrastructure.

So if they have a predictable workload or a, a base workload, how can I move that into something that we control the entire infrastructure stack for as AI deploys, I think right now there's a huge and ML ops, you know, a deploys into the infrastructure and we want these data pipelines and we want, um, snowflakes and all those types of things. We're building them up and then tearing them down. There's a lot of opportunity for cloud.

I think as those things become predictable, it will look more like a DevOps deployment and it'll be, how do I manage traditional SLAs and hold myself accountable to traditional SLOs, to my, to my application teams or my data science teams? Yeah, It also sounds like the role of the data science team is evolving where for the most part, those are actually data scientists and they're keen to train things, but I think they, um, took on responsibilities around inference engines and the deployment thereof that maybe push comes to sho they just as soon have somebody else do on the DevOps team because, um, you know, that's at the end of the day operations, It's actually something that enterprise doesn't need to think a lot about. 'cause data science teams are keen to run a, run a process or a, a python, you know, a Jupyter notebook and build out their models through something like that.

But those aren't, those aren't often un productionized workloads and they don't have the robustness, the observability and the, all of the, the niceness that you would expect out of something productionalized. And so, yeah, I think they do wanna hand 'em off. And we do see this a lot where a DevOps person might spin up a non-production workload inside of internal infrastructure or click go on hugging face and it just, you know, runs in the cloud on a credit card.

And then when they're ready to go, a developer takes that model figures out along with a, a more traditional DevOps team, how to operate it in a scaled environment where you have distributed inference and all that other stuff that you would want. Do you think there'll be a greater appreciation for data engineering on these DevOps teams as we go forward as a result of all this? 'cause sometimes I've watched some DevOps teams, uh, work on some initial projects and it was like they were discovering a whole new world.

It, you know, I hope so would be my answer. I think, uh, sometimes I wonder if DevOps teams appreciate anyone that gives them a, a, a goal, but that's, that's just the, the personality type. I think that the, the data teams, as they build more and more application like stacks with real run, like with real run and management, the SRE DevOps blend will really start to appreciate what's happening.

What, what's tough for a DevOps team right now is the unpredictable workloads that come out of traditional data, data teams where they're very exploratory and experimental. And it's why that the ML ops kind of sits on the side. I think They say that we are gonna see more software deploy in the next two years than we've seen in the past decade.

Not sure if that's true, but if it is true, are we really prepared to handle that from a DevOps perspective? I mean, mean, it's one thing to build it and deploy and it's quite another thing to live with it forever. Yeah, I think we're, yeah, I'm an optimist.

I think we will, we'll meet the challenge presented to us, uh, and a lot, a lot of companies will come out of the woodworks to help us do that. If I were to invest or tell my kids or something in, in a new skill inside of op, the the technical stack, I would, I would push them towards infrastructure operations and understanding how, how we actually manage and build machines. It, it, it feels like a full circle from when I started programming, which was, you know, the first, the first application I deployed, I, I used, uh, one U in my house and I set up a lamp stack on it.

And I think a lot of developers have lost connection to the, the systems that, that run their software. And especially with ai, the deep coupling to the, the capabilities of the hardware, make a software, make a software developer really need to understand, or an ML ops person or an ML person really understand what their hardware is capable of so they can fit inside that. Um, over time, I, you know, the cost of GPUs and the cost of tokens and all those things will, will drive to insignificant, similar to CPU and ram.

But for now, it, it is the, you know, tokens are the currency of compute and how can I generate them the fastest is the most important thing. I agree with you on one level, but on another one, I also see as more code being pushed out to the edge. So we wanna process and analyze data at the point where we're it's being created and consumed.

And I wonder if the, there's just always gonna be a limited amount of compute capacity on the edge. So maybe we are moving back to a world where we do appreciate the finite limitations of infrastructure. It's a good point, and especially, you know, edge can mean a couple things, but when we think about edge processing on personal devices or even like closer to the enterprise itself rather than in, in the cloud, you, you might be right, big, big workloads I still think will be pushed to dynamic compute can be pushed to dynamic compute, but it's a good point.

Yeah, we'll see how the world develops from there. Ultimately, what is your best advice to DevOps teams who, you know, let's be honest, they experience a lot of toil, there's more bottlenecks in the systems than we care to admit and, and, and a lot of manual effort, and I think there's a lot of stress that goes with that, and now they're looking at all this AI stuff. I mean, you know, should they just kind of get a stiff upper lip or is there something here and they should be doing?

I Think there's a couple things as, as the DevOps world is embracing true observability, uh, and tracing over logs and metrics and all these things, I think recognizing the, the power of generative AI specifically is in making, bringing reason to significant amounts of unstructured data. A lot of applications produce significant amounts of unstructured data. And so where the legacy systems sit and all the headaches that you've been managing, uh, can you use language models, retrie, retrieve, augmented generation and kind of AI as it as it were to help you with your management of these applications?

And then how can you help your teams understand the limitations of deploying, of deploying AI applications, whether it's data governance or controlling, um, spend or whatever is important to a traditional, you know, to your organization at the time. But I, I would say lean into what AI is capable of. There's a lot that I've seen in demo and real including, you know, we're big fans of honeycomb and honeycombs capable.

You know, you can ask like, what is my most latent service? And Honeycomb can look at, you know, can look at your unstructured logs and your traces and start to make it, you know, the language models can start to make decisions and judgements on what your most latent service is and why over all that data where maybe a developer would spend hours or days, you know, looking through. So really cool stuff coming out of looking for holes or, or inefficiencies in a system.

So should we kind of maybe start thinking a little bit more about how to break down our workflows into more discreet sets so we can figure out, you know, where to apply an AI agent where it makes sense to go manage that specific task and then we'll have a better understanding of how to orchestrate the whole thing end to end. But it just seems to me that a lot of the processes we have in place have been there so long that maybe we forgot exactly what they're made up of. I think it's a good point, you know, going back and, and reconsidering our entire deployment lifecycle, you know, from, from, you know, get push all the way through the operation and observability of that application.

There's a lot going on and I think, uh, looking at how artificial intelligence could even help accelerate things like filing change orders for compliance or something like that could increase iteration speed and really decomposing and returning to like what are the core, the core principles and the core components of my SDLC and my deployment pipelines is a really great place to start. All right, folks, you heard in here, and the thing about AI and DevOps, most important thing to remember is, hey, start at the beginning. Hey Austin, thanks for being on the show.

Of course. Thank you for having me. This was great.

All right. And back to you guys in the studio.