Adoption of GenAI in DevOps with Tabnine’s Brandon Jung

This is Textron tv. Hey guys, thanks for the throw. We're here with Brandon Young, who is Vice President of Ecosystems and Business Development for tab nine, and we're talking about the state of gen AI as it applies to DevOps.

Brandon, welcome to Shout Blake. You like so much from me. Really appreciate it.

Happy to do it. It seems like we went from a massive rush of irrational exuberance to, um, somewhere along the line, I feel like maybe we're a little stuck in our adoption of gen ai. Some folks went, ought to become great product engineers and others are kind of trying to figure out how to inject this into their DevOps workflows.

And part of the issue, I think, is that a lot of the output is probabilistic and people seem to think that things should be more deterministic. But what's your sense of where are we on this journey and, and what do we need to do to get to the next level? Well, so we are on the, uh, slope of cope, as it were on the back end of the like cycle.

Just cope with it from now on. I sold it from my team brother just for reference credit where credit is due. But, uh, I think there's, I think there's maybe two aspects that we're sort of looking at.

One, I think there's a whole lot more attention being put into the security and personalization of the models and, and, and that part which we can get to as maybe a separate, but I think directly to your point, um, the notion of a prompt engineering, I think the part that maybe is missed a lot of this is, uh, first off, to be a good prompted engineer, you have to know what you're trying to prompt for, right? You have to have a very good understanding of what, uh, and, and what that outcome might look like. 'cause you might prompt and you get it back.

You have no idea where for sure, let's get it out. Except, so I, I think there's multiple pieces. I think that it's, um, it's too probabilistic.

Uh, it's not tight as, as, as, uh, personalized to, in the case of code or other things. Yeah. Personalized enough.

Um, and so I still think there's a tremendous amount of value here, obviously. Um, but I do think that this next phase is gonna be a lot of around personalization, uh, security, and then also training developers, um, to, to think about and explore code in different ways. So there is an, the quantum engineering piece, I think we're all gonna have to get better at that, whether we're a software engineer or something else.

I also Find people are struggling with the whole, uh, how do I customize the LLM to with my data? And the issue that they're discovering is that, well turns out that a lot of the data that they might need to show to the LLM is kind of a mess and it's all over the place and it's all these different tools. And, um, a lot of the DevOps folks that I know are waking up and they're like going, wow, I don't know anything about data engineer and I don't even know where to find these people.

So, um, how do we kind of get our arms around the data? Sure. So I think there's, there's a couple ways to think about it.

Uh, first off, just the personalization, uh, or customization of these models continues to move forward at a very rapid place. So tools and ways of going about that from fine tuning a model to rag to databases to graph data. I'd say like we can get to those if you want, but they functionally always personalized.

But we're coming back to first principles, good data and good data out. You have to have good data and you have to be able to trust the data you put into these models. And I, I don't think this was the, this is well known to be the case before.

It's not like this has changed. It's just that all of a sudden you have the most value. You have a very valuable application that if you have your data in good, in a good, you know, well packed up, able to put it in.

It's super powerful. So we knew it was powerful before, it's just that now you have a great, uh, tool to really get at it. And so that does put more value to the teams that have done good work around basic data quality and, and cleansing and machine learning to a certain Degree.

I wonder if we're just seeing a natural DevOps split 'cause the community is in a lot of differing, uh, phases of their journey. And about a third of them seem to be fairly well along and they're probably the market leaders, and then the rest are kinda, um, you know, close by followers and then the proverbial laggards. What's it gonna take to get the other two thirds of the community to kinda embrace gen ai?

Do we need to just make this stuff simpler for them or, or what's that missing link? I think there's, uh, a little, they're simpler then there's always just the hard part of u ui ux or a good application gets the right suggestion, right? Use the right time, right place.

Like that'll continue to be an aspect that we can always get better at. Um, I think though the two thirds, so if we kinda break them into a third, this is, is bifurcated before, to your point or in DevOps space. Um, I, I guess I am kind of encouraged about the notion as people dig back into this, that this is actually just another way to accelerate or another carrot to encourage people along a good DevOps journey of right quality code, know where your code comes from doing good security testing.

I, I see this as a bit more enforcing. So yes, authority's gonna be able to take care of this Yeah. That, um, uh, or be able to take advantage of this more so than the other two thirds.

Uh, but I think that these tools will bring, uh, a forcing function perhaps to help everyone come along there because the return so far. So I guess I'm, I I'm, I'm encouraged even for the, the other two thirds and we hit some other, other trends that I think will help that two third catch up and, and, and you know, along with the one third as well. Yeah.

Well, What are those trends? What are you looking at over the horizon that will bring the other two thirds around? Sure.

So I think a lot of what for the other two thirds has been, uh, real questions around security. Uh, trust are a lot of the issues that have slowed them down, uh, on adoption. And so, uh, what we've, I think what we've now seen is whether it's a combination of regulatory that's probably gonna push a bit this way as well as, um, a lot more, um, options for you to use fully trusted LLMs.

Uh, and to your point, some of the customization before means that the outputs are more useful in moving a team forward. So kind of tied back to the DevOps, you know, entire lifecycle, everything has come into where the suggestion is consistent and, uh, relevant to everyone in an organization or a team. Uh, and so I think a lot of this customization, uh, is something that will make these a lot more palatable and trustworthy.

And that has traditionally not been, we've kinda just thrown very large models at the prop. Like whoever's got the biggest model with the most parameters that wins, we've seen that shift a lot more to specialized and smaller focused models. I think that'll continue As we can kind of go down the path a little bit.

Um, there's a lot of folks who are worried that the developers are using these tools to write more code than ever that is gonna overwhelm our DevOps workflows and pipelines. 'cause we don't have the testing processes in place and we don't have the continuous delivery side of it as, uh, tightly knitted up as we might like. So, um, are we in danger of kind of creating warm bottlenecks?

Yes. There, I'll just leave it there. Yes, everything you just said is indeed.

Uh, what do we chat with customers? I I would say like that's one of the first, the, a very common question is will these tools help me write more secure code? I said, they'll write you good code.

Are they gonna write you more secure code that is never a substitute for having a great DevOps pipeline and process in place. Um, you DevSecOps pipeline let's, you have to have it. So oftentimes honestly, if someone asks that question, like, next spec is, so are you using for security and testing and do you have these in place?

Because honestly, I'm only gonna hand you something that's could cause more pain than how if you have a large number of developers to just keep shoving more stuff into that pipeline. So yes. Um, but I, I think i I still look at this from a positive standpoint is that's going to force some more on the front end that you can develop, will require more use of the tools that we've already had.

And the good news is like, we've got had to do this. It's a principle and a process that can be implemented all with some, some pretty straightforward discipline by any company. So I'm encouraged it'll help us all.

I'll move this along a bit faster, but there will be some pay, So there'll be something in the cost and A lot more code. Yeah. So there'll be, so there'll be something of a constitutional crisis where, uh, eventually I have a lot more code that will require me to essentially embrace more mature and better best practice.

Yes. I I don't think there's any, we, we've been seeing this as you could argue that we could hockey stick, but we're already seeing this, um, with the, I mean, over the last 15, 20 years, the, the huge growth to open source. So the volume of code, reuse of code the code, introduction of code into, into people's environments, um, that is nothing new.

It's just able to out create even faster. So, um, yeah, well we've gotta get those processes, uh, automated and, and, um, done well. Do you think That gen AI also might help democratize DevOps in the sense that it will become more accessible to a broader number of organizations that today can't really afford the level of software engineering expertise required?

Maybe if, if I make sure I'm hearing your question fully do, uh, I think that there's parts of the machine of generative AI that will help the actual process itself. Whether that's faster code reviews, uh, unit testing, uh, I think a ton of that will just better processes that, that honestly developers tend not to want to do anyway. Um, so I think, I think it'll help a lot of those mundane practices, right?

So we're gonna do some of the stuff we haven't been doing because it's as easy as saying, write me, you know, 10 tasks to cover this, this, and this great test coverage dramatically better. So I think there's definitely and pluses, uh, for for sure there. Um, I think, I think if there's, it'll be interesting to see how this plays out in terms of consolidating, uh, the DevOps space too.

I, I'm not yet sure whether this creates sort of more opportunities to think about DevOps dramatically different or it tends to push towards what we've seen a bit over the last, you know, five, 10 years of a consolidation of DevOps, uh, like a platform view a bit, uh, more like a, a GitLab or a GitHub path. So I think it's gonna play a little bit of both ways. Well, it would seem to me at least that there's a lot of, uh, small to mid-level organizations that can't really do DevOps today.

And theoretically they might be able to do it tomorrow because the, you know, I'll be using graphical tools and prompts versus everything won't have to be a script that I gotta go get somebody to write and maintain. So yeah, in that sense, maybe DevOps becomes more widely employed For sure. So I'm gonna be super specific.

Definitely in terms of old scripting, everything around like infrastructure's code for example, like it is all suited for that. Because while we may like or not like the fact that we are fairly consolidated to a handful of platforms, uh, and a handful of tools, uh, to deploy and everything else, platforms, that certainly will be the first, like that's already, and you can see that really in the large clouds doubling down on, um, uh, terraform and uh, cloud information scripts. And each one of those is being by very first places that, that people move.

So, uh, I think it will help for sure, it'll help, lemme say it this way, I think your core DevOps team can be more efficient in adopting these to cover, get better coverage, faster velocity, but they're gonna have to wrap their head to r uh, that, so yeah, so I think it'll be helpful, but it isn't gonna be without pain and a pretty good learning curve. Ultimately. I mean, some folks you talk to, they're worried that there'll be more code moving through the systems that'll be generated by large language models that, uh, we're trained using code that was, shall we say, imperfect and there's more vulnerabilities than ever going through their system.

But to your point, if we have more domain specific language models and smaller ones, at the end of this, might we wind up with better quality applications? I I, I'm super encouraged that we can, and, and that again, I'm, I'm, I'm pretty at first going from the very large models that is what we, we were talking about to specific custom models. And if we think about that a little bit more broadly, when you look at who's the expert domain experts, you know, if I'm writing a, you know, uh, uh, complex front end application, I want the best, you know, the, the best way of running JavaScript.

But I also want the best practice from Mongo, and I want a little bit from Redis and I, here's what my stack looks like, right? And we do have an opportunity is if we put trusted data into these models, they can really help developers, for example, new developers not gonna keep up with all the newest best practices or security concerns from, uh, you know, a large is v those ISVs. If we can work this, um, more holistically together into specialized, specialized models, now I have a model that's very specific to my stack with my svs with best practices all kind of baked into it.

And I think that's a, to me, that's, I think where we end up, it's a lot more efficient. Uh, it does require the developer like, choose, like you're gonna have to know which model you choose. 'cause if you do a backend versus front end, one model will not work, and the other, you can't just, they're interchangeable.

Um, but I think once you go where that hurdle where developers are used to going and selecting, um, a model to match to their development environment, what do they accomplish? Yeah. That the nearness of specificity will allow for a dramatically both better model and a more trusted model.

So what does the future of a DevOps team look like to you? Uh, in my mind, I can imagine, um, yeah, i's say a handful of human DevOps engineers augmented by a bunch of AI agents that are performing specific tasks. But then I gotta figure out how to like orchestrate that and put the handoffs together between the humans and the AI agents.

And is that gonna be the art of this thing? Uh, the art of the thing is, is gonna be, uh, and I think this cuts this generative AI in, in, but even our society in general is like the value are gonna be that t-shaped though people have a, you know, fair, fairly good at a wide number of things, but won't be as good in the depth, uh, of those. So we think about those different capabilities, there's gonna be high, um, value placed on someone that understands all how the pieces fit together, more architects, you know, maybe a level deeper than an architect.

I think those very specific super deep skills in, um, a specific light or just specific area, those are gonna be the areas that are gonna be most easily replaced, adjusted, uh, uh, made more proficient then. So I think back to your point, I think it's understanding how all the pieces fit together, because you're gonna have to look at it and go, what's working, what's not, what's broken, what's not? These tools will certainly help diagnoses a little bit easier, but it doesn't, what stops with us, right?

The end of the day, whether that system's working or not working, that code is secure or not secure. Um, that comes back to the, the team that that wrote it and the team that deployed. Mm-hmm.

There is this debate that's going on out there. Um, and the question is, is will Jen and I make it easier for experienced software engineers to take care of more processes and therefore we won't need as many entry level people? Or will Jenna and I make it possible for the entry level people that do more sophisticated things and we won't need as many hardcore specialists?

I think the answer is you'll, it'll actually affect both and in, in, in ways that a newer developer can come up to. So I'll take it from kind of both sides. So I'd say both and a newer developer will be able to get up to speed better, faster, particularly if they have a specific model and a, some a situation that's more personalized to their use case in an enterprise situation, much, much more valuable for a new developer to come in and, and pick up and contribute.

Um, but, uh, on, on the, on and on the senior side, um, obviously it takes off a lot of the burden for most senior engineers in a small team, uh, to be able to take care of like test coding and a lot of the, uh, work that was, you know, um, less le less or more repeatable. So writing your tests and, uh, doing all your testing and stuff. Um, I, I think that the challenge on that is still going to be though when you run into a challenge where something's not working.

I think think that's, uh, I think the senior engineers always has to be there because the end of the day the senior engineer is gonna have to go through not working. They're gonna have to figure out where that broken piece is, uh, and why it's not working. So I think it helped for both.

That said, it's also gonna be a challenge for both, right? Um, I think the other part that hasn't been highlighted as much, sorry, just a side note is a lot of senior engineers time right now is reviewing code from other developers. This, if you can personalize the models, enables that to work much better and allows a senior developer to spend more time in writing complex and solving harder problems and less of the review.

So you have to optimize it, you have to good custom model, but there's potential for both sides to new developers write better code and senior developers to spend less time on the reviews and the coaching that maybe they had to it, uh, at the beginning. So ultimately, what's your best advice to DevOps leaders about how should they bring all this in? How do I acclimate Every optionality?

I, I, I think you have to just be, I, I think you have to optionality. So, um, where this is gonna evolve is anyone's guess in terms of how we make these models more, um, usable, uh, secure, et cetera. So, uh, we always start to customers look for example, do you want to use the best stuff?

Uh, inops three, like cloud three five, which is doing fantastic, maybe yes. And there's a plus of that, but you may be European country that in six months you're gonna get the answer that says, Hey, that's nice, but I have to use a model that has only trained, that has no copyrighted code in it. How do I sit in a world where I need to be able to work with developers on any ID I need to be able to customize against my code and anywhere I'm gonna move and I need to be in a position from a regulatory standpoint, if it gets either resolved in the weight route of I have to use, I I have to have a very secure controlled model, or I can open it up and go after the biggest models anywhere.

How am I in position that I can start today? Whether that's, you know, gates wide open, go forth, go fund quad three five, or you wanna go fully secured view, uh, where I know exactly what's coded in it. Am I adopting using a tool that meets all my developers wherever they are and in a way that gives me optionality going forward?

Because the truth of the matter is no one even that pretends, oh, this is going awesome 'cause I don't, so I think optionality is the piece that we really talk to people about is, yeah, it's changing so fast. Try to keep those options open, uh, in terms of what models, um, however, which what it works into your, um, DevOps, um, platform, wherever that might be. Uh, I think those are probably pretty important 'cause it's far from decided how this is gonna play out over the next two, three years.

Alright folks, you heard it here. If you feel like your options are narrowing, you probably heading down the wrong path. So take a minute and make sure you keep them open.

Hey Brandon, thanks for being on the show, Blake. Thank you so much for having me. All right.

And back to you guys in the studio. I.