The Future of AI-Augmented DevOps with Tricentis’ Mav Turner

This is Textron tv. Hey everyone, it's Alan Sch Schell. Welcome back here to techron tv.

I'm having to be joined by my friend Mav Turner. Thanks Ram Mav, it's great to have you back on. It's been too long.

Um, ma'am is our chief pro. He's not our chief product officer. He works for a different company, but Mav and the Chief Product and Strategy Officer for Tricentis.

Um, if you watch our DevOps Unbound show, mam has been on a few times over the years as part of our panels. They're both the live version and the recorded, uh, round tables. He's also, usually, he's a fixture at Q Condo.

I'm bumped to hear that he's not going to be personally at, uh, salt Lake City in the next two months, but we'll catch up what we're catching up to him now. Um, and he's always the man in the know with timely topics of what interests folks like you out there. So Matt, I hope I didn't embarrass you first of all, but thanks for coming on.

Um, I, I said you're the chief product and Strategy officer. I told you a chief product officer strategy and new portfolio added to that, or was it always, and I didn't realize? No, it's, it's, we change it all the time, so it's not, oh, check not on you.

Yeah, yeah, don't worry about that. So, you know, we, we've been making, tus has been growing a lot, uh, over the last several years, and so as we're growing, we're expanding my role continuously kind of changes and shifts around depending on what we need. But, uh, you know, I I, I think most of your viewers are probably familiar with Tricentis, uh, but for those of you that are not, uh, you know, Tricentis really is known for this, you know, quality engineering, quality automation story products like Tosca, qTest, Neo Lo, uh, recently, you know, we've added a lot of products on our portfolio with Testim and recent acquisition of sea Lights.

And so we're really just continuously trying to make sure that we have the right tools and recommendations on processes for businesses to deliver services with higher quality, uh, faster release times, you know, increase that, that cycle times so that you can have those, uh, business services just running as expected. And it's, it's a really hard challenge, obviously, for a lot of, a lot of us have to do it daily. And so that's what Tricentis is really trying to solve.

And, and my, my role, obviously, I lead all of all of product at tricentis. Um, but the other part of that, the strategy part, you're, you're tying on there is as we continue to grow and expand, what more can we do? Where are we missing areas in the, and in the market that customers might be having pain points.

And so that's a signal from my CEO that he wants me to spend more time on that versus the, you know, operational getting the, the, the, the products out the door. So that's kinda what that means. Actually.

I'm surprised they picked up on it. See, like when, when, when was that acquisition? We just closed that in, uh, August.

I can't remember the exact date, but about a month or two ago. Yeah. So that's been, that's a recent acquisition.

I know the Sea Light did Israeli company, aren't They? Yeah. Israeli company, Iran.

You may have met Iran before. Yeah, yeah, Yeah. I, I've, uh, we've been following them for years.

Good for them. Congratulations to Tricentis and congratulations to the Sea Light team. We, we've done a few interviews with them over the years.

I followed, they did some nice IP product there. Absolutely. They've been in the industry for a long time, and so they understand some of those new problems and, you know, um, how to solve them.

And that's really what what we're looking at is there's a lot of just critical work we have to do in our e products, but it's also how do we continuously move the envelope forward? How do we have the newest thing, you know, a lot of my customers will say, great, I've deployed everything. I'm fully automated.

I've got all these insights, everything's awesome. What's next? What do I need to do next?

And this is where CLI come in with this quality intelligence story around how you can get insights through your build process, which ties I think, really nicely to some of the things that, uh, that's recently report we had around, uh, around DevOps and, and AI and how that's all coming together because they, they, they, they're, they sit in the BUILD pipeline so they can actually see the build, inspect it, and then map changes from, you know, where you have missing, um, code coverage and whether or not you wanna allow that bill to pro to progress into production or to your next level environment based on ensuring whether or not you have proper code coverage. So, uh, and, and they have some interesting AI things that are, that are more around the corner than today, but, uh, but that are exciting and nonetheless. Excellent.

I I love it. Um, you mentioned this recent report, and it is on, you know, kinda trends between DevOps and AI and every look AI's everywhere and everything today, obviously, right? But, but specifically, this is on the future shaping the future of what we're calling ai augmented DevOps.

And before we even jump into the report and findings and stuff like that map, you know, that's a loaded term, ai, augmented DevOps, it sounds kind of, uh, what Robocop ish, right? Augmented cyborgs or what have you. Yeah.

What do you, what do we mean by AI augmented DevOps? Yeah, I, I agree. It's a very sci-fi term when you talk about augmentation.

It's, but, but I think it's a really important term when we think about how AI is impacting everything to do with software development, testing, delivery, et cetera, because we're not at, we, we don't wanna replace people, right? So this is not saying like, oh, okay, AI is replacing everybody now we can just go hang out on the beach all day. As much as I would love that.

Um, we're very much augmenting, right? I'll, I'll use the Iron Man suit example, right? We're giving you super powers to go faster, do more, deliver higher quality.

And that's, that's really kind of what we're trying to unpack a little bit, and which is how much are people actually using AI in a DevOps process today? And that was one of my, my surprising, uh, surprise findings actually, was just a number of people that are saying they're already using AI in their process to augment their work. Now, they're not saying, therefore we, we need half as many people.

Um, because the, the, the reality is the business expectations are always growing faster than we can handle the work. And so we believe that AI can help, and that's, we just wanted to get more details about how people felt about it. And that's back to your point, like, what did we mean by augmenting?

And that's really the, the what we were trying to piece apart, right? Absolutely. So you, you know, you, you can't leave that hanging, uh, purchasable there or whatever, but how many people are using are, are augmenting Yeah.

Yeah. With ai. So it was a, it was a global survey, uh, across all company sizes, about 500 DevOps practitioners, managers, executives, and of that audience, when we look at what they said, um, how many are using it versus plan to use it, uh, one of the other, before I answer your question, uh, one of the other important parts of this was, um, how they define where they're at in a maturity curve, right?

And that's a self-assessment. They're saying that we're immature, very mature, et cetera. Um, but in that, when we, we kind of segment the data out, when we look at the more mature teams, um, they tend to be, uh, using more ai.

So the about 30, they're 30% more likely to be using AI if they self classify as mature, um, and say that their teams are effective or very effective. That's kind of the, the terminology we use there. And so about 30% of those, um, are, are using ai.

And there's specifically, if you look at a breakdown of that data set, like how are they using it? Um, developer efficiency about, uh, 60% usage there, reducing the skill gap of developers. And this is if, if, you know, I jump from the quantitative research results to the qualitative customer conversations I have when I talk to people about how it's helping them, typically they say it's more about how a junior or more entry level person can be more performant versus that principal engineer who's typically, they might use it in small parts, but they know how they want to craft this and deliver it.

And so it that, which helps with the skills gap, right? It helps these newer people coming into the field be more productive faster. I think we saw that bear out in the, the survey results here.

You know, I, I, I think it's important that we emphasize this is not just QA folks, this is DevOps people, which ranges, you know, from left to right all the way through the, the, uh, CICP pipeline development and everything else. We've been doing a lot of reporting and discussion and content here around this whole idea of just how embedded, how adopted is AI in the DevOps lifecycle in the SDLC. And I, I think you hit on something that we want to, we wanna hit, you know, make emphasize, you know, right now there's probably, they say, I don't know, 27 million developers in the world, some number like that, depending who you believe.

And that in the next three years, three to five years, that number might go up to 40 million, big jump, 20, you know, 150%. Yeah. Um, however, when you look at what AI can mean to developers, you know, by 2035, we may very well have 500 million developers because everyone will be a developer.

And How you find developer. Yeah, yeah, Exactly. It may not be the developer you're looking for, you know, to quote Star Wars.

These are not the developers you're looking for. Yeah. But, um, but anyone will be able to, you know, talk to their AI and natural language and say, build an app that does this, this, that, that.

You may want to tell it what language it is, or just use the best language for it. It'll do it. Um, this is, I think that's a really interesting, sorry, Alex, if I can kind of go ahead.

I think it's a really interesting trend across society. I really love how you capture that, right? What is a developer?

Is it somebody that can create, has a concept how the problem they wanna solve and they can deliver a service to it? Does it mean that they're writing code or operating their pipelines or operating SA service? It's, it's how they can interact with these, these different, you know, we talked a lot about this back when low code started to really kind of gain steam, right?

Go. And so there is kind of that next evolution. So like citizen developers, right?

Citizen Develop. Exactly. Yeah, exactly.

And so I think this is the next order of magnitude step and the ability to get towards that, which is, which supports kind of your theory around how many people in the world you would classify as a developer. And I think that's the right way to think about it, right? It's, it's, um, it's somebody who can deliver some business service, some application that didn't exist before, and do that with quality and, and in consideration of all the operational concerns that come, come with doing that and updating it and making it accessible to users and, you know, um, reliable under load.

All of those things are very challenging. And I think that, um, AI can, can help a lot with that. I, I agree a hundred percent.

Now, map, let's take this to QA for instance, right? You know, one of the things, one of, one of the big changes that DevOps brought to the QA world was, rather than the QA person actually having to sit there and write tests and run tests and watch the test and, and all of that, um, a lot of QA people were freed from the mundane aspects of that with automation and stuff, but they had to know what tests they needed to run for adequate coverage and then script that out so that those tests would run, and then those tests would run sort of on automate, right? Well, now with ai, it's the same kind of thing.

I don't actually have to script out what tests I want to run. I I could tell the ai, or if I can't tell them now, probably in the very near term future, right? Give me, you know, broad spectrum coverage of testing for this application, usage testing, whatever kind of test I wanna run and go do it.

Right? And that makes everyone a QA professional. You're at a some level, right?

What, what, what's this gonna Mean? Yeah, yeah, yeah. Yeah.

A couple of things on that. Um, one of the things that, with the way I look at it, that trend over time, I think of that movement from QA and quality assurance and that being a very manual thing where you're looking at a script and clicking buttons on a screen, then you get more to quality engineering where you build out automation and you're looking at cycle times and you're starting to think through, uh, instead of a human doing this part of it, now they're doing a higher, you know, value task. And then that the next big stage is quality intelligence, which actually ties into our sea lights conversation from earlier, which is I'm now looking at the insights and the data and, and starting to think where do I have gaps?

Where do I have coverage? And, and, and not to say that people, testers, quality engineers didn't want to do this before, didn't know that these things existed, but they were so stuck with this, these other tasks to deliver that it was really hard for them to get out. And so that journey from QA to q uh, E to QI is really what we're trying to accelerate for customers.

So that, yeah, they don't have to spend a bunch of time writing scripts or, or building the automation and more they can leverage the thought process about what do I need to cover? Where do I have gaps? Where's my negative test coverage, my performance test coverage?

And, and from us, from an AI perspective, we, we, we, we talked about that in the context of autonomous ai being able to kind of generate all this out. Now it's still a drafting stage, it's still a recommendation. You still want that human to look at it, but hopefully you're just re freeing up enough time so that they can be thoughtful.

And the next thing that we're working on more a next year thing is that, that imbuing that AI with the knowledge of testing so that it can help that that person look at this and, and make open it to anybody who, who may not have as many of those skill sets and experience and background to look at a problem and look at application, understand, Hey, it looks like you're missing this. Uh, again, still interacting with the human right, still augmentation, right? Not, not, not replacement, right?

But trying to embed those skills into that AI so that the human can deliver better quality application and services. Yep. So bringing it back to this survey now, this is why I'm not surprised to hear that, you know, maybe more junior people are, are using AI or more likely to use AI more because it does that augmentation gives them access to capabilities they may not actually have yet.

Where a more skilled person, you know, they, they have that down, they, you know, they've already, they're further along, they're more mature, to your point, that automation cycle and that you that path. So it's interesting. I want to talk more about, you know, we went down to, we made a little left turn there, but let's bring it back on the highway.

Yeah. So Matt, tell us more about the, this report. Yeah.

Uh, a couple of other key findings from this report. Um, you know, one of, one of the other exciting topics for people to talk about always is compliance and regulation. Um, but in the context of ai, it's actually, you know, there's a lot of things evolving very quickly here.

And I was really surprised that, uh, the results we saw was 63% of those surveyed viewed that an increase in regulation was a way to build confidence in AI across the organization. So I think really quickly, all the way down from the technical layer, all the way to the business layer, everybody understands the risk associated, maybe not the full details of it, but that there is risk that needs to be managed. And so they actually view regulation as a positive thing, which is pretty surprising that it's not normally the case.

Um, but it's a great, it's a good, I, I, I'm a big believer in it. Um, and then on the, on the flip side of that, about 16% felt that that increased regulation would stifle or hinder, um, the impact of ai. So they thought it could be a little bit, um, slow down for them.

So this, you know, it was, it was interesting results that I, I was surprised people were more positive to generally, I see those ratios flipped whenever you talk about regulation and compliance and technology. Yeah. Well, I, I think part of it is, look, the media and, and others have really been banging the bells about the AI boogeyman and, and you know, we need some sort of guardrails.

I think people like having guardrails. Yeah. The flip side of that though was someone who's been in security for a long time, in many ways, regulation and compliance was the worst thing that we had in security because it tends to be very low bar, right?

Yeah. And, and I, I, I'm not saying that's the case today with AI yet, but I generally speaking, you know, that compliance or regulation is kind of a least common denominator. Not necessarily the, the high thing Necessary, but not sufficient.

And the challenge is customers will often think, oh, we did this so we're good and secure, check the Box, right? And check the the, and then, yeah. So that's the risk.

To me, it's a good thing 'cause it sets that standard like you have to be doing these things. I think there's a lot of value in that en forcing, you know, these best practices. 'cause it's easy after, and we're going down another side road for a minute, some security event happens, you get the ambulance chaser saying, ah, follow best practices.

And then it's like, what are the best practices? Why don't you tell us? It's easy to say that after the fact.

Um, oh, it's, but, but so, so the standards help with that. But, but it can be make businesses complacent and think that they resolved or manage the risk that hasn't really, uh, been fully managed. Well, they've done something.

They, they've done something. Yeah. Better than Nothing.

You know what? So I'll fall back on my legal education. I think when you do what the regulations ask you to do, it's hard to say you are negligent.

You've done what a reasonable person should do. Yep. And reasonableness is the antithesis of negligence, right?

Negligence is unreasonable. Um, so in that regard, it is what it is. Could you do more?

Should you do more? Yeah, probably. Right?

Yeah. But nevertheless, it's there. I think the flip side of that though is do they believe there's enough regulations in place, or do we need more regulations?

And that would be Entry state. Yeah. The result was that, that those folks were, um, relying on more regulation in order to help them get the rest of the business comfortable with deploying of technology.

'cause there was, I, I'm sure you saw this, right? There was the immediate, everybody got super excited when chat CBT and OpenAI. And then all of a sudden everybody, once they found out they were updating public models with confidential data, then all of a sudden all these initiatives came down.

No ai. And so these kind of guardrail to guardrail, uh, which again is new technology. That's, that's, that's the nature of it, right?

Yeah. Now, you know, I was talking to someone, we were, I guess it was over the weekend, we were out to dinner with a couple of couples, and one of the folks was saying that in their company, they're not allowed to use AI except for Microsoft copilot. Yep.

Yep. Alright. You know, if that lets you sleep at night, God bless you.

But, um, but there's, I think that is the world we're living in. Any other interesting facts, findings? Yeah, I think, um, you know, there, there, there's a ton of data here.

com resources reports, you can find it there. You can just Google it. That's usually the easiest way.

Uh, it should just simply, uh, you know, Google ai, augmented DevOps trend shaping the future. But, but one of the other things I'd like to mention, uh, real quick is just the, the human and the loop expectation. And again, we talked about this a little bit earlier, um, but when we, when we did the survey that people were saying that over over 71% of the respondents said that they have a human checking the outputs at least half the time.

Um, and then a, an incredible, what I thought was, uh, 19%. So one in five roughly said that they check every single output. So kind of that, where, where's the, the right place on the spectrum where if you're checking every output and you're putting all the energy or getting the benefits of the technology, uh, Well, but I I, but it's the stage I suspect that'll come, I suspect that number will come down over time.

Yep. What it really says is we have a lack of confidence in the veracity, and we're worried about poisoning and false information. Yep.

And you know, and that was a very real Thing. A hundred percent. It is still a real challenge and a lot of our customers struggle with that.

Uh, but to really get the benefits of it, we have to figure out how to improve that. Reduce hallucinations, show the value. Because if we have a human sitting and watching a screen versus doing it, then we haven't really moved, moved the ball forward, right.

For business value. I, I'll tell you what I think, you know, that I mentioned, we've been talking about this topic here on Textron text, on tv, on our Textron gang, which is our daily Yeah. Kind of the view meets Fox and Friends meets the morning joke show, got everyone covered.

Um, something like 30, over 35% of organizations are now using custom, uh, LLMs custom data Yeah. For their, for their ai. So what they're saying is, look, we could cut down on the bad data, on the hallucinations, on the mistakes by controlling what data goes into our model, right?

Because our models are then more specialized, sanitized and, and all of that stuff. And I, I think that's another big trend in this augmented DevOps ai, augmented DevOps space is people are increasingly relying on their own models. Yeah.

I, I think, um, I think we will continue to see that in, in more spec special purpose balls deployed in specific scenarios. I would just, yep. Um, kind to go back to the other extreme on the risk management, um, just always remember that by definition, if you're using generative ai, uh, your outputs are non-deterministic, which is why it's really bad for like math problems 'cause and in really bad application for it, right?

So, so making sure your system, your AI system has the controls in place, particularly given the specific application you have, such that if the outputs, no matter how, you know, fine tuned they are in your environment, your testing, uh, when they're in production, that you have controls in place to capture that. Because, uh, I think we all apply our mental model of deterministic systems and computers where, you know, we know exactly how they're going to respond. And even in those systems, we have software issues and bugs and all this complexity.

When you add this non-deterministic nature in there, that that completely is a different paradigm. And we need to be mindful of that when building these systems and ensuring we know how the different data connects. Even, even if it's very fine tuned and specialized, uh, you're still gonna get, you know, different answers sometimes.

Absolutely. Yeah. As usual, a great conversation.

I can talk to you all day, but I got people in the waiting group for our next, uh, bottom of the hour event. It's great seeing you. Great catching up with you.

I'm sorry I won't see you. It's all Lake City. I think London is the European cube con.

Yeah. Uh, this spring maybe. We'll see you there, Babe.

We'll see you in London. And as always, great Alan, thanks for having me on. Great talk with you.

Absolutely. And come on, we haven't had you on DevOps Unbound in a long time. Talk.

You've got some pull over there, see what you can do. Yeah, yeah, yeah. I'll see if we can do, do something about that.

All right. Thanks Alan. Mav Turner, chief Product and Strategy officer at Chi Sentence here on Tech Drug tv.

We're gonna take a break. We'll be back in a moment.