Mav Turner on Revolutionizing Test Automation with GenAI

This is Textron tv. Hey everyone. We're back here on Textron tv.

I'm happy to have, you know, before Christmas, and he's in his red sweater too, so how cool is that? One of my good friends, Mav Turner, Mav, if you don't know him from his past appearances year, is the Chief Product and strategy officer at t Tricentis. Mav.

Good to see you, man. I'm glad we had a chance to hear, to catch up before the holidays, I hope all's well in your world. Yeah, doing great.

Thanks for having me on. Uh, great. To kind of close this year out, uh, with the, with the talk with you about, about ai.

It's always fun. You know, we were doing a, uh, a review earlier on Textron Gang for Tomorrow's Textron Gang, and it was sorta, so we had Steven Foskett who does Tech Field Day, I don't know if you've ever done a Tech Field Day, but sto he was talking about, uh, you know, big stories. So the big stories for 2024 for him were AI and then networking in ai, data infrastructure in ai, storage in ai, this in ai, that in ai, everything and ai.

So why not talk about AI in this, in this segment as well? Right? It's par for the course.

That's, I think when we look back, that is the story of 2024. Absolutely. So Interesting and Funny you mentioned, uh, Stephen Foskett and the Tech Field Day crew.

So at, in the previous, uh, companies, I did a lot with them actually many, many, many years ago. Um, so I've done several of their events. Uh, I think there's some YouTube video from, you know, 15 years ago with me.

Yeah, Yeah. No, like 15 years. Well, that's gotta be when he first was starting out.

Like at the very be that's how long it's been, about 15 years. Yeah. Uh, well, you know, as, as you probably know, we're, we're in the middle of a merger with, uh, with a Futurum group, which is the owner of Tech Field Day now, though Stevens still runs it.

Yeah. And so Stevens, we actually show all the Tech field days on Tech Drunk tv. Yeah, they're, they're streamed live on tech Drunk tv and, and you can also get them on Techron tv in addition to YouTube.

And Steven's a regular on our tech, uh, Techron gang show. He's on usually every Tuesday show. Uh, but yeah, it's a small world, six degrees of separation in tech, right?

Yeah, yeah, Exactly. Yeah. So, Bab, you brought it up.

Look, you've been around tech, you know, for more than a day, right? And you're now Chief Product and Strategy Officer at Over at Tricentis. But give us a, you know, a a an idea of kind of your background and what you know as Chief product and Strategy Officer, what, what your responsibilities are at Tricentis.

Yeah, Absolutely. So, you know, like you said, I've been around for a couple decades now. Uh, technology, oh, actually, I'd hate to say almost 26 years probably now.

Uh, deploying technology, managing technology, building technology At Unleash. You have hair. Yeah.

You know, good for You. I've been fortunate to keep that. Yeah.

Uhhuh. Um, so, you know, one of the things that, that I can, you know, go back to is just how these trends change over time. And so, you know, we will get into that a little bit more, right?

So when Cloud started to pick up, and everybody said we wouldn't use cloud, a lot of the AI trends when mobile started to pick up all of the, you know, just how and how technology evolves over time and, and we're in that cycle now, and it's been interesting to see with ai, how, how quickly that's been going. But, you know, at Tricentis here for about three years now, and really just focused on how can we help teams that are delivering business solutions ensure that they're doing that at quality, right? How do we ensure they have proper testing and then that, that's done in a modern way.

How can we enable these teams to ensure they have visibility over what they're delivering? This is where products like Q tests come into play. You know, do we run all of our tests, whether they're manual, whatever tool set you're using, and how do we bring that visibility to ensure we are ready to release, right?

And, and that those cycles are just accelerating, uh, compared to where it was back on prem or where we had boxes that we were, you know, flipping disc inside doing the, I remember doing Windows 95 installation from 30 floppy disc, right. You know, we're, we're, we're past that stage. You mean both?

Yeah. I remember both the bigger disc and then the little three and a half inches. Um, man, those were days anyway, you know, Matt, we've been Mav, excuse me, Mav, we've been lucky for the last four years or so.

We've partnered with tricentis on our DevOps Unbound video series and webinars and so forth and round tables. And you know, so over those years we've had a really a chance to really get to know the ENT team and the Tricentis story and the product line and customers and, and so forth. And, and it's interesting, I used to always feel confident in saying, you know, we're happy to partner with tricentis, the worldwide leader in continuous testing.

Right? That was the tagline. That was the tagline.

Yeah. And I've used it for four plus years. Um, but over those four years, continuous testing itself has changed, right?

And those small part thanks to ai, right? AI's been a huge, huge player in there, but the t Tricentis game has changed, right? Acquisition, so ho you know, or both organic and and acquisition kind of additions to the thing.

I think our audience knows T Tricentis, they've heard me say the world. I feel like Keith Jennings on the wide World of sports, the joy of victory and the agony of defeat that, that dates me. But anyway, you know, t tricentis the worldwide leader in continuous testing, but what's behind the tagline these days, Matt?

Yeah, yeah, absolutely. And, and you know, not that I shy away from using that tagline, but it is a bigger story to your point, right? We've built a lot of technology, um, we've acquired a lot of technology.

And so really we try to expand that out. And one of the things that was big for us in 2024, other than ai, which, you know, we've, we've been doing AI for many, many years before I go into tricentis. Uh, it just as that technology has changed and evolved.

So if we, um, but really what was big for us in 24 was if you think through QA and the term, a lot of organizations go from QA to qe, right? Quality assurance to quality engineering. And really for us in 2024, with the acquisition of Sea Lights and the mindset behind that, it really evolves to the next stage, which is quality intelligence, right?

Which obviously kind of plays in with some of the AI themes. Um, and, and really being intelligent about how you're thinking about your testing. Not implying it wasn't intelligent before, um, but the insights that you need to order to like operate a mature organization.

Uh, when you look at some of our most mature companies and how they operate, how they look at the data, how they quantify it, how they improve their processes with that data, that's really where we, we kind of we're going today and, and is enter continuous enterprise testing part of that. Absolutely, it's the foundation of it, but what does that look like? It's, it's, it's more than, and some of the features that we've had in those products has have helped, you know, what do I, what are the test case that I wanna prioritize based on risk, risk space testing?

All of that is, was a core part of that enterprise continuous testing message. But now when we think about quality intelligence, it's that next generation of thought process. When I think about how do I deliver business services with the highest level of quality, especially given the shorter cycle times, uh, where I can't afford to run all of my tests.

How do I make sure I've got the right coverage gaps without having to go through and manually look through things, right? So that quality intelligence journey, mainly on, on the, you know, propelled by the sea lights technology has been a big evolution for us, particularly in this, uh, this, this last year. Absolutely.

So what, is there a new tagline I'm missing? Uh, no, uh, I mean, you know, we, the, the, the main tagline to, to think about for, for tricentis is really just delivering quality business solutions, right? So it's kind of upleveling from an enterprise continuous testing, which is a little bit more technical, and try to level up that message to a higher corporate value.

Because that's one of the things that we've seen is a lot of the, the companies that are the most mature, the most successful, are really ones that are thinking about quality holistically across their entire enterprise. Maybe they have a COE. Um, they can, they have these expertise that they can use to infuse across, not only in their IT teams, but also in their engineering practices, which historically have been a little bit more, uh, distinct teams and operations and tool sets.

And we're starting to see folks that can really leverage that together across, or the ones that are really growing. And so that, that therefore levels up the conversation in the enterprise. Fantastic.

Could you also say that, happy to announce that our relationship with tricentis on the DevOps Unbound, uh, series, this video, as I said, it's videos and round tables, a web, web webinars is gonna continue in, uh, 2025. So we're really excited. That's the fifth year, you know, you don't have these kinda longstanding relationships in tech, right?

People, people hop around jobs every 18 months for god's sakes, right? Uh, so it's, we're really excited and proud and happy to, to, to have that relationship. Absolutely.

We, we, and we, we, we value it as well. We think it's a great way to, I mean, just such a great community you have, uh, and the content you bring, there's not a lot of other places like that for people to really figure out what's going on. So, but we appreciate it too.

Thank you. I appreciate that. Alright.

Enough, enough Naval gazing. Let's, let's jump into our topic of discussion here. 'cause I'm blushing.

Um, you guys recently announced Q test copilot, right? And Yep. You know, originally when I saw everyone using the name copilot, I said, oh, Microsoft's gonna sue them.

You know, they, Microsoft sues every, but, um, the fact of the matter is copilots become a very generic term, right? Yeah, absolutely. For AI assistance.

And so we have Q test copilot and for test automation look from, and because copilot has become a generic term, we could kind of guess what it is, but give us the real, the real deal. What is Yeah. QTS copilot all about A Absolutely, and I'm glad you called out the, the kind of branding, the naming here.

'cause you know, whenever you're building products, that's one of the things that's really important. And, um, I'll, I'll be honest, I'm not, that's not my favorite thing to do is come up with a brand naming, naming name. I was always from the school of name it what it was.

So like, when it does, I did a, yeah, yeah. Mitchell and I, you know, we were two of the three co-founders of a security company, and we came up with a product that was endpoint configuration check, uh, enforce Management or something like that. It was like, it really rolled off your dude.

Yeah, it sounds like Tongue. And then Garner called it nac. Yeah.

And so that's when it, there it became network access controlled nac, thank God, because our name was ridiculous, which I, I hear you. Um, So it, but it's important what you say, which is people recognize it and when they hear that copilot name, they bring assumptions to what it does. And so it simplifies some things there.

Now it also complicates things because people think, oh, it's just, it, they think it's Microsoft's copilot. And we're like, no, no. Um, and to make matters even more confusing, it is not Microsoft's copilot, it's not related to GitHub copilot, and you give that, but it is BA based on the Azure services.

So the underlying large language models, the foundational really. So we try to make it really confusing for everybody. But, you know, we use some of the, That is, that is a, a wrinkle into that.

I wouldn't even tell people that. Should's going to confuse them, but go ahead. Yeah, yeah.

Well, it's one of the questions we get a lot when you, in generat value, particularly at the beginning when it started, when everybody really started to blow up was, you know, what is the model using? Because particularly when you're using OpenAI directly, there wasn't a lot of controls and security and foundational models are getting updated with confidential information. So that's honestly, that responsible AI topic is usually when I'm talking to customers where I start, I'm not gonna spend too much, too much time there today.

'cause I wanna get into some of the, the meat of the tech side, but, um, No, no, but you're right. I mean the, the model and, and the other thing we're seeing is that a lot of end user organizations don't want to use those Uber models, right? Right.

Those big models, they want their own models, you know, whether it's small language, moderate modules, s SLMs Yep. Or, or something, you know, based on their own data so that they can control a little bit more of, of what that is. But anyway, as I said that we could talk about that till the cows come home.

Yeah, yeah. Let's jump into qTest. So yeah, so exactly the, that, that whole topic is evolving massive rapidly and, and, and massively and the changes.

But when we look at qTest, really it is around, you know, first what is qTest? So we, everybody kind of knows copilot. That means generative AI is gonna help me do some things, right?

That's what a copilot name means. Yep. Q test test management, right?

So this goes back to how do I pull in all the signals across my organization to know that I've done what I'm supposed to do to release my product and then, and I'm not gonna have, um, quality issues, right? That's the highest level. So test management bringing this in, and the larger the enterprise, the more complex the systems un the under tests, um, the more the need is for this type of solution.

So Q test copilot is a copilot that helps you do this. It helps you write or generative AI that helps you, um, write out test cases, make sure you have proper test coverage and really elevates overall your quality. And so when I think about, like, some of the customer codes, I can't, I can't name them, but, um, you know, things about, uh, I can get this complex test case in seconds compared to 20 or 30 minutes it took before.

So that's great, great productivity value in game, right? That's something where before, um, the way qTest works is you can have a little, a sentence, a description, and the copilot will just always generate test cases based on that. And so it helps you do it.

And so one of the cool things there is when you're looking at it, um, my favorite two quotes on this, were q test, copilot got me thinking about how to test my requirements in a different way. So that's amazing, right? That's if you're, if you're really challenging how somebody's thinking of it, that's the, that's the goal ultimately of a copilot in any situation, right?

This kind of pair programming or something. Mm-hmm. You got somebody looking over your shoulder, looking at it, challenging thinking things differently, and ultimately elevating the, the, of what you're doing.

Which is, you know what, again, one of the, the rep from one of my quotes here, Q test copilot has helped elevate our expectations of test case quality. And that's, that's great as well, right? Whether that's, you know, just standard English, you know, or whatever language you're, you're writing in having well articulated and form formed sentences and, and structure.

So people, when you're communicating to, uh, a bug back or a test result back, how do I make sure I'm clearly communicating this? Um, but also again, that, that, how do you challenge and elevate your overall testing? So that's really our initial goal with this, with what's GA right now with Q test copilot, and we've got got some great results, right?

The, when we, when I look at the charts of manually for customers that have adopted this manually created test cases versus AI generated, and then you see those curves cross and you start to see that real, uh, value get created there as far as like leveraging that. Um, that's really, for me, the first stage of value creation. The next stage then comes, okay, that's, that's great.

I've got more test coverage. Does more test coverage need higher quality? Not always, right?

So how do I then look at my results, um, from the test case coverage and the higher quality signals? And now this comes back to that quality intelligence theme that I mentioned earlier. Yeah.

And so that's for us, the next big stage with Q test co-pilot is that, and what you actually mentioned earlier, what other models can be used and what are their techniques in addition to, um, things like rags to bring in more custom, you know, brand, um, to that, to the, the standards that different companies use. Say, this is how we structure test cases, or this is the terms that we use for these systems, as opposed to it being this generic, you know, foundational model. How can I pull in all the other data from Jira or from from qTest itself to then formulate and recommend so that it's kind of like you bring in, if you, if you're bringing a person into your organization and say, start working, they would use different terminology.

They would have different process. And, and that's kind of what the first generation of, of this copilot is, is it's super helpful, it's doing a lot of work for you, but it doesn't know your business as well. And this next generation is, okay, now let's really Sorry, the intelligence And bring the intelligence.

Yeah. So, you know, so Let me, I gotta ask you a hard question on this math, because unlike a lot of other fields, specifically QA was using, you want to call it machine learning ai, you know, versus generative ai. You guys use machine learning AI to basically develop test cases.

And that's something Tricentis has done for a long time. I I remember talking about this, right? And that was one of the, that was a big selling point, right?

Is that the QA engineer, developer, whoever's setting up your test cases, right, didn't have to manually define the test cases, you know, the, the, the ML ai we could call it that kinda set up the here's here's the test we think you should run. Right? And that probably was 95% of the coverage.

Yeah. But of What you're saying, go ahead. Sorry.

I'm sorry. No, no, you go. A lot of, a lot of the, a lot of what you're talking about is more like on the functional testing with vision ai.

I mean, we had expert systems, we had neural networks and all of that to help create functional tests from, you know, from images and from pictures and kind of walking through and creating locators that are based on more than, um, more than just the, the object identifiers we would traditionally use. And so that was really, you're right, like that first kind of generation of like, how can we really go faster here? But that's before this more natural language interface that a lot of the generative AI technology opens up for us, right?

And it never, it was good at maybe defining your test cases, but other than sort of wrote, if you pass 80%, let it go. If you pass 90%, let it go under, don't let it go. It didn't supply any real intelligence.

And I think that this second generation that you're referring to is where the gen i gen AI stuff gets interesting right? Now, not only am I doing a better job at defining my test cases, but based upon those tests, I'm going to give you insights more than pass fail more than pass. 'cause pass fail is it's black and white gen, you know, literally.

Um, but now we're gonna give you real insights and advice and, and taking that to the, you know, second and third level. And to me, that that's where stuff gets real exciting. Yeah.

It's, you know, a lot of times when I, when I talk about this, people say, oh, so that means you don't need QA anymore or this whole how does, and and I, I would actually argue very strongly against that when I'm, what, what it does though is it changes the nature of the job, right? And the number of people you have working on different things. And so, you know, instead of people having to spend so much time writing test cases and, and, and, and writing, uh, whether that's just the description or it's the actual functional test behind it, um, AI can do a lot of that heavy lifting so that now I can use more time for QA strategy, I can use more explor, do more exploratory testing.

I can, it really elevates that position, allows QA to be more strategic versus like, I have to spend 99% of my day just operating the tools and putting them data in and managing it. And now I don't have any time to, to kind of step back and look at it. And this is, this is what I've seen, again, in the more mature organizations that are, are leveraging kind of our latest, they are able to get those insights, and that's why, you know, elevating their overall test strategy.

And I still think there's a lot more, um, that we can do there that to help. But that, that to me is the goal of tools overall, right? If you go back to not having any test automation and you're doing it manually, right?

That's, that's a valid way to do it. But that speed and scale are hard to achieve in that. So then we have tooling and now, okay, great, I've got tooling.

I'm still having spent a lot of manual time care and feeding of my testing, uh, and I think a gen AI can really take a lot of that off. So now I can spend more time on the strategic aspects, the insights and, and, and, and, and keep up, right? Because the other aspect is with, with the GitHub co-pilots and the other, uh, AI gen for code generation.

Now code is gonna, you know, services products are gonna be delivered faster. That just means I have to keep up with my testing, right? So that's the worst case is that we have, we have to leverage those just to stay even.

We may not be able to get ahead of them, but I think we can. But, but that's the, you know, that is one scenario that could play out. Absolutely.

It's exciting times and, you know, we're geeks, I get it. And so we could talk about this and it gets us excited and, um, but our audiences too. So this is, this is very exciting stuff.

This is kind of game changing, especially with the q and A professionals out there. You're right, right? A lot of people say, oh, is this the end of q and a?

Great. I want to be, I want my job to be obsolete by the time I'm 35. Like that commercial used to say, remember, but no, that's not it at all.

It's taken this to a whole nother plane, right? Yes. Of, of being able to really help and, and move things along and go faster as well.

So, Mav, is this available now? Pie in the sky, when can we get our hands on this? It's, it's available now.

We've got customers using it in production, um, existing. Uh, or if you're in, if you're not a customer, that's fine, you can get a trial. com and go to qTest, um, product trial.

You can, and you'll sign into the SaaS in minutes and you'll start using it and, and experimenting with it. Um, so yeah, very, very low bar. Uh, so I would encourage you, even if, even if you don't have an active project and you just kinda wanna learn a little bit more, um, and ex and, and, and, and watch it, definitely recommend that you go check that out.

com. Love it. Hey Mav, thanks for coming on and getting us a little smarty here about QT a copilot.

It sounds like, you know what? I'm gonna have to keep learning new taglines for T Tricentis. I see the lay on that Bar higher.

Yeah, always. That's our goal here. Thank You, man.

If I don't see you have Merry Christmas, happy New Year to you and all of our Tricentis friends, we'll pick it up with DevOps Unbound in the new year. Keep doing what you're doing, man. We love it.

Thank you. Thank you. All right, Mav Turner, chief Product and Strategy officer from tricentis here on Techstrong tv, tech Strong tv.

We're gonna take a break. We'll be back in a moment.