COCC and SmartBear Advance AI Test Automation

Hey guys, thanks for the throw. We're here with Derek Larson, who's from COCC, and Todd McNeil, who joins us from SmartBear. And we're having a little chat about test automation because Derek's been up to some new and interesting things.

Gentlemen, welcome to the show. Thanks, Mike. Thanks.

Derek, you guys seem to have revisited your entire test automation workflow with a little help from SmartBear, and that's not a small undertaking these days, but it seems like there's a lot of folks who are interested and maybe thinking about this issue, especially in the age of AI. But what kind of pushed you down the path to rethink this workflow, and what exactly have you guys been up to? So I think it's top of mind for a lot of software development companies these days.

With AI accelerating the development of code at such an impressive pace, we're now moving down towards having to increase the pace of testing and getting automated tests together. And unfortunately, what comes along with that is difficulty organizing. When you're pumping out many tests and you have a lot of different key stakeholders inputting tests, organization becomes quite difficult.

So COCC recognized this a number of years ago, so we started looking at different test management platforms. We've gone through a couple different ones. Most recently, we migrated from SmartBear Zephyr to SmartBear QMetry.

And this wasn't a change due to anything being broken. It was more so getting us more aligned with where COCC wants to be here in the days of AI. QMetry brought in some more AI aspects, more API capabilities, and it's just really been overall a great product for COCC, managing all of our test cases, reporting, auditing, so on and so forth.

It seems to me, in talking to folks, that more folks are using AI to automate testing than they're actually using the coding to create something that winds up in their production environment, which hopefully they're testing more often. But what are you seeing here with AI and what's going on with testing? Because I feel like there's a lot more people like Derek out there revisiting these things.

Yeah, it's really about, I think, keeping up with the pace of development. Things like Claude certainly accelerate how quickly you can get features and bug fixes out, but you still have the challenge of knowing, is it working, and what does working mean? So, for a lot of organizations, that means relying even more on your requirements, relying on your test cases, but just kind of trying to keep up.

And so that's kind of where we see ourselves helping is keeping up with that new pace so that the quality team isn't the bottleneck. Mm-hmm. Derek, how hard is it to switch out these platforms?

" How complicated did all this wind up being, and how long did it take? So honestly, our migration from Zephyr to QMetry was quite seamless. We worked closely with SmartBear in the weeks leading up, had a migration planned, and we actually did this migration during our heaviest regression cycle of the year.

So it was around December. We had multiple applications under regression by multiple different teams. A lot of visibility into it.

There was concerns there. But you know what? All of these teams were working.

They signed out of Zephyr on Friday afternoon. Monday morning, they logged into QMetry, picked right up where they left off. It couldn't have gone more seamlessly from my perspective.

So, I would definitely encourage folks that have that concern, look into it. I think the more you work with companies like SmartBear, the more comfort you'll have with it and the better position you'll be in for what the future is bringing. Todd, I don't think it's much of a secret that usually the first thing that gets cut when we're under pressure for a deadline is testing.

But as we look at all this stuff today and we have more code to test than ever, are organizations testing more these days because it's simply just more automated, so in theory, I should be able to test more and maybe even spend more time crafting the tests? Or what is the ultimate impact of all this? I think one trend that we are seeing a lot is organizations have gotten licenses to Claude or another LLM, and they're trying to understand how they can leverage those tokens.

So a lot of tokens are going to go towards development, but the quality teams are looking to see how they can best leverage that. I think one of the advantages of SmartBear tools is that it's a workflow that the testers are already familiar with. You don't have to basically re-implement your workflow entirely with Claude, but you can use an external LLM to augment that.

So that could be using an MCP interface to help you generate test cases from requirements or using an external agent to help you build out automation based on your test cases in QMetry. So it allows you to leverage it while not basically running out of tokens or kind of blowing up your entire workflow and starting from scratch. How long do you see the benefits of this taking to manifest themselves?

Is it something that's instant, or does this take a little bit of time before I start seeing higher quality applications that maybe, I don't know, contain less vulnerabilities or are just easier to use? What's the ultimate benefit and impact? So it's going to be pretty instant, right?

A tool like Reflect, I see it as a force multiplier for test development, right? So where I work here at COCC, we have a team of five automation engineers, and we support upwards of about eight different applications. It's just not practical for them to keep up with that many developers and that many applications.

A tool like Reflect is something that business owners, QA personnel, product folks, they can all be in there creating automated tests that could be instantly integrated into pipelines. And the tests are there. They're ready.

They were crafted by the subject matter experts. It's a way more efficient process than having a product owner reach out to an automation team, explain what the business impact or business properties of this new feature are. They're just in there creating the tests themselves, and it's right there.

So instantly with Reflect, with very light training, I have basically expanded my automation team from four or five people to include all of the QA folks, all business analysts, all product analysts. So it's pretty impressive what it's done for us. And obviously in this day and age, with development going out so quick, that is huge.

Having that automated test instantly covered within the pipeline when code is checked in, it's a big one for us. Todd, when you look at all of this, where does testing fit now in the DevOps workflow? Because I think there was always this tendency to like, we're going to build a lot of code, we'll run it through the pipeline, and we'll test it at the back end.

Is that changing? Is it shifting further left? And my next question, of course, is are people or developers testing their own code using the same AI agent that they used to create the code, which is kind of asking the AI to grade its own homework, right?

Mm-hmm. Yeah, there certainly has been a shift to the left, I would say, starting from when moved to cloud, and you saw a lot of adoption of CI/CD tools like Jenkins and the like to try to get closer to deployment from the time that you're writing or designing the code. I think that's just accelerating even more now.

I think one of the things that's interesting is, yes, there is an advantage, even though there is a challenge with AI to hallucinate or garbage in, garbage out if you're not working with the right data. The benefit of the AI is you now can use a lot of data that you already have on hand to help inform how you build the tests and how you build your automation from that. So I think the code itself is a really good kind of source of truth for how things work.

It may not exactly define if it's working correctly, but it defines how things work, and so that's a good input. But things like requirements, test cases, past test results, things of that nature can all be used as input to the LLM, and that's pretty powerful. That's a way to build your test cases before your functionality exists.

So that's kind of shifting left, but also very quickly turn those test cases into automation once you have your working application and you're ready to automate it. What reaction do you get from the developers? Because sometimes they're a little, shall we say, persnickety.

And so when you look at this stuff, and you present that to them, what's the feedback you get? The feedback we're getting is there's a lot of excitement right now around it. There definitely was concerns.

A couple of points Todd just made there, where if you are purely basing tests off code, you could be automating a test to a bug, right? But that's where that additional context comes in. Requirements, acceptance criteria, QA sign-off of the manual test cases that your automated test cases are going to be based on.

So once there was an understanding around that, the developers were super excited. Right now, the current process, there's a gap in between when new features are checked in and when automated tests are available. So there's a very good opportunity that in one sprint, development checks new features in, and it's not till the next sprint that QA gets to it, and they're like, "Hey, we're seeing a bug here.

" No manager out there likes that word. It's a dirty word, rework. So they're quite excited to have that instant feedback so that they don't have to break their focus and they can really just stay with an effort until it's concluded.

Todd, you can't walk down any DevOps aisle these days without somebody talking about some advanced AI model and what it means for security. But is that going to change the way we think about testing as well? Because it would seem that beyond unit testing, we also got to test for vulnerabilities and all kinds of things, and the testing workflow is going to maybe get a little more expanded and a little more near real time.

Is that fair? Yeah, I think there's a lot of implications to security. Certainly with the latest models and the concern around their ability to find security vulnerabilities, that's an interesting aspect to it.

I think in the context of quality and testing, a few things that spring to mind are something that Derek mentioned earlier, which is that a lot of organizations, especially folks with complex applications or in regulated industries, it's really the business owners that know what quality is defined as. They know what actually the requirements of the application are, if it's medical codes or things around finance legislation or compliance things. So giving them a tool that they can go directly from the requirement to a test case to automation rather than having to work with a separate team is really powerful.

And then I think also in terms of security, LLMs, while a lot of conversation is around Claude and OpenAI, there's a lot of great innovation in open weight models, and that has a really good security story in the sense that you can run those within your own infrastructure, and not have that being shared with a third party. I know that's a big concern for a lot of companies. Derek, when you think all of this and you look at it and you say, "Hey, this is great.

" What did the senior leaders say? Because so much noise we hear in the world talks about how, well, we won't need this one or that one anymore in the age of AI, and I'm sure you've heard people say, "Well, this is great. We won't need a dedicated testing team.

" But, what's the reality? So the reality is that the testing teams are still very much needed. Their focus is just shifting.

They're going to shift their focus from the lower level mouse clicking and following test plans and gathering test plans manually, which take a lot of time, and it's going to shift more towards release readiness. What is our risk? What has been covered?

What hasn't been covered? So what this AI is going to allow us to do is use that human thinking where we truly need human thinking. AI doesn't understand release readiness.

That needs to be made by a human. Governances around release readiness, that needs to be handled by a human. So it's not removing testing teams, it is just shifting their focus to be more properly aligned with what they should be focused on within these development life cycles.

" Yeah, I think the trend that I see is that we have Claude tokens or we have a tool. How do we back into what we can do with it? It's a tricky way to go about generating ROI for your organization because it's almost like a solution in search of a problem.

I think what can really help is when you start with things that you know your company can rely on, so things that are key to your workflow now that has proven to work well. So again, I think test case management tools like QMetry is one where they've been around for decades, and they are great repositories of information for AI, and you can rely on something like QMetry to be the basis for your AI augmentation. You don't need to re-implement that.

And so what we're really trying to strive to do at SmartBear is have these sources of truth and these engines like Reflect, an automation engine that can really integrate seamlessly with them and are really flexible depending on if you want to use an external agent or your agent that runs securely in your own infrastructure or use the agents that are already within our apps. Derek, what do you know now that you wish you knew six months ago? Just how far this AI technology would go.

It's moving so quickly that someone in my position, the best you could do is point towards the technology where it is today and know it might change tomorrow. I think COCC as a whole, we've done really well keeping pace with some of the innovation in the industry, making sure we are up to date with our test management utility, and we were ready for this Reflect integration. We kind of already had our, I'll say, problem themes that we want to focus on in the future.

We had those in mind, those being visibility, scale, and governance. We're a fintech. Governance is our life.

So having those in mind, making sure our tool set was well-aligned, making sure our vendor was well-aligned with our visions, and ensuring that their visions were close to ours so that we could keep pace with them. That was the huge ones for us. I don't think I would've changed anything in the last six months if I knew anything different.

I probably would've moved quicker, because this stuff is all very exciting. But yeah, no big changes in the last six months other than we got to stay on top of this stuff and see where this current takes us. Todd?

Looking in your crystal ball, where do you think we're going to be when it comes to test automation a year from now? Well, I think everything's going to be, all boats will rise from the improvements of the LLMs. So just given the amount of infrastructure investment and R&D investment, I think you can expect them to continue to be faster, more accurate.

Cheaper, I'm not so sure, but hopefully cheaper. And then from there, I think vendors will continue to learn about the use cases that are most best cases uses for AI, and we're learning from our customers. So we see how our customers are using AI and where it's adding the most value, and we can provide that to our customers in a way that's secure, and a way that is a workflow that our users, our testers, and the business analysts of the tools is intuitive.

So I think you should expect to see that we'll continue to, along with just the industry in general, we'll continue to evolve pretty rapidly. And hopefully, what that results in is for users of these tools, allowing them to keep pace with this kind of new age where things are moving really quickly for them as well. All right.

Folks, you heard it here. AI is here. It's a reality, but it's, like most things, a little uneven in terms of adoption.

But one of the things that I have seen is that where AI succeeds the most is somebody finds something that nobody really likes doing, except for a few passionate few, and they apply AI to it. So maybe testing is the place to start. Gentlemen, thanks for being on the show.

Derek, Todd, good to see you. Thanks, Mike. Thanks.

All right. And back to you guys in the studio.