AI-Driven Testing and the Governance Gap

Hey, everyone. Welcome back here to Techstrong TV. My next guest is Ole Lensmark.

I hope I pronounced it right. Ole is the CTO at a company called TestCube. Let's welcome him.

Hey, Ole, nice to have you on here. Hey, Alan, nice to be here. Great to be here.

I'm looking forward to talking to you, too. Yep. Ole, we're going to get into TestCube.

We're going to talk about AI-driven engineering and AI-driven testing and all that. But before we do any of that, let's talk a little bit about you. Sure.

Give our audience maybe a brief on how you got here. Of course. So I've been working, I guess, with engineering since the mid-'90s.

I very early got into APIs and testing. I was the creator of a tool called SoapUI, which was a very popular API testing tool, had millions of users. And we launched that in 2005-6, and that kind of then was absorbed into SmartBear, which is a software testing tool company.

Sure. There, I kind of worked with tooling, continued around APIs, and SmartBear acquired the Swagger technology, which is still- Yes ... popular.

So I was very involved in that. We spun out the Swagger specification itself into a workgroup under the Linux Foundation. I was the chairman of that for four years, so kind of the governance of the Swagger specification itself, which was then renamed to the OpenAPI specification.

For various reasons, the tooling stayed with SmartBear, both the open source and commercial tooling that was built around that. So I've really been in open source and testing for many years, I would say. So that's kind of my high-level engineering background.

I don't know if I'm-- And then kind of about five years ago, I'm just going to jump into how we ended up with TestCube. Obviously exposed to a lot of testing over the years and different types of testing, and as people started adopting more and more CI pipelines and tools to run tests, specifically, I was kind of annoyed or kind of felt that those tools weren't really made for testing, right? So CI tools are great for building and maybe for deploying, but they don't give a very test-centric view of what they do, right?

They're all about pipelines. They're not about tests, right? So if you have your Jenkins thing and you want to see what tests am I running, it's really hard to figure that out, right?

Unless you kind of push your test results to some external tool. And at the same time, people were adopting GitHub Actions, and there was all of this stuff happening, and Kubernetes was happening with cloud native, and we kind of saw that as maybe this is an opportunity to start decoupling a test execution specifically from CI and moving that into its own platform, right? That is really focused on running your tests in the best way possible, not coupling it tightly to CI pipelines, allowing you to trigger your tests not just from CI, but also manually.

Testers often want to say, "Hey, I just need to rerun these tests. I don't want to rerun the entire build. " And also the CI platforms themselves, the runtimes are not really built for test execution at scale.

So that was also something, if you want to massively parallelize your Playwright tests or your load tests or whatever, that's not something those tools are built for. They're built for building artifacts and compiling, and they're great at that, right? But there was a lot of lacking kind of with the QA and tester's mind or eye on those systems.

So that was really kind of the itch that we wanted to scratch when we started TestCube, right? To build a platform that's really focused on, that kind of offloads or delegates test execution and automation from your CI platform into its own thing, and then that has a bunch of things that are native more to the heart of testers in QA. So I don't know.

That's kind of the preamble. And then obviously, when we started down that route, and initially, just like the other projects I mentioned, TestCube was an open source project, still is an open source project. You can get it off GitHub and run the kind of core engine of it.

But over the years, we've added a lunch, a lot of features, and commercial tier to kind of put food on the table as you say, the open source space. And now at the age of where AI is really kind of taking a big stance in not just development or code, but also in testing in QA, that's also where we're kind of embarking on our AI journey in the product. Get it.

I get it. You know what's interesting, QA testing, probably more than coding, more than security, was kind of out ahead in terms of automating testing before this whole GenAI and now agentic AI thing burst onto the scene. They were doing automated testing, even with open source testing- Mm ...

Selenium and all that stuff. I remember Sauce Labs doing it. The smart-- Many companies.

That was the key, right? Where the big switchover, especially around when DevOps came about, was a lot of the QA people went from actually doing testing to just more setting what the testing coverage looked like. Mm-hmm.

And then the testing became automated. Mm. Now, of course, though, with AI, all bets are off, right?

Now the whole thing is potentially open to agentics ... looking, doing the testing, deciding what tests to test, reporting on the testing- Mm ... if, in fact, not even fixing what the tests- Mm ...

find, right? Mm. And so it's a whole different world.

I totally agree with you. It's so fascinating also to see people who are really trying to figure out what is the best way to augment QA with AI, right? So there's, to your point, should AI be running tests or should AI be writing tests, and how good is AI at selecting which tests to run?

What if it selects the wrong test, right? And the tail end of your tests at troubleshooting, root cause analysis, remediation. There's so many use cases, and we see this.

Obviously, if you're in the communities of QA, it's already out there, right? And you can do everything, but if you go to people, customers, or organizations who are just getting started here, they're still trying to figure out where do we start and what's actually the right place to start. Where does it add value, or is it just a nice thing to get an analysis from an LLM, but where does it actually speed up our process or make us deliver better quality code in the end?

Agreed. So, Ole, it's interesting. The interview I did before you was with a large security company, and they're working in the SOC now.

And for them, the dream was sort of an agent-augmented SOC, security operations center. I think we're past that dream when it comes to testing. We already have agents doing the testing.

I think it's a question now of governance- Mm ... when we talk about this. And in order to do governance, you've got to have visibility, observability, all these things, which leads to MCP servers.

At Testlio, is that where the focus is on the governance, or still on getting the agents to do the testing? So Testlio is, like I mentioned earlier, at its heart it's a test orchestration and execution platform. Yeah.

So, we take any test that you have, right? So your Selenium tests or your JMeter tests or your security, whatever test you have. We take those tests, we run those in a Kubernetes infrastructure, which is great at scaling and all of those kind of things.

So we're not asking you to rewrite your tests or anything, we just take what you've got. And in that context, the value that AI adds, the main at this point is twofold. One is really at the tail end of running your tests.

The troubleshooting, analysis, root cause analysis, remediation, those kind of things. So what we're really helping our customers with now is just getting those agents into place, building trust around those agents, helping them build the right prompts. What models do you use?

How do we do this in a cost-efficient way that doesn't cost us too many tokens if we automatically analyze all our failed tests with AI? There's just so many tactical things that happen. " Okay, get it.

But what's the expression? Rubber hits the road or whatever. There's just a lot of things that still need to be figured out.

So it's both about just getting those agents into place and understanding where to insert them into your pipelines, making sure they actually are reasonably deterministic, especially with QA, right? You don't want an agent to tell you things are working when they're not, or vice versa. And that's the whole prompt engineering thing is a big thing, of course, and that's something very tactical.

And then I think when people figure out, okay, we've got our agents, they're being triggered, they're doing reasonably what we want them to do, then you come to governance of the agents, and I think that's something our platform supports, but it's not really something we see our customers struggling with yet. I think that's still ahead of them. Yeah.

I don't know if that- Well- Does that align with your experience, or do you feel like... No. So let me caveat that.

It's all over the place. Mm. There are some organizations who I think are very forward embracing this, and that they're running as fast as they can, and that's kind of where they're up to.

There are a lot of organizations, though, who are a little bit more cautious, right? Mm. I don't know if they're a little more cautious or just something else, but they're not embracing it- Mm ...

for whatever reason, right? They're doing what they were doing, and is it fatal to their business at this point? Well, no, not necessarily.

Mm. Will it be a year or two from now, or even six months from now, the way things are going? Very possibly.

I know. Where I think it makes a bigger difference, Ole, is the life of the tester. Right?

Look at testers who are embracing this versus testers who not, and I don't really care what company they work for particularly. It makes a big difference. Mm-hmm.

I see it here in Textron, right? Not just with testers, but with my web engineers, my AV teams, my marketing people. Those who are leveraging this technology are accelerating.

Mm. And the other people aren't. And that gulf grows wider, right?

Mm. As a CEO, I will tell you, I'm not firing anyone because of AI. But when you start seeing the gulf of the productivity of the people using it versus the people not using it, well, then- Mm ...

it makes sense. You got to do what you got to do, right? It's so- No, I totally agree.

I think the The role of a tester, to your point of testers, QA, everyone in engineering, whatever you're doing, needs to embrace AI for what they do because AI can help with most everything you do, and obviously in a sensible way. But definitely, I think that, like things I said, QA is more moving to what prompts do we create? Where do we inject agents into the pipelines?

How can we help AI select the tests that we want to run? What metadata does it need to be able to do that? Or how can we successfully use AI to improve test coverage?

Is that even realistic, or is that even mandated? There comes downsides with that as well. So I think you just have to rethink your role as an engineer, if it's QA, dev, whatever, and when you suddenly have this extremely powerful tool at your hands and how that augments what you used to do.

Me being more of a developer, I would never go back to developing the way I did a year ago now that I have- No ... amazing tools and possibilities. " What an amazing place to be in, to suddenly have this tool at your disposal where you can suddenly do all these things that would've taken you weeks.

I know. Look, I think it goes, name a job, and I think it's very relevant to it. Ole, we're running low on time.

For people who want to find out more about Testcube, where do they go? How do they engage? Sure.

io. There's a free trial they can sign up to the product where they can experience the product, they can experience our AI features and all that kind of stuff. If they're more into open source, they can go to GitHub, and find us on GitHub, and there's a path there.

There's plenty of documentation on getting started with the core engine, which is the core to the commercial product. So you can start with the open source, and then when you feel like you need more, you need visualization and AI and stuff, you can expand into that commercial realm. io.

They can find me on LinkedIn and ask me questions. Anyway, happy to engage. Happy to have you on here.

Thank you so much. All right. Ole, thanks for coming on Techstrong TV.

Come back, keep us posted. Best of luck with Testcube. Thank you so much, John.

Take care. Thank you. We're going to take a break here on Techstrong TV.

We'll be back and we've got more coming, so stay tuned. We'll be right back.