Fitz Nowlan of SmartBear on Why AI Is Forcing a Rethink of Software Testing and QA

Hey guys, thanks for the throw. We're here with Fitz Nowlan, who's vice president of AI and Architecture for SmartBear, and we're having a chat about well, integrity, at least software integrity, because well, it's becoming a bigger issue in the age of AI fits. Welcome the show.

Thanks so much for having me. Great to be here. Mike.

What is going on these days? 'cause I think we are all super focused on coding faster than ever, but I'm not quite clear that that has, uh, improved the quality of the applications we're building. So what do we need to be thinking about here as we kind of go forward and what are we overlooking?

Yeah, it's a great question. Uh, obviously AI is transformative technology, and I think a lot of the attention has been paid to the software development side of the house and how it's changing, uh, the speed at which we can develop new features and we can prototype new applications. But ultimately what that means then is if you're producing product at a faster rate, you need to ensure quality at that same rate, uh, or else quality will just be a bottleneck for getting all those great new features you're building out into the market.

So, um, that's really where we're focused a lot at SmartBear on the integrity of your applications, the correctness of your applications in this new world of software, uh, development driven by ai, Who's taken responsibility for that. And I asked the question because a lot of times developers are like, well, I checked my code in and that was it for me, and away I go on to the next thing. So somebody on the other side of that check-in process is probably looking at all this stuff and there's more of this code than ever for them to review.

So how do they keep pace? Yeah, it's A great question. I think what you'll see is, and kind of where we're placing our bets or one of our bets is on, um, AI powered quality assurance or bringing the qa, um, task, uh, into the, the velocity of the software development world in with ai.

And so if you have AI powering your software development, you need AI powering your qa. And so, um, you know, we have AI infused to different points of the quality assurance, uh, life cycle. So whether that's managing your test cases or helping you execute test cases, or even helping you author brand new test cases using ai, we have, uh, you know, solutions in that space.

And, and that's really where we're focusing, because again, all the money right now really seems to be going into software development. If you look at the biggest names in ai, they're all focusing on coding assistance and, and, um, you know, coding agents, uh, you know, we're, we're kind of trying to meet them in the future with, uh, QA powered by AI as well. So to your question about who owns that, um, I think it's QA teams.

I think it's, um, a lot of it actually will go back to product management teams, the folks who define the functional specifications of software and, um, you know, the people who decide ultimately what they want to build. So whether that's developers at small organizations or PM folks at larger organizations, um, we think that functional specification really acts like the source of truth, both for what you're building, but also for what you're testing that functional spec becomes your set of test cases in a sense. Will those folks there maybe just call up a DevOps engineer to fix the problem then?

'cause it seems like the pace at which we're moving, maybe they don't want to go back to the original developer, and so perhaps they're gonna have somebody else kinda review that code and, um, put in the fix as it were. I think it, I think you'll see more of, um, almost like a blending of roles. So it, it could be a DevOps role who's kind of straddling, you know, infrastructure and, and source code.

Potentially. It could be a product manager who's straddling, uh, functional specifications and quality assurance. Um, it could be a developer at smaller organizations who's, who's wearing all the hats, right?

Certain developers at, at startups, uh, will often be doing the infrastructure of the DevOps role, but they'll be doing quality assurance as well. They might even be specking certain behaviors out like a PM might do. So it, it's going to allow AI will allow a blending of roles across the different, um, points in the software development lifecycle, I think.

Mm-hmm. Do we need a different AI model to review the code that was created using another AI model? I think there's a tendency where people are kinda using the same AI model to kind of test things and it's like, well, I don't think that AI model is gonna catch the original error.

Uh, 'cause it's kind of like asking the fox to guard the henhouse, right? That's right, that's right. Who's watching the watchmen?

Um, so totally, I think it's a good question. The, the, it's sort of a subtle one. I don't think you necessarily need a different model, but you need a different structure and framework and composition of, um, your task for QA versus for software development, you could use the same model, but, um, that's at the lowest level above the level of the model.

You have all the structure of the questions you're asking and the task you're trying to achieve. So in the case of software development, you have things like planning, um, implementation, execution, unit testing, and then post merge or post deployment to your staging environment. On the QA side, you might have things like definitions of correctness, um, visual accuracy or visual correctness, and then functional or operational correctness.

And then even a third check kind of a little lower level. And this is maybe where we bring in DevOps. Um, you know, does the database integrity, uh, look good?

Uh, is the, the volume of traffic going through our system, is that normal or is that abnormal? Is the latency correct? Things like that, you know, you kinda have different areas that you focus on.

You could use the same model. You could use Gemini or chat PT or Anthropics Claw, whatever you want. Um, but it's the structure of those prompts.

It's the focus of the prompt. It's the way you use the model that is different for those different purposes. So I think you're okay to tech test your AI with AI as long as you, you have a fundamentally different construction of the problem.

Mm-hmm. Um, as you kind of think this through for a minute, are you concerned that maybe we're gonna have a spate of applications showing up in production environments where the quality is even lower than it might have been historically, and we're gonna have these, um, some sort of situation where the number of incidents are gonna just dramatically increase? Yes.

There, there's absolutely that possibility. There's a bunch of different ways to take that one angle, um, which I'm, I'm not, uh, too deeply entrenched in. It's like the cybersecurity angle there.

So I'll kind of mention that That's absolutely a, a consideration. I think I saw one of your, one of your previous, uh, videos, but touched on that a little bit. Um, an area that's pretty interesting for us though, uh, is, is what you were saying, you kind of hinted at it, you might have apps that are lower quality, but who's consuming your app in the new world?

Is it still those human users that it's been for the last 70 years of software? Or is it AI agents? And if it's AI agents, should we be building apps a little bit differently and testing them a little bit differently as well?

And here's why. So in the past, right, if you had an app that that, um, really made your user feel great, uh, you can almost imagine like commercials, right? They like bring tears to the eyes of, of the viewer.

It's really compelling and captures their attention the, that application developers should be thinking about. And it all relates to who their, who their ideal customers, who their ICP is. Um, and so I think that that may change how we test and, and, um, you know, ultimately how we structure our applications in the future.

Mm-hmm. To your point about that, um, are there tells in the AI code that we should be looking for as it pertains to quality? And I asked the question because sometimes, uh, the AI generates code that I would say is a little overly verbose and then we wind up with a situation where maybe it costs more to run that software because it's just consuming more memory.

But are there, are there things that people should be looking for specifically that are kinda like, you know, top of mind issues that should just be, you know, things you address right off the bat? I, I think, you know, functional integrity is first and foremost, the application must always still do what it claims to do. And, and if it doesn't do that, you will lose your users, you'll lose your a ai agentic users as well.

It has to function correctly. So that's always the first level of check, I think, um, to your point on, on like, uh, the veracity of the code that's generated, or, um, a lot of times what you'll see is, is the code related to veracity, it'll generate the same, um, functional code multiple times for different parts of the application. It won't effectively refactor that code, you know, to use kind of a, um, a single de-duplicated implementation.

Um, the flip side of that argument is if it costs you less money to develop your application because AI generated it, then maybe you don't mind spending a little more to run the application, uh, because you were already saving so much money 'cause you didn't have to pay the human developer. Um, that, that would be, it's kind of, I guess, a different debate. Different debate there if you're willing to pay, you know, go down that bridge.

Um, but uh, to your question on kind of like what to look at, I, for me, I always just come back to functional integrity. It must do what it says it will do. If you're solving a big enough pain point for your customer, they'll overlook the, the visual wart or, um, you know, the, uh, not aesthetically pleasing UI you have if it's solving a functional problem for them.

Um, so that's always, that's always step one. And then when you get to step two, which is like, well, how easy is it to consume? How performant is it?

How enjoyable is it to use? That's where I think you kinda have to go back to this question of, well, who's using my app today? Who's using my app two years from now?

Um, and what's the best experience for those different users? Are you at all concerned that maybe we'll just get lazy when it comes to application development and, you know, we'll just let the AI generate some things and we won't think through the process and maybe we won't even know exactly how the thing works? Uh, definitely it's, it's possible, it's a concern.

Um, you know, it kind of depends, I guess, how futuristic you are. Uh, and you know how much you buy into the narrative that we hear. Um, I think like a lot of things, it'll be gradual over time.

I think there will be certain applications or certain experiences that become commoditized and, and they kind of are known to be consumed by the AI agent for this or that business or this or that consumer. Um, and then there will be other applications which are, uh, they're still always gonna be consumed by users and, um, you know, they'll always be, uh, the quality there will always be important. I think quality, again, has a lot of different meanings, a lot of different connotations.

One of them is functional correctness. And so that's always going to matter that you can't, you don't have software if it doesn't do what it says it's gonna do. Um, but beyond that, you know, to the aesthetics, I think again, it's gonna be a gradual thing.

It'll be stuff across the spectrum where you see some usage, uh, is AI driven and other usage is human driven. And, and I think, you know, we'll define that happy medium. Mm-hmm.

There are also a lot of folks who are talking about the rise of vibe coding, and we will finally have these citizen developers generating all kinds of interesting things and the definition of what a developer might change. But does that create some quality challenges? 'cause the folks who are building that software don't necessarily have, uh, shall we say, an appreciation for the fundamentals.

Absolutely. So for, for in with the vibe coding question, really what you're seeing there, I think is that, um, the source of truth for an application is really no longer the source code in the 100% vibe coded application. It is more the prompts and the, um, the functional specification, if you will, of the application that generate that's used to generate the code.

So you will need some expression in that same level of abstraction, uh, on the quality side, on the functional side, uh, that can be used by QA agents to test the application. Uh, because you, you won't necessarily be able to use, uh, QA to test the source code that's generated in vibe coding because it, it could be so massive and, and it could change, it could be lots of churn, right? Every new session of vibe coding my application, the, the LLM might rewrite half of my application.

And so the actual source code becomes a little less important. And what becomes important, the source of truth are the spec, the, you know, the specifications or the context that's given to the, to the LLM to generate the app. Um, so what's your best advice to folks, therefore, as you kind of look at all this?

'cause I think we've been struggling with this quality issue for a long time, and there's more, people seem to be concerned about speed than quality, and that's always been something of a mix match. So how do we get more people to care about quality? Um, I mean the, the, the proof is in the pudding.

I think you have to show people that quality makes money, you know, quality applications make money. And I think you have to, I think, I think it's, it's reasonable and it's a reasonably compelling argument to say, look, if you're producing software at 10 times the speed today that you were before, you need to test your software at 10 times the speed you were before. And so you can't do that with a human in the loop because you're not doing it with a human in the loop on the software creation side.

So I think it's compelling to say you need AI in qa and, um, you know, from there it's, okay, well what's the source of truth that's being fed to that ai? Well, it's very likely the functional specs, the definitions of my applications that I want built those requirements they serve as the guardrails during qa. So, um, you know, the, the, the argument then becomes, use good tools for documenting your applications, use, use good tools, AI powered tools for, uh, testing your applications.

And, um, you know, with that you can, you can maintain the same velocity on your testing side as you have on the development side. Hmm. What is that one thing you see people doing over and over again that just makes you shake your head a little bit and go, you know, folks, I wish we would think this through a little bit better.

Uh, well, so this, this goes back a little bit, um, but it's still present today. So I, I started a company called Reflect, which, uh, was an automated end-to-end testing platform, and it kind of competed with a lot of the code-based testing, uh, approaches out there like playwright, selenium, Cyprus. And, uh, I thought then, and I, I think it only more strongly now that the exclusively human authored code-based testing of applications is, um, is rapidly going to be overtaken by, or if it hasn't already been overtaken by, uh, AI powered development.

Uh, and, and what I mean by that is if AI is producing your, the source code to your application at a much faster rate, you can't have humans offering playwright tests to, to test that software. You have to have AI in the testing, uh, lifecycle and in the qa, uh, framework so that you can keep pace. So, but I, I, you still see, you still see folks today, they're using AI to generate playwright tests, which is a little better than humans generating the playwright tests.

Um, but, uh, but you need, you need AI in, in the, um, in the QA process because you need that, uh, human-like intelligence to decide when a bug is a bug. You know, the, the famous problem with code-based testing is that, um, it's brittle to, uh, semantic, um, to, to different implementations with the same semantic. So if you move the location of a button or you change the color of a button in your application, some minor, you know, uh, literal change that it didn't change the, um, intent of the application that those tests are gonna fail.

So that, that's a, a behavior that I still see that I, that makes me shake my head and, and say, you know, you gotta graduate to, to AI in the QA loop. Mm-hmm. You know, we always talk about testing and quality assessment in the context of the people who built the software, but I wonder in the age of AI, if we won't see more organizations using those same tools that are AI driven to test the quality of the applications they're being asked to deploy and use.

And the whole notion of quality assessment is gonna change because, well, the end customers are gonna get a lot savvier about doing those assessments without regard, without any aid from a developer per se, and then they'll give their feedback accordingly. Yeah. I guess, could you go one level deeper for me?

Uh, what's, are you thinking on like the, the end user consumption side there, or Indeed, I'm, I'm thinking there'll be some enterprising CIO somewhere who will take these tools and apply that to some custom application that they've bought or some even commercial application just to see if this thing will stand up to the test of time and it'll become one of their buying criteria motions. Yeah. Got it.

So, so you, you're kind of, uh, positing, is there a, a future where enterprises use AI agents to validate or test the software that they're purchasing from third party vendors? Sure. Why not?

Yeah. Yeah, absolutely. And this gets back to one of, like a central theme of our position is like in the future, two years from now, will it be humans using your app or will it be AI agents?

And, and if it's AI agents, it's probably different things that matter to the AI agents than it mattered to the human. Uh, and so for that CIO example, they probably care a lot about governance, compliance, um, correctness, uh, things like that, you know, integrity that might be a little different than the end user who's trying to use the software just to do their job, kind of not as part of the bigger picture, but sort of a, a smaller picture. Uh, I think it's really interesting to, to wonder if, getting back to this notion of like if, uh, selling software was about, it was about visibility, getting in front of your customer, but then also telling your great story kind of almost even beyond, uh, accuracy or functional corrections was telling a great story.

That's what got that sale. And then, you know, obviously functional correctness keeps the sale in the AI agentic future. Do people and businesses give their AI agents personas just like the, the former buyer, the human buyers of that software had before them?

And, uh, you know, do, do software vendors then need to think, not just is it AI or human, but if it's ai, is it gonna behave and, and want things like the previous human user wanted? Or is it a different thing that, that they want? Um, you know, it's a, it's a really crazy notion to think about it.

You can kind of go in all different directions, but, um, it, it's, it's stuff that's possible I think. All right, folks, we're at the beginning of this adventure, but I would say one thing, it may feel really good to write a piece of code and be done, but it all feels really bad when the minute somebody calls up complaining about that very same piece of code. Hey, Fitz, thanks for being on the show.

Thanks so much for having me, Mike. Have a great day. All right.

And back to you guys in the studio.