Generative AI and Application Testing – David Brooks, Copado

This is Textron tv. Hey guys, thanks for the drone. We're here with David Brooks, who's senior vice President for Evangelism for Capto, and we're gonna be talking about the impact that AI is gonna be having on testing.

David, welcome to the show. Thanks a lot, Michael. Nice to be here.

Nice to be. I think it's fair to say that we probably don't do enough testing, but the testing that we do do probably isn't as efficient as it might be because, well, sometimes we're testing the wrong things at the wrong time. So what is your sense of what will be the impact that AI will have on testing for better or maybe even worse?

Um, I think on the better side overall, I'm an optimist, and I think it, it is gonna be for the, for the best. Um, I would say that what we've seen in, in our early research is that it is, uh, it's, it is very possible to create automated tests, uh, test scripts from, uh, intentions, from a description. So, um, I, I think you're right.

We don't do enough testing today. We, what we see in the industry is that, that people don't even do enough manual testing, and then when they do manual testing, they, the people that understand how to drive the product are typically this SMEs for the product who are not selenium coders. So, um, what, what you see is people that just don't have enough coverage, um, they may actually cover the happy path with an automated test, but they don't know how to do edge testing.

And so I, you know, if I had to guess, I, I would say, you know, worldwide software base coverage in the 20% range. And so I think the major impact of AI will be that it, it will remove the barriers to getting at least a test with a happy path. And what you'll see is, um, is a, a huge increase in the coverage of code in most enterprise companies.

Will that increase come because we're basically automating the testing of routine issues and discoveries and things that probably aren't that complicated, but I wonder if we're in danger of, well, if we rely too much on ai, we won't have humans around to test those edge cases that are actually the ones that probably bite us somewhere down the road because no one thought about it. And then it pops up later. I, I think what you're gonna see is that, um, there's a lot of time and effort the people that are putting tests in place are spending, um, the capacity they do have on the happy path, uh, and, and manual testing.

And so what I think is gonna happen is this, the, the manual testing, uh, we have something called exploratory testing that'll be available later this year, and that's gonna make it easier to take someone who is doing manual tests and just automate the, the capture of that, uh, effort into an, uh, an automated test that can run, uh, consistently after that. So we're trying to take the percentage of manual testing and then immediately convert that into, uh, into automated testing. Um, that's gonna happen when that happens.

The people, the, the, the human in the loop in AI will be available to identify where those corner cases are, where the edge cases are. And I think what you'll see is that as soon as AI can take care of the bulk of what's being done today, you'll see people be able to focus at a higher level of maturity on the edge cases and corner cases. And I, and, and, and trying to validate in the, in the beginning, uh, of course validate that the AI is producing a legitimate test.

Uh, we think that's gonna be important in, in the beginning, human in the loop is gonna be really critical to just make sure you're not, you're not testing the wrong thing, uh, on the happy path, and then to, to provide guidance going forward on the, on the edge cases. We've been talking about shifting testing left for a while now. And do you think AI will accelerate that shift?

We will more developers, or for that matter, citizen developers who will build in applications in something like a SaaS environment, be able to test on their own more reliably? And if so, what does the testing team then focus on? Um, absolutely.

Yes. And the reason I say that is, um, if, if you think about ai, generative ai, generative AI is capable of generating not only the code, but also the testing for the feature, but garbage in, garbage out. How, how would you cre, how would you use generative AI to automatically create a test?

You do that by making sure that the user story is very well written, that the, uh, acceptance criteria are documented. And so capto, for example, has a product on our DevOps exchange, um, uh, called, uh, AI companion. And AI companion actually is something that a product owner can use on the user story to enhance and enrich the user story.

And why that's important is because that becomes the basis that general, the, the input to the generative AI that's gonna create the test. So to come back to your question, if the user story is well written, as soon as the user story is complete, uh, you can click a big green button and the test will be generated, which enables the developer to actually run the test against the code when it's ready. Mm-hmm.

So it's gonna make shifting left a lot easier. There are multiple flavors of large language models these days, and some of them are more general purpose than others. Do you think we're gonna need more domain specific, large language models for testing that are optimized for this particular task versus a general purpose platform that seems to pull in stuff from all over the web of varying quality?

Um, I think that what you're gonna see is, well, first of all, the cost of generating the true large language model. If you look, I think Amazon just invested up to $4 billion in a, a large language model. And they did that because the, the amount of, of money that it takes to generate a large language model that covers the general purpose environment is huge.

The problem with that is, uh, and I think if you, if you look at the messaging from Mark Benioff at Salesforce, you'll see, you know, he says that ai, AI is data greedy. They want data. And why do they want data?

They want data to train their, their models. Enterprises do not want their data to be shared with this large language model. So what I think you're gonna see is a bifurcation where the large language models, maybe you see three or four very large ma you know, open AI and, um, and the others that are the basic language models.

And then you're gonna see a layered on, um, you're gonna see layers of domain specific expertise put on top. A great example is we've got a product, uh, called capto, G P t. Did we, we didn't create our own large language model.

We're using open AI under the covers. But what we did was to take all of our documentation for se uh, for KADO from our, uh, community, and we trained the language model layer on top of the large language model to understand KADO specifically. And what that did was it, it enabled our, uh, uh, support people to reduce support time to time, to first response by 60% because they could simply ask the AI the question and immediately the documentation that was relevant came up and the answers came up.

What you're gonna see is that same thing with a customer layering on top. So large enterprises are gonna take the basic large language model, then there's going to be a domain specific language model on top of that, maybe like testing. And then there's gonna be a model on top of that that represents their specific code and their way of doing, doing testing and doing a code development in their company that is completely private.

Right? And that's where the sensitivity is gonna come in. They don't want any of that data to make its way back into the large language model.

Well, and ultimately will be the impact of this improved testing. 'cause I think it's maybe a dirty little secret in the land of applications, but a lot of our apps that are deployed in production environments today have tons of vulnerabilities and lots of inefficient code that cost us more to run. So will the quality of applications improve in the months ahead?

Um, again, I'm an optimist, so I expect they will. I, I think we were also, we were asked the question not not too long ago about what's the, if you had to choose one metric for quality that, that you would get behind, it would be the, the number of bugs that actually made its way into production. We understand that you're gonna have bugs, but you need to catch these bugs before they get into production.

So on the one hand, what you would expect is that with increased testing, uh, and, and increased coverage, that you should see fewer bugs hitting production. The only thing I'm a little bit concerned about is that one of the benefits of AI is increased in efficient, increased capacity. So you can do more with the same number of people that you've got.

So if you reduce your, your bugs by, by 20%, but you increase the amount of code that you're writing by 20%, you know, you start getting the same number of bugs that reach production, but because you're, you're, 'cause you're pushing a lot more into production, um, you know, maybe it's not, it's not gonna, it is you're not gonna see a huge decrease in the number of bugs. But I mean, the optimist in me says that if people, if people do it right, then they will basically automate their testing and make sure that they are producing good code before they go fast. You know, we have a, um, a model of five layer model of DevOps maturity and, and one of the things that we debated internally was is the second level, is that automated testing or is it automation?

And we came to the conclusion that you better au you better automate the testing and make sure that you're delivering good code before you start doing it fast. What are the challenges to bringing AI to testing? Are there things that organizations should think through before they just, you know, start launching this stuff?

I think the privacy and security is, is important, but I think you, you hit on it earlier, there is, um, you know, there's a mindset to a tester that is, how do I break this thing? And a really good testing agent will, will understand the, their, they're devious and, and they, they, they think about these edge cases and they think in terms of how to break it. Um, I think that, that the, the happy path is gonna be covered very well.

Um, but the, the real key is gonna be how do I, how do I train, uh, if, if I think in terms of automated testing eliminates the testing team, then I think that's wrong. That's wrong minded. I think that if you think in terms of it enables the happy path to be generated very quickly so that the testing team can focus on the, on guiding the AI to create edge cases, then what you're gonna see is better test coverage, not eliminating the testers, but, but really increasing the quality of what's being caught.

I guess Do we need to think through the security of this process because, well, we are testing a lot of things that have a lot of intellectual property wrapped into it and there might be a target, a lot of talk these days about software supply chains. Do we need to think through how we're gonna secure the AI models that go into these software supply chains? Um, absolutely.

That goes back to our previous conversation that I think there's gonna be a layered approach where, um, where you have a trust layer, you have a trust barrier if you will, where all of the data that you're using to train these tools are gonna be kept inside the enterprise firewall and not shared with a large language model. There's already a very simple way of anonymizing, uh, information before it goes out. I think that's, uh, it, it is probably more in the, it's probably more sensitive in the code area.

I think people are thinking about that more, uh, with co-pilots. Um, and, but what they really, they really need to think about that when it comes to testing as well. Because, you know, the quality, the way that a test runs if it's, if it's well written, can expose a lot about the way that the code was developed as well.

So, you know, and, and I've been thinking I think more in terms of functional testing, but um, you know, the same can be said for, uh, code coverage and automated, uh, um, code coverage, uh, fun, um, unit testing as well, uh, that needs to be considered. And what you're gonna see is when the, when the AI builds the code and they build the unit tests, then, you know, it's, you know, who's the fox guarding the henhouse? I guess, I don't know what the right analogy is there, but I think, I think the human in the loop is gonna be, is gonna be really important in the, in the short term on that is make sure that, make sure that there's that, make sure that the architecture is set up so that you're not having your intellectual property leave your firewall, but you have all the benefits of the larger language model will provide for you.

Maybe is the right summary. I may therefore need an AI model to guard the AI models to go to the, so the fox isn't guarding the inn house, right? So I've got some help with another L L M to check the other LLMs, right?

Yeah. Maybe you train it with one of those devious testing people too, or actually know your chief security officer because if there's anybody that's paranoid, you know, the most paranoid person in your company, it's your, your ciso, uh, maybe they're gonna train an AI that is the CISO AI bot and it's looking for all of these things in, in the structure that you put together. We also have a lot of different testing processes.

Well, those start to converge in a land where I have AI to automate some things. And 'cause today one of the issues is, you know, I got everything from unit testing. They'll load balancing and performance and all kinds of motions.

So can that get streamlined? Um, streamlined is an interesting word. We've been proponents, um, in fact, I wrote an article not too long ago where, you know, people are saying you should shift left and, and yes, you should shift left.

And then people are saying you should shift right? And use these same tests in a monitoring capacity, in production test, in production. And so if you look at DevOps, it's plan, build, test, uh, deliver and monitor, uh, I said for a while that testing is not a discreet phase of DevOps.

That what you're seeing you should see. And, and when you think about streamlining, what I'm thinking is you embed testing from the very beginning and, and this is like writing the plan you use, you use AI to do things like look at your plan, look at the user story, look at the impact analysis and, and what the, what, what, what your change is going to do to the rest of the system before you ever write a piece of code. And then as you're writing code, use these tools to generate, um, unit tests to, to generate everything that you need.

You can even at the, at the early stages, do a release into a performance testing system. Of course, you know, capto deals with, um, with cloud systems, especially Salesforce. And the benefit of that is you don't really have to worry about the performance side of things or the penetration testing on the, on the, on the, uh, on the server side.

But instead of thinking of it as being streamlined, what I think you're gonna see is you're gonna see embedded testing from the very beginning of plan all the way through shifting right to monitoring. And it's, it's just gonna be embedded throughout the process and it's gonna be easier streamlined from the perspective. It's going to take less to get that coverage, um, uh, and gonna be more efficient and, and and more thorough than it was before.

All right folks, well you heard it here. The conclusion is that with ai there's gonna be no excuse for not testing 'cause it's already gonna be right there in the process. There won't be any friction or there'll be a lot less friction and hopefully we will get better quality applications at the end of the day.

David, thanks for being on the show. Yeah, thank you very much, Mike. All right.

And back to you guys in the studio.