TestRail Director Hélder Ferreira on How AI Will Boost Software Quality Through Continuous Testing

Thanks for the throw. We're here with Hilda Ferrera, who's director of product Management for TestRail, and they have a new report talking about software quality issues in the age of ai among other things. But we're gonna go and dive in and say, what are the challenges?

Because, well, it is a whole new way of thinking about writing software held there. Welcome, Michelle. Yeah, thank you so much for having me, Mike.

Alright. Walk us through some of the high points of this report, if you would, and is there anything in here that kind surprised you, especially? Well, surprised?

No, but one of the main, um, challenges that we have with this report, we, we try to expand the scope of the, the report, uh, trying to reach as many possible, um, actual users of the app so that we can get very insightful feedback so that we then can act upon it. Um, one of the things that is v very interesting is that everyone is talking about automation, how to increase automation. That automation is the way forward, but in the reality, the implementation of automation is still lacking and we are not near the numbers that everyone was trying to, to reach.

Right. So that came as, um, a surprise. Why is this happening and why are we not, uh, our companies are not managing to have the levels of automation they would like to have.

We of course, have been talking about DevOps and ruthless automation for years. So what's the challenge? What is the issue?

And 'cause it, to your point, it does feel like there is still a lot of manual bottlenecks. Exactly. So, um, in a nutshell, the majority of our, uh, responses indicate that it's not as straightforward as, as the thought, right?

In the, in the beginning. So it's not the actual automation itself, although we do have a lack of resources, experts in the area for testing, but it's also the way that, uh, it's implemented. So you implement the automation one time, and we think that it's done deal.

We have the automation, hooray, but it's actually not like that. Again, software development, we understand that things break, right? So these automations sometimes are not re resilient as they should be to make sure that they keep working throughout the new versions, iterations of the products.

And here is where we see a lot of issues on the adoption of automation. Not only that, but also taking into consideration the so many tools that we have on the market to automate point A or point B or point C, integrating all these tools to work combined, uh, in a combined way in the flow in the workflow. It's not easy and then enters the lack of experts in the area.

It's very difficult for the companies to be able to add the resources they need to not only create these automations, but also to sustain and keep these automations working, uh, uh, across the years. And to your point, a lot of the automations are brittle, but will this get any better in the age of AI or will it get worse when we just add more tools into the equation? The, the purpose of AI is actually to help and support in this.

So when we are talking, for example, on automations or, um, tests that are breaking the ai, actually, one of the benefits I see here is the self-healing. The ability that the AI will have to help the QAs help the testers to ensure that the process are still going. Identify, double check, have the trigger warnings about some issues that might be ha be happening.

And then the AI will step in, uh, try to fix on the fly to make sure that the process are, are still kept. This will allow the QAs to focus on other areas so that they can expand their testing scopes. Mm-hmm.

You know, there's a lot of folks who are saying, we're gonna build more software in the next few years than we have in the past decade, and the volume will increase. We're all just trying to figure out if the quality of that software is gonna get any better. So what's your assessment?

The, the velocity that we have with all this, and again, with the AI being able to generate code, we will have more and more software, more releases, faster time to market. How I see this is that the human intervention, the human, um, manual tests will still need to happen, but what we will be able to do to achieve these goals faster and keep up with this speed, is again, making sure that we have AI to support on those daily tasks that are repetitive, that are time consuming, so that we can shorten all of that extra load out of the QA teams and then the QA teams can focus on other areas like compliance, security, uh, uh, so that in performance, for example, so that we keep all this key, um, handle and actually we can even improve in the future. Mm-hmm.

Are we gonna test more? And I'm asking this question because, uh, one of the first things that usually gets cut when a project is running behind is testing and we tell ourselves we relate and we had to do something. But I also feel like a lot of instances we just don't like testing, we just don't wanna do it.

And the next question I have to you is, you know, do you envision a world where maybe we just give that whole testing function over to an AI agent who does it every time for us? And, um, we don't all have to be as deeply engaged. I don't see a future where the human part of it, so the manual test will be completely removed.

What I think we will do, and we will see the shift, is that the human intervention, the manual test will be done more as in exploratory testing, performance, security, compliance, all the manual processes that we have in place right now that will be handled by an ai, but it will leave time for the testers to focus on areas that they weren't doing right now. Right. So in a nutshell, manual and automated process, AI will all be working together in a hybrid approach where again, the human, uh, it's at the driver's seat, he's in control of the ai, the AI is only there to support so that we then can focus more into manual tests.

Mm-hmm. Where should this testing take place? I think a lot of times we get it into our minds that we're gonna build the application and then we'll test it.

But I feel like that's kind of a flawed approach, and maybe we should be thinking more about something that feels like continuous testing, but no one's quite sure where to begin that process. And where does it end The, actually the, the testing process? Uh, for me it's the shift left approach, right?

A a and again, we also see that with the, the answers to the survey, the, the goals, the, the companies are not achieving that goal yet. So what we should do is consider testing as part, uh, of the development process from the beginning, from the moment that we get the idea, we are talking with business, with our sales teams, and we get an idea when we start brainstorming in terms of product testing, uh, teams should be right there from the beginning. So if we do this from the beginning, test our across the entire development process.

So it's not only that, oh, this is the final step that we need to do. No one wants to do this, uh, phases because again, they will generate some bugs, it'll delay the time to market. So having this fully synchronized teams working together with the testing teams from the beginning, I think it'll remove that.

Where does this need to happen and when does this need to happen? It's part of the entire process. Mm-hmm.

I also think one of the traps we fall into is if it's broken, we'll just fix it in the next release and the next update. And that'll be coming along anytime now. And then, you know, we get sidetracked, derail, there's some other feature becomes more important and it just gets added to the technical debt.

Do you think though, that in this age the, the tolerance of the end customer for buggy software seems to be dropping and that's gonna force more people to focus on software quality? Exactly. Um, again, the competition, the speed that we have new features to be going live, um, we see that there's no bandwidth for buggy software, right?

So in, in a way, what we need to ensure is that we explore all the areas, not only like, oh, let's just do like the, the, the happy path test the, the feature from A to Z, then let's ship it. No, we need to start exploring our other areas, compliance, security, even with the use of ai. And more and more, um, uh, uh, apps now are deploying AI solutions.

So security is a real concern from our, these customers. We need to ensure that all the, the companies focus not only on having less bugs going to into production, but also expanding their testing scopes to making sure that they are, uh, uh, uh, uh, going into areas they weren't going before. Again, no one likes to have a bug in production, and the customers are starting to be very resistant to see live production bugs.

So we need to be extra careful and going to these areas that we were not going before. Mm-hmm. You mentioned security, and I've often wondered, it feels like today we have separate gates in the development process for quality and testing, and then for security, should that all just merge, They should be done through welding pro, the, the, the same process.

So merge, yes. We should be, uh, able to decide on the, the, the normal development workflow. Someone needs to go and say, my test plan, for example, will include security testing, will include performance testing, uh, compliance testing, all of that, um, uh, uh, grouped on that flow on that testing plan.

But yes, all the security concerns and even with ai, more and more security concerns are being, uh, raised. This should all be merged, ensuring that we have the, the quality, uh, uh, that we want when we are doing oral visits. So as you look at all this and you look at the report, what's your best advice to folks?

Or conversely, what's that one thing that makes you shake your head and go, folks, we need to be better than this. So, exactly. So there's, there's a lot that we can, uh, we can see from this feedback that we got from the report.

The first one I would say trying to have integration. Don't have tools that are like, you need to go outside of your daily work or of your daily tasks. And oh, okay.

And now I need to run security testing. So I need to go into this other app and do security testing. Try to choose tools that have fully integrated, secure performance, quality, uh, uh, compliance topics, all in the same workflow.

The less tools you have integrated within your development pipeline, the better, the more concise, uh, uh, uh, and workflow related processes you will have. The second point is looking to AI as a way to optimize all of these processes. Don't be afraid of having tools that have integrated AI fully baked in on your current workflow.

Don't select tools and have a lot of tool chain on your process that will require the end user, the tester to jump around between apps. And the third advice I would give is if the, the resources and the talent is very difficult to, to get on the current market, try to train your teams. And by leveraging AI within your development process, these teams might have more time available to explore other areas, exploration perfect, other scopes amazing, but also make sure that you train these, uh, these people, these, uh, resources ongoing into the next level, maybe training these, uh, teams to have security related, uh, uh, they are security, uh, experts or compliance experts.

This will make sure that the entire scope of testing and the quality overall of your tools is even better when you are launching. All right, folks. Well, you heard in here, like most issues, the longer you put it off, the bigger the problem it gets.

Right? So er thanks on the show. Absolutely.

Thank you so much, Mike. All right. And back to you guys in the studio.