ASTQ Summit 2023 – Arthur Hicken, Parasoft

This is techstrong tv. Well, the great pleasure of being joined by Arthur Higgin. Arthur is evangelist with Parasoft.

Welcome, Arthur. Hi. It's great to be here.

Um, good, good to be on talking with you. Always fun to talk to, uh, someone in an evangelist role because you like to talk about what's going on and what customers are doing, and that's all good stuff. Before we get to that, and we're, we're gonna talk about software testing and some interesting aspects of that.

Would you tell us a little bit about yourself and tell us a little bit about Paris? Yeah, so, uh, I, I'm the evangelist at Parasoft, which is, uh, it's a great ice icebreaker for any situation. It's an interesting title, but I've been at the company since let's say before the worldwide web and, and leave it at that.

Okay. Um, we make all kinds of automated software testing tools, and, and it's, it's been an interesting and fun journey. We, we are always looking for ways to take something that's tedious, boring, and free people up to, you know, do something clever and interesting and hard because software engineers like to do challenging things.

They don't like to do boring things. Imagine that. Mm-hmm.

I can relate to that too, being a software engineer way back and, and today for that matter. Yeah. You know, talking about that kind of longevity in the career in your career, Arthur, you know, testing has certainly changed a lot, just like software development with Agile and new kinds of architectures like Cloud Native, et cetera.

And, uh, one thing that I find fascinating, we did a study well while back where testing was kind of thought of as a back, back office thing, sort of, we might do it if we still have time, you know, at the end of the release. It always got squeezed today. It is vital.

I mean, and people know it, people know that we've gotta do a good job of testing and build that into the continuous process about how we create software. Are we, are we, I'm assuming we're in the similar neighborhoods on that same page on that? Yeah.

Yeah. You know, I, I hate to say it. I remember when, uh, an enormously huge application might have been two or 300,000 lines of code.

Right. Like, we thought that was, that was really something. And now, you know, a car with over a hundred million lines of code is, is kind of normal for modern high-end cars.

Mm-hmm. But yeah, the, you know, unit testing, did you have to do it right? Like, did it really matter?

And, and people are still following on that journey in their own organizations, but yeah, it's, it's just one of those things, static analysis, right. That was definitely, uh, you know, an interesting technique and now you just wouldn't think about living without it. Right.

And it, yeah. It's, uh, it's a much more mature discipline. You know, the, I remember one of the early automations that we made internally was a, a simple script to run our, our regression suite over and over.

And, and it had some little cues in it, like it would summarize the failures, and if it was over 80%, it would say, ship it. And if it was less than, you know, 80 and more than 70, it would be like, maybe you should check it. If it was less than 70, it would be, you know, you need to do more tests.

If it was less than 60, be like, go fire somebody. Right. That was like our first level of automation.

So what we need is a bash shell script telling us whether to ship it or not. Yeah. Yeah.

But that, but that is, you know, I mean, that is the goal with C I C D, right? Mm-hmm. That you have some kind of a, a definition of done and that the software can move itself along based on hopefully more than just like, past fail rate.

Right. But, uh, yeah, it was something we were trying to do. I mean, it was, it was ridiculous.

But it's, in, in today's world, you have all this complexity, so many lines of codes, so many dependencies. You, you know, you're, I, I always think about like rebuilding my sprinkler system with software and, and connecting it to like, to Noah, so I know when it's gonna rain and I can turn the sprinkler off and I can connect. I live in California.

We have regulations about when you can water, and so connecting it to that and the time of day and is the ground wet? And, and maybe I can tell if a pipe is versed, you know, like, like all the complexity in the third party, the dependencies that applications have now, everything used to be monolithic, right? Nothing's monolithic anymore.

When you hear there's all kinds of stats. D people have different numbers around how much of our application, how much of a code that we ship is actually open, Sur source, third party, other systems interfaces to online services, you know, upwards around your numbers tossed around 70, 80, 80%, which probably is true in the end when you really think about software stack that we're operating on both the application infrastructure of the app itself and Yeah. All the underlying technology.

I'm curious, what's your perspective on, you know, we talked about, so my, I remember many times, you know, when I was running product development organizations ship, it was, I would go talk to the QA team and tell 'em, tell me what's going on. And I would know by the kind of, and the severity of bugs that we had found, whether we're gonna him make it or not. I had no weeks before, like, yeah, we should be here by now or we're not there.

But that, those sort of gut feels, I'm sure probably still are part of the equation, but how do people make that, that decision now with quality and q QA and, and, uh, you know, development and, you know, platform engineering and cloud engineering, all the different, you know, pieces that go into shipping software. I, you know, I've, I did an interview years ago, and one of the guys is like, gut is all that matters. And if you have a small group and a well-defined project, like if you can hold it all in your head, it probably works as well as anything.

But again, we have, you know, distributed teams and distributed technologies and, and you're really DevOps. You're taking into account the deployment platform and the third party dependencies and, and the, the quality of the open source you're using, uh, your gut can't be it. And by the way, if, if this is a machine and it's kicking off a build and then a set of tests and standing up virtual machines and, and setting up services in the cloud, automatically deploying, how does your gut fit that?

I can't even keep pace with it, right? Yeah, yeah. They can't be a, yeah.

Check how Arthur's feeling press here to continue. So A lot of log, a lot of logs between, I, I do think people have to do it. And, and when you've had, you know, been relying on your gut, you have to step back and say, what was it I was noticing?

What was I paying attention to? And I do think that understanding what were the criteria we were basing it on? Was it, was it the volume of the test?

Was it the failure rate of the test? Was it the severity of the tests that were failed? Was it the, you know, in a, in a simplistic traditional model, we looked at if you had good beta testers, was the flow of noise from them shrinking?

You know, the, what's the variability in the, in the failure rate day on day, there's lots and lots of numbers you can look at. And I, I don't believe that there's a single number that's right for everybody. I do think it's really important today to have some idea of what you're measuring, why you're measuring it, a way to measure it so that you can tell if things are getting better or worse.

And if, if your gut says things are better and the metric says they're worse, fix the metric. And if you're, if your gut says things are good and, and things aren't good, fix the metric, right? Mm-hmm.

But get one that, that works for you, right. That, that matches your gut. Right.

It, it seems like we, we've elevated the role of testing in our delivery process now, partially because, you know, developers more involved, oftentimes not always in, in creating unit tests and automating testing with, with QA teams, but it's happening so much earlier in the process, and it happens continuously, you know, as codes checked in and environments are set up, um, it isn't the thing that gets squeezed. Documentation in QA gets squeezed at the very last, you know, hours of the, oh yeah, we did some before we ship code. Um, fortunately, we'll, I hope those days are gone, but it seems like the discipline of testing of QA is, uh, it, it's got more engineering to it.

It's got more visibility to it. It's a profession. Uh, not that it wasn't before, but it's even a, a, a more, a greater profession than it may have been a decade or two ago.

Is your sense the same thing? Yeah. Yeah.

People, again, people were relying on their gut and doing a pretty good job, but they, you know, they formalized and learned best practices, and we share those best practices and, and we, you know, rationalize the techniques that we've used over the years and decide which ones are actually having in effect. You, you can do a lot of testing and not end up with much of a difference if you're not doing the right thing. You know, the, the common answer I used to get, if I'd ask somebody, are you doing you to test?

Yeah. Well, how much are you doing? And I'd get a quantitative number, we've got 10,000 tests, we've got 20,000 tests.

I'm like, I, I don't know what that means. I literal, you know, like people testing without coverage. Uh, so I don't, did you do a good job?

Did you do a bad job? Like, I, I can't tell that. So those kinds of things have definitely changed, right.

We've, we've learned to quantify what we're doing to measure what we're doing and, and to share that knowledge, right? It it's not just that you'll hire, you know, an old guy with gray hair and have him tell you how it works, right? Yeah, yeah.

Exactly right. You can trust me. You can trust me.

I know what I'm doing. Every software release, it gets a little grayer. Yeah.

Yeah. Well, let's, let's jump ahead. You know, the hot topic right now is of course, ai, ai, ml, generative ai, of course, G P T.

I'm curious, you know, every, every tech company is either has incorporated AI in part as part of their product and or, or is in the process of doing that. And of course, I think as technical people, we're always skeptical, right? Is it, you know, is it a case statement or is it really a machine learning algorithm, and what is it doing?

Or do I need to really know that? I mean, that's, that's probably the better question is I don't have to be an AI ops person to run your testing product, but I do want some of the benefits you can gain from ai. Yeah.

Yeah. And that's, you know, that's, that's really how I like to roll and how we like to roll with, with ai. There's a, there's a lot of startups that are playing with some really cool tech right now, like abs and everywhere in every area, right?

This crazy, crazy cool stuff. But some of 'em, you have to really understand ai and, and even I think about using like, you know, generative ai, you can ask basic questions and get amazingly good answers, but sometimes to get a really right answer that a human would've done, you've gotta go back and understand how to ask the question. Mm-hmm.

How to prompt the thing. And, and I feel like I want my AI to be behind the scenes. I, I, yes, it's telling me what to do, but I don't want it to feel like it's telling me what to do.

I want to feel like it's just making my life easier. It's doing, it should be my servant. I don't wanna be its servant.

Right. Or of Assisted to you, you know, something That's helping. Yeah.

AI augmentation, taking over tedious tasks and, and, you know, watching and viewing and monitoring. So if I stray from best practices, it can gimme a tap on the shoulder and say, Hey, hey, Arthur. I wouldn't, I wouldn't do that that way if I were, you.

Remember, we have to, we have to do this in Europe and we have to comply with gdpr, right? Or mm-hmm. We have to worry about privacy.

We have to worry about performance and, and the system reminding you, or, Hey, you know, your coverage is a little low here, which you'll like me to try and bump it up. Right. I, I've got a few things I can try to increase the number of unit tests and, and I think, I think AI under the hood, just helping you work is much better.

And, and as much as, you know, engineers, we all want to know how the AI did it, but you know, if it starts doing the right thing, it, it matters less and less. Right. You learn to trust it.

Trust It. Right. Exactly.

Yeah. Now, that may be the most dangerous thing ever because, you know, if you've played with these things, you see that sometimes they give an amazingly good answer. Sometimes it's like the obvious and sometimes it gives you an answer that looks right and is just wrong and that's insidious.

Mm-hmm. Right. If you're generating code and the code looks right, and compiles code can compile and be horrible, right?

Like, I mean, it can compile and pass static analysis and not meet the business requirement. Mm-hmm. Right.

1, 1, 1 minor thing in an interest calculations is, Yeah, so I, I think that the generative AI stuff is cool, and it's gonna be interesting to see how it really shakes out and how we, how we learn to use it in a regular life. But I think it's made testing far more important than it was six months ago. Mm-hmm.

Right? Because now we really have to look at things and because they're already done, they've missed that first layer of eyeballs. Normally, developers are staring at now, if the developers cutting and pasting from an AI thing, he may be doing a quick once over, but hasn't looked at it as deeply as he would've if it was written, you know, by himself.

So yeah, testers really have to be out there understanding is this working, not just is it well built, but does it really do what we want it to do in all the cases that we wanted to do it. Mm-hmm. And, and I think we're still in the formative, uh, phases, if you will, of how much of that are we going to use.

Now we have, you know, in intelli code and co-pilot and things like that are built into our IDs, and those have been with us for a long time, and those, they're becoming more intelligent, um, gender of AI based. There's also the go into chat G P T and say, write me a Python script that does blah, and it might work, you know, I'd have to like, let's run it and play with it and test the code and really see, but it's sort of like taking a, it's, it's kind of a touring test for code, right? All right, let me see if this really actually does what I asked it to do.

And if you weren't the person at the right level of skill, so maybe we just take that, paste that into their, uh, browser or their runtime at home or whatever. And I think, you know, it's kinda like that spreadsheet that's rud with errors, but you don't know it unless you know how a spreadsheet works. Yeah, yeah.

And that, that's the the most scary thing, right? Because some tests are completely safe and innocuous. I mean, even if they're giving you a wrong answer, they're not, they're not harming anything other than lying to you.

But, you know, imagine that you're working for a bank or whatever, and you're doing some real testing and, and somewhere the script does something really stupid and starts changing people's balances, right. Or, or deleting accounts. And it can be a real nightmare, you know, and it's certainly plausible that this is going to happen.

It will, it will be one of the outcomes. I don't think it should stop us from using ai, but I do think that we, we need to really take a, a cautious look. I I almost feel when I'm having these discussions that we need to have like an s ae level like we do for driving.

Mm-hmm. Mm-hmm. Mm-hmm.

Right? Where we have level one and level two is the human's really in charge and level three, the, the system's in charge, but the human needs to keep an eye on it. And I feel like we're probably at like level two right now, s SAE level two with, with some of this stuff where we're asking it to do a thing and it's giving it to us.

And, and we may move very quickly into a level three where it's the driver and we're supervising, or we may not, that might take five years. We don't know how big and how quick the next leap will be. It'll be fascinating.

But at the moment, I don't think AI's gonna like, destroy tester's jobs right now. I think it's gonna create them. Ultimately, change always destroys, but I, I think in the short term, I think it's gonna help testers get more work done.

And, uh, hey, um, Before we ran outta time, I make sure we, we have some time to talk about your conference that's coming up. Yeah. Uh, tell us a little bit about that.

Yeah, so this is the automated software, uh, testing and quality conference. We've been doing it for a few years now, and it's a really fun event where we actually reach out to our customers and, and we, we reach down into real practitioners and talk to them about problems they have. So they're on and they're doing a short presentation, maybe 15, 20 minutes, talking about an actual business problem, what they did to solve it, what worked, what didn't work, how they figured out their metric, their determination of success.

And so, you know, my goal is that you could show up there, no matter what business you're in, here's someone else's problem. Maybe they're a bank, maybe they're a car manufacturer, but you can hear that problem of, you know, expensive resources, decoupling, uh, doing security earlier and learn what someone else did and go, Hey, I, okay, I can apply that in mind. My goal is at the end that everybody's learned how to solve some problem, not how to use the tool, but just how to approach the problem that you have that other people have the same problem.

Guess what? Every, we all do have the same problems, right? Well, that's, we're sharing those stories, right?

Sharing those experiences. Yeah. Yeah.

So it's, I think it's really fascinating. We also have a, you know, a speaker, uh, from Forrester, uh, Diego Uch talking about AI in general and about touring bots. And so that's some fun stuff and we'll, and we'll cover the spectrum ai, uh, use at Parasoft, we basically implemented little AI augmentations at every layer of the testing pyramid, whether it's, you know, UX testing or, or all the way down to static analysis, just different aspects.

So we'll talk about that. And it's, it's, there's some pretty interesting stuff I'll just designed to make people's everyday life better as you're, you know, either developer or a tester. Very cool.

Now, is this an all virtual conference then? It's on May 16th, right? Is that Correct?

Yeah, yeah. It's May 16th and it's virtual. It's, uh, four hours starting at 8:00 AM Pacific.

So whatever that translates to where you're at, uh, again, the sessions are, are pretty short. They're roughly 20 minutes. There's a panel.

We have the, uh, the ability to do q and a with people, so you can ask live questions of the speakers without pestering 'em at the end. We'll, we'll get a few people together so you can ask. So if you've got questions, it'll be great.

But, uh, I think it'll be a lot of fun. And of course, afterward we'll make things available through, you know, various channels like YouTube. Excellent.

Excellent. com site. Find out more to sign up and of course, check out your products and the capabilities that you have.

Yep. Go to parasoft and then hit that register button and we'd love to see you there and hear from you there. And if you've got a great story you'll wanna tell, reach out to us.

Very cool. I'll bet some folks do. I hope.

I'm sure they'll be some bright ones. Well, Arthur, thank you very much for spending some time with us and chatting a little bit about, um, both the software testing and kinda how it fits into the today's age of software development and where we're kind of going with AI and some of those capabilities. And it's been a lot of fun.

Wish you the best at the conference. Thanks. It's been my pleasure.

All righty. Please check 'em out. com and you can, uh, find out more about the automated software conference.

All right. Thank you much. We'll see you soon.

All right. Okay.