DevOps Building Blocks Part 2: Power Up Your Automated Testing – DevOps Unbound EP 39

Hey, everybody. Welcome to another episode of DevOps Unbound. Yes, you're here back on DevOps bound with us again.

We have a great topic today. Another fantastic panel. Uh, my partner in crime, our, my co-founder Alan Shimmel, is on a world world tour somewhere who knows where on the planet he is today, but he's off, uh, doing some important wor work, uh, and not able to join us today.

So I'm, I'm doing both co-hosting moderating duties, so I don't know how you do that, but anyway, we'll, we'll jump right into it. Hey, I wanna say thank you to the folks at Tricentis Tricentis sponsors this show, uh, which, which brings a lot of things with it, um, brings the unique perspective to add it to some of the conversation and topics and thinking about speakers. And, you know, we co-collaborate on that and, and work together on how we kind of think about different topics.

Sometimes is about testing, sometimes it's not. So I want to thank, uh, Lanier and the team at, uh, Tricentis, and also Jody, our, uh, producer, executive producer of the show. So, why don't we jump right into our topic.

I'm gonna start by introducing our panel. Um, and just to kinda give you a flavor, teeing this up, this is part of a building block series, uh, part two, powering up your automated testing. So, uh, Tracy, would you kick off introductions?

Absolutely, Mitch, good to see you again. And sorry, Alan couldn't be with us. He always adds a little bit of a flare to these.

Yeah, he does. I am Tracy Reagan. I am the, uh, CEO of Deploy hub, and I also manage a open source community called TIUs.

And I'm honored to say I also do, uh, tech strong women, uh, for Techstrong tv. Fantastic. And who should introduce themselves next?

Oh, I'm sorry. How about Raju? There You go.

Hello, uh, thank you so much for giving me this opportunity. I'm Raju one of the principal architect, um, at Software Quality, um, and release organization and T-Mobile. As part of my job, I try to focus on two things, increase automation, uh, have more reusability, and that's where I build the architectural solution to help my organization.

Martin, you are next. Thank you. And thanks for joining us.

I apologize in advance from a voice. I just, uh, got over a bad call, so you can probably hear this just a little bit, but I promise, uh, it's getting better and I'm glad we're on Zoom, so I'm not sharing, uh, too much, uh, or more than I wanted on, on the call today. But, uh, my name is Martin Laus.

I lead, uh, customer engineering at, uh, Tricentis, which is a new role. And, uh, the charter for us at customer, in our customer engineering is to help drive the adoption of more automation, uh, across multiple disciplines, but also including Generat, AI and some of those new technologies, and to capture those building blocks and blueprints, um, that we can sort of scale out, um, across the, the entire install base. Fantastic.

Thank you everybody. And I interested myself, I'm CTO at Techron Group and also run our analyst business to at, uh, tech Strong Research. So let's jump into our topic, and we're talking about this in the context of DevOps.

And there are a few core principles around, um, kind of implementing DevOps, and there's a lot of flexibility about how you do it, but one of them is automation, right? We're trying to increase the velocity, shortened cycles, produce software more quickly. And of course, to do that, you don't have to automate a lot of things.

Uh, and just kinda sharing my own background, you know, and running engineering teams and being part of delivering products and softwares. And it is, you know, testing is so extremely important. You want, you wanna automate to improve quality and maximize your resources.

It's also a double-edged sword. You don't wanna spend all your time maintaining tests either, right? So what do you automate?

What are the best strategies for doing automation? And how do you fit that into a DevOps pipeline, into a workflow process, to pipelines of work that's traveling through, uh, you know, a DevOps cycles that are landing into test environments and, and production environments eventually. So we're gonna talk about that and, uh, Tracy, I'm gonna go back to you to kind of kick us off.

I'm curious, you know, you, you have such a great sense of software engineering and software architecture. How do you think of testing and test automation and, and how you've put that into context of a DevOps workflow pipeline? Thank you for saying that.

Well, you know, I really think that testing has been sorely, um, overlooked over the years. Um, we hear all kinds of things about the DevOps pipeline. We talk about the build, we talk about the deployment.

Um, we might even talk about, you know, uh, source code analysis, but we don't ever talk, we don't talk about testing enough in the DevOps pipeline. It is, um, you know, it, it gets forgotten unfortunately. And it's such a critical piece of the DevOps pipeline that these conversations about adding testing and making sure testing's party op, the automation process is really critical.

Now, there's different kinds of testing, um, and maybe you can't add all testing to the DevOps pipeline. Uh, but I think if we think about the traditional pyramid, um, kind of unit testing, uh, functional testing, for example, user acceptance testing, the areas that we really should be focused on automating, especially in a kind of decoupled environment where we have functions and we have contracts, is the unit testing. Unit testing should absolutely be part of the DevOps pipeline.

And if you're not doing that, you're missing, um, a big component. Amazing. Raj, I'd love you to jump in.

Um, first time, definitely first time guest, longtime listener, right? You know, yeah. First time.

Welcome. Yeah, I definitely agree with Tracy. Um, testing was definitely forgotten, or it's being forgotten.

We'll be looked only when we, uh, when there are issues in the production, right? Um, but in order to have more automation in DevOps, we need to see what are the things that needs to be automated, right? Basically, what are the things that can be easily automated to get started?

So if we talk, you're talking about the unit testing, we are focusing only on a specific piece of code. However, the, in, when we are talking about the entire, uh, module or the functionality, we need to focus on the integration testing, right? We need to mature enough, uh, from the unit level to the integration level so that now you are not only focusing on a single component, you are making sure that when that particular component is deployed, it is talking to that other component that are dependent on this particular component properly.

Right? That's where we need to focus on. Um, as Tracy said, probably we may not be able to, uh, incorporate all the automation, uh, into the DevOps.

However, uh, we, at least when we go step by step from unit to integration, uh, because integration will be in the APL level, it'll allow us to automate quicker. It'll allow us to, uh, do it faster and slowly we can ramp up to the functional, slowly we can ramp up to the, um, uat, right? Um, however, uh, and also we need to think about, okay, now the automation is running.

How do we make sure that if, what is going to happen, if the automation fails, what is the criteria for us to say, okay, if these test cases are failing, then rollout, roll back my code. If these test cases are passing, then don't roll back my code. So the criteria needs to be defined properly.

Yeah. You know, I heard a, uh, talk, this is actually, it was at the end of 2019, the last conference I went to. I went to a FENOs conference in New York in December, um, and there was a, I don't remember the bank, but there was a bank there talking about what you just described.

And they had created truth tables to make those decisions based on a particular modules. Um, I thought it was a fascinating way to see, um, how to, and they added that automation into their DevOps, their testing pipeline. So they added DevOps pipeline, and they also had a testing pipeline, but in their testing pipeline, they added truth tables to do what you just described.

Yeah. Because that is going to definitely help, um, the, um, people who are actually working on the deployment, right? Um, we don't want to blindly roll out, roll back the code if some tasks are failing, having a best criteria to decide what needs to be, what should be the true decision to deploy or not deploy.

Mm-Hmm. If I could just add something to the point that Raja made, uh, which is, um, that most companies have their own way of deploying features and releases into production, right? And so the automation really needs to fit that model because some organizations have deployed, you know, feature flagging as a way to introduce new functionality, uh, to into, to the, to customers.

Other companies use AB type deployments, or, uh, other companies are actually, uh, have an ongoing sort of opt-in beta program where they're basically, you know, uh, leading with new features. And then you can roll back to the, to the classic or the, the not yet fully GAed version of, of the software. And in any of these scenarios, and I'm sure there's nuances to that as well, um, that has an impact on how you think about a test automation.

Because you may not be able to do full end-to-end test automation in a staging or, uh, you know, Q environment. Uh, when you have production data that actually is very critical from an automation standpoint. And you have to think about, you know, what can be tested automatically, uh, in a pre-pro staging environment, and what actually do you want to test in production as part of like a shift right?

Strategy, uh, because you just can't replicate your production environment, um, you know, uh, at, at high fidelity in a, in a development environment. And so it, it really depends on like, how you think about your delivery process. How do you think about, you know, the way, introduce new functionality, how do you think about, um, you know, test automation in a context and what can be done on a developer's, uh, laptop, uh, as part of a unit, uh, testing, uh, mechanism and what can be done later stages, uh, before it hits, um, sort of the, the G environment as well.

And, and we're, and we're really focused on code testing. You know, just make it clear we're talking about code testing on pin testing or other kinds of testing that you may wanna do in your production environment, and it must do in your production environment. But I think the concept of being okay with, uh, with understanding that you still may be continuing testing even after you do a production release is important.

Because to be quite honest, we never stop testing our end users eventually might be the last testers. 'cause they're the ones that often find our bugs or the ultimate testers. They're the ultimate testers.

They are the ultimate testers. Um, but how do we, you know, make sure that they don't find as many as, uh, we might in the earlier stages. And that's where the automation has become so critical.

But what I have found is that in you, when you look at a lot of these, uh, when you look at a lot of pipelines and you're looking at how people are constructing their, um, their, their testing, they may run just one test script. They may use a selenium and feel that that's gonna be sufficient. And I think what's important to understand is there is, uh, context to testing and sometimes adding additional tooling, um, helps improve the overall quality of the software, because you have context around it.

And I, I can't really explain that per perfectly, but there's, there's more to just, there's more to just scanning code. Let's just put it that way. Let me jump in on that because one of the things I wanted to to highlight is, you know, I, I no longer think of testing as a discrete function.

I think of it as a continuous process from the moment code's checked in. Maybe I even run some tests before I check in code. But that, in that build process, right?

We think of DevOps and kind of the center of the picture of at least people start with CICD. But it's more than that. It's more than the integration process that's kicking off.

It's the test process that's kicking off. Matter of fact, that can happen when, you know, just code check in and, and, and I think, so it, it's continuous happening whether you're doing it on your own developed machine, if it's part of the workflow of checking in code, doing builds, and then also pushing, uh, code into test environments. And one of the things that you said, Martin, also I think is, is because of automation, we can push code for testing into multiple environments, right?

How many times do we have to test different versions of the products or different feature flags turned on, or, you know, we've had to separate and we were in multiple systems because of separating customer data. 'cause we're not multi-tenant or whatever it might be. There's al always cases where we need to go back and fix something in a prior version, or maybe we're running multiple environments and we have to be able to test across that.

Doing that manually just wouldn't be practical, not on any scale. And that to me is sort of the end, one of the end results of having tests automated is by the time it reaches that step, it's already been tested hundreds, maybe thousands of times with test scripts that kick off at different points in that process. So that, that's why I like to think about it as the flow of how testing happens, not testing happens here year.

Yeah. And, and the success of this also depends on the config, how good your configuration management is, right? How, how good you are understanding which code is going to be deployed when.

And we, we understanding because of this particular code, what functionality is impacted, um, until that configuration management is not in place, then you will not see the success into what you are trying to achieve in the DevOps automation. Uh, because you need to understand what exactly needs to be tested, as Tracy said, if at all, you just have one unit test case, uh, uh, just to check the check box and, uh, to say, well, my, my DevOps is, um, everything is automated in my DevOps, right? Having one test case is not going to suffice you.

Right? Understanding what exactly needs to be tested, why it needs to be tested is important. And Raja, you said the magic word that nobody ever says, har, I hardly ever hear anybody use the term configuration management.

Yep. It is the essence of everything we do through the DevOps pipeline and this new conversation around supply chain, uh, whether you wanna call it CICD, but configuration management is the essence of everything that we do. And it's in so important and critical to testing.

So for example, you know, uh, key value pairs can really ruin your day. If you have two key value pairs in two different environments are acting differently, it's really hard to find, um, those, those, uh, kind of hidden, uh, configurations. And that's why configuration management has been a discussion for years, but I feel like it's been forgotten.

Uh, kinda like testing gets forgotten. Um, so I really love that you're bringing back that term. It's super critical, and I hope our audience understands that it's not just an old term, we're still doing it.

Yep. And, and it also allows us to eliminate the duplicate testing, right? If at all you understand, based on the configuration management, you'll be able to understand what exactly is going to, uh, change why it is going to get changed.

And now you are, once you define that particular package, now you understand what exactly needs to be tested, so you're not over testing it, and you are also not repeating the testing or a period of time, whether it's automated or manual, forget about it. But are we doing the right testing? And that's, that's key.

And in order to make that as a key, you need to make sure that you understand your configuration management better. And, uh, when are we done testing? Because one of the things that I hear sometimes from customers is there's this enormous, uh, test matrix that, uh, starts to explode every time you add a requirement and you add a new business rule, or you, you think about, you know, exceptions, uh, and, and what that means in terms of, uh, implications on your test matrix.

And while you think you have a fairly comprehensive test matrix, there's always, you know, the, the, the sort of the, the doubt of like, is this it? Or have we not covered a particular edge case or, or a situation that is not in the happy path, but it's sort of at the intersection of multiple business roles and exceptions conflicting with another. And you can only find it out, uh, when you're actually taking the application through its paces in production.

And so one thought that we've been thinking about is like, if, if we assume that you're never gonna be done, like, or done, done, uh, there's always something that you are not going to catch, uh, before you go live. But it's gonna happen, uh, after go live. And as Tracy mentioned, the end users are the ultimate testers because eventually somebody's gonna run into that situation, uh, at some point in time.

And how, how do we sort of identify that and have more of a, sort of a rapid response, uh, approach to the situation to say, we assume that we're not gonna be a hundred percent complete ever. There's always gonna be a certain percentage that, of things that will not catch. But if, if we, if that happens, then how do we then sort of identify that situation and respond to it as quickly as possible?

So this whole idea of observability integrated with, uh, test automation might be an important topic for us to think about in the future to say, you know, uh, you know, slows new downtime, but maybe there's some other early indicators that would, uh, sort of allow us to detect when these sort of edge, uh, cases and situations occur that we can then as a jump, uh, uh, on it, fix it, but then also then capture that situation and add that to a, a regression, you know, uh, suite, uh, for the future, right? Because it's an important sort of, uh, situation that we, uh, uh, haven't thought of of before. Yeah.

And I think, uh, you know, as, as, as we all are, right, testing cannot be done, done, done. It's, it's always, we are gonna test, but in order to do the right testing, we need to make sure that we prioritize our test cases. The prioritization is very important, right?

What is my key priority, right? What are my test cases, um, that if I run, I know these, these many customers are gonna go getting impacted, right? Uh, if these are gonna customers.

So you'll be able to prioritize your test cases, and then you run your, um, uh, test cases through that. Yeah. Dependencies become part of that configuration discussion too, right?

Yeah. All the way down to transit or dependencies. You know, the more we talk about open source security and SBOs and trying to understand what you're consuming, you know, there's an argument to say that if an open source, uh, module that you're pulling in, in your package, uh, that's a transit of dependencies that should actually kick off a new test.

But we hardly, we, we really can't see that, uh, deeply into some of these packages. Um, and if we look at the, at higher level, the organizational level in a, say a decoupled environment in a microservices environment, which happens to be one of my, one of my favorite topics, because it, it really makes production look like this massive transformer. And you're, you're updating components all day long.

And Raju, as you pointed out, you know, the, in the integration testing is so critical. How do you continually do that? How do you continually test these integrations?

As we're doing agile, uh, development, we're doing microservices, we've decoupled everything, and one single microservice could blow up several applications. So the configuration management becomes even more essential. Understanding dependencies become more essential.

And personally, I think that kind of intelligence should be part of the DevOps pipeline and pass to testing tools, because that shows where the risk is. So in other words, if you have a, you have a login routine that everybody's using, 'cause you wanna standardize it and you wanna make sure it's secure and it gets updated, that has a higher risk, uh, value than other, um, something else that might just be changing the color of a, you know, of a window frame. Um, so how do you understanding when you should test what you should test and the risk value is something that we should be gathering and, and reporting on through the DevOps pipeline to make it easier to do that testing.

Completely agree. And apart from this two, uh, apart from this discussion, what we have, uh, done so far, I think we also need to, in order to be more successful into the automation space, we need to make sure that you are talking about the test data, right? Without, without the proper test data, you will not be able to succeed in the automation, right?

Uh, whether you have built a proper, uh, solid script that is going to run, but, uh, the goal of automation is to run, run, run, right? If at all, we don't have a proper test data defined for those test cases, then you'll fail, right? Um, either, uh, the, the people who are going to run the automation is going to get trusted to understand why the tests are failings, though I did my best job to, uh, build a proper script.

Um, and they'll spend too much of time triaging, uh, if at all. We don't have the, uh, best test data. Um, and then the second thing is, when they're building this particular automation, in order to be more efficient, uh, we need to think about the reusability concept, right?

You, you build your automation in such a way that you have a reusable block that can be used between the test cases so that you're not scripting entire. Um, so for example, if at all, you, you want to talk about login? Yeah, login was automated.

Now the next technique is I want to make sure that I want to do something on the dashboard. So already login was automated, if at all, you want to leverage the dashboard test scenarios, leverage the login that was automated, stitches together, and then you have that usability. So that way you are spending right amount of time and right amount of energy to build a proper test case.

Have question for everyone. And, and thank you Raji for that. Do, do you think about testing differently?

Do you approach it differently when you know you're gonna be integrating it into a DevOps pipeline versus maybe traditional software development where we, maybe we're doing ible, we're not doing DevOps and have tool chains that are integrated, things like that. Are, are there, are there different thought patterns or considerations that you need to make? Yeah, the thought pattern is to make sure that you are just not writing a test case.

Just someone told you to write a test case or build an automation test case because someone told you to. You need to understand, as we discussed earlier, right? You need to understand why, what is required and how do I implement it, and how do I have that particular dependency, uh, to ask yourself whether I'm doing the right testing for this particular changes.

Anybody else perspective on that? You know, I've thought about that, you know, my brain really, it has, um, hardened in this area. So sometimes it was hard to like relax it because I still always think about that traditional pyramid, you know, that we have to go through.

But I do think that we should be, um, breaking down testing into, uh, a more modern bits and pieces, uh, because that's where we're headed. And it may be the case that we don't always have the benefit of that traditional pyramid. Um, like I said in the, in the microservices environment, I think we have to really see how we can do testing differently.

In that case, I don't think we can always have there, it's impossible to have a single microservice go through absolutely everything. So yeah, we, um, we have some work to do in that area. Mitch, I think it's a good topic.

It's, it's interesting because you almost have to design your testing like you would design software. Think about it as a mod, you know, like an API first, if you're thinking microservices is kind of a design, right? It's not one big monolith suite of tests that you wanna run.

And, and to your point, Raj, it, we talk about the tests themself, it's the data that ultimately matters. 'cause that's also what's gonna drive logic through paths in the code or not, right? And represent, you know, the flow of, uh, you know, use cases, transactions, things like that.

So it is a pretty complicated matrix of things to consider. That's, Yeah, when RA said data, you know, one of the things that I've thought about on the data side is when we start talking about ai, AI needs clean, consistent, honest data, I, I feel like we have to start thinking about how we test that to determine if our, our machine learning is being based off of clean data. And I feel like that's a realm that the testing community has to start thinking about.

I don't know if we have yet, but it, if we look at the future and we look at, you know, building out better machine learning around the DevOps, uh, problem in particular, having that clean, solid, consistent data that we can have, so historical trends becomes really important. And I have no idea how you test that data. I think it'll be an interesting challenge for us.

Yeah, It'll be interesting challenge. And also, um, when we talk about data, we also need to be more cautious about the data because a lot of, we don't want to involve customer data, right? Because of the CPNI.

We want to, we want to make sure that we have data masking in place so that you are not giving the confidential information of any customer. If it all you are testing ai, AI or any, any place. We want to make sure that wherever we are talking about data, we are not actually using the right, um, um, actual customer data, right?

That's where, uh, data masking will come into the picture. Is, is there a difference between data masking and anonymizing the data? Or is that the same thing?

Right? Uh, so I think anonymizing, you, you are still doing some kind of, uh, so for example, you are saying, okay, uh, instead of saying ra, you are going to say some something like anonymous num name, but it's still in name, it might be relevant to someone else. Um, and now when we talk about data masking, there are, there should be some tools and technologies which is going to mask the, the data in such a way that anyone should not be able to recognize that particular customer, right?

It cannot be, it could be anything. Even the credit card information, even the address information. So there are a lot of information that needs to be, uh, data masked.

Very good. Um, let, let's turn our, our attention to thinking about the tools side of it. Obviously, you know, tricentis and, and being in this business, uh, you have a, a very keen perspective on how customers think about tool selection and integration.

Um, I was just working on a study about integrating the DevOps pipeline. What are the considerations around integration, security automation, um, telemetry and data that you use, you know, output for decision making and things like that. Martin, maybe if you can, I, I love that you're in the customer success business.

So as you're, as you're guiding, and I know we're not, we're not doing a commercial for tricentis and understandably, but you know, we obviously come with that, with our perspectives, right? I'd love to hear your thoughts on how you help customers walk through that decision matrix of maybe they're already using some of your tools, maybe they're new to, to coming on board. What are the, some of the, the things that you ask and help people think through?

Yeah, I think the, that's a great question, right? Because in our experience, we, we sometimes run into situations where customers think about tools as, as a way to solve all the problems, and especially people and process and maturity, uh, questions. And that's not it.

If you, if you think about buy a tool or suite or platform, what have you, and that will solve all the challenges, um, that may not be the right investment, I think you end up with, uh, you know, having tools on the shelf and not getting the value, uh, you know, out of, out of the investment. And I think, I think the most important aspect in, in our view, is always to take a step back and look at where are you today? What are you trying to accomplish?

Uh, where are you having success already? And where do you think there are opportunities to improve speed or automation or reusability or coverage, uh, from a, from a tooling perspective? And then figure out what is the right way to do that.

Um, because we've seen customers that have success with scripted automation tools, uh, because they have, um, some sort of an engineering team that is really good at, you know, uh, looking at automation as a product, and therefore it's constantly maintained just like a code base is, right? And so they never run into a situation where they're being frustrated with, uh, you know, scripts have break, and then they find out too late and spend too much time fixing the, the scripts rather than adding more, uh, you know, automations to the repertoire and library of automations. Uh, in, in our experience, it's usually a good sort of, uh, starting point to, to look at where are you today?

Where, what is your maturity in general relative to, uh, the use of automation tools, you know, for, for build automation, CSE automation, deployment automation, security scanning tools and so forth. And then also from a, from a, from a team's perspective, what, what is the skillset that you have and what's the overall maturity of the teams? And thinking through what needs to be, to your point earlier, Raj, like what needs to be tested?

What is the, you know, what has the biggest business risk from a, from a, from a customer standpoint? And then prioritizing those things as well. Because there is a, a point of diminishing return of automation where the more regression you do, it's not going to sort of, uh, filter out in additional issues or defects, but it's going to add time, uh, to deployment cycle.

And it's about figuring out how do we run automation in the most optimal way, uh, you know, so that the CSD times don't slow down, uh, but it actually capture things that need to be tested, uh, correctly. Right? But I think tools, in my view is sort of like an important, but not the most important aspect of the equation.

It's always about the people and the process first and the mindset. Uh, and then figure out how does, how do tools, open source, commercial, what have you, uh, support the process in the most optimal way. Definitely agree with Martin.

Yeah. And as he said, not all tool, not a tool will solve all your problems, right? You know, to identify the right tool to solve your problem.

And when you're trying to solve that particular problem, what is the benefit that you're going to get outta that particular problem? Yeah, I think we have to look at testing like domains, right? Mm-Hmm.

I like to talk about domains because they're important. Uh, 'cause if you can define the domains that you want to ensure testing, then you can add it to your DevOps pipeline or not. Uh, you know, where it has to be done.

You know, production might be a domain or pen testing might be a domain. Security testing might be be a domain unit. Testing is a domain.

And what are the tools that best fit those particular domains, and how's the best way to automate them or not? Um, because there may be a case where this is not something that can be automated, but for the most part, I think most of our testing tools can be, and adding, creating a DevOps, uh, testing workflow is a really good way to think about your domains in a different way. And not just dev test prod, but what the testing is doing, what is it, what particular area is it, uh, is it serving?

And build a pipeline that that is appropriate to those domains. Yep. And, and as we are talking about the DevOps and automation, I just want to highlight this to everyone that I think we still have a lot of opportunity to mature in this area because the tool, any tool that you choose, there is no proper integration between the dev pipeline and automation, right?

It's a patches of work that is being done, right? They're, they're, they're, they're trying to just integrate, um, with patches. It's not a seamless integration that would allow, uh, anyone to adapt to this particular framework, right?

We don't have a defined framework with the proper integration. And that, I think that could be one of the reason that people are, uh, not automating, um, in the, or not in including automation into the pipeline, right? Um, there might be just making sure that, okay, one checkbox, if at all my leads are asking me.

Um, but if at all we, we have not seen any, uh, huge, uh, focus in this area where we have a seamless integration with automation, test automation and DevOps. Now I'm gonna follow up on something Martin said. Um, I'm not sure if you used the term diminishing returns, but there is a point, right, where adding more tests to your test suite doesn't necessarily improve the quality or security of software.

How do you know, I mean, how do you know when you're approaching kind of that zone of, are we, are we got enough? Are we over, are we over-engineering this overthinking it? How do you, how do you recognize that you've kind of entered that zone?

That's a tricky question. That's the 30, $30 million, uh, lottery ticket question. Yeah, that's exactly right.

It's because it, it has to do, and I'd love to get Roger's thoughts on this as well, because it has a lot to do with what is the rate of change? Like, what are you adding over time, uh, to your application? Is it incremental?

Um, are you adding new functionality? Is it a, a new application? And also what, if you think about modularity, what actually counts as an application in its own right, versus a service that's being used in, in, uh, in a composite way in other applications.

And so, um, you know, in, in general, I think automation is a good thing in general. You know, the more you can continuously, you know, test and validate, that's a good thing. Um, but it does, you know, I would say if, if you look at sort of your de deployment times or cycle times in general, that's oftentimes a good indicator that, okay, so we're, we're slowing down here quite a bit and, and now we're waiting, uh, you know, more than we are comfortable with because it's sort of now we've sort of shifted the, uh, the bottleneck, you know, downstream just a little bit more.

And, and figuring out like, is, is the automation as as, as sufficient as possible could be one way to look at, okay, so do we have, you know, what we need? Uh, what are we catching, uh, you know, uh, what, what do we feel? How do we feel about the maturity of the code base?

Um, and then where do we feel comfortable maybe to turn off, uh, certain things that have been tested a gazillion times that don't need to be retested again? Um, which has implications on test, on, on test automation maintenance, right? Because maintenance is not about fixing scripts are broken, but it's also about sort of does my test matrix still reflect accurately what my application looks like?

And, you know, do we have aging, uh, in our test cases that, you know, we've tested things, you know, five years ago, we don't need to retest 'em again because we feel really good about it, it's stable, it's not changing anymore. But maybe we should focus more on some of the, the new functionality we're adding, you know, on a, on a more regular basis. Yeah, definitely agree with Martin.

Um, the changes, uh, understanding what changes. And also the other way are to look around and understand whether we are doing the right, um, uh, scoping or not as to historical errors. Uh, so understand whether the, what kind of issues you are finding, um, is it reoccurring, um, in which areas those are primarily focused on or where are the, uh, key, uh, issues that have found in this specific area, which, which tells you the, the story, if at all.

You don't have the configuration management. Okay, there, this could be the area where a lot of changes are getting deployed, right? And that would help us to, uh, focus on, uh, increasing automation in that particular area.

And if there is some areas where the, you're not finding an issues after running the automation or manual test case, then that is the area where you can minimize your automation or test cases. And I think that this, this juncture, we should give a shout out to value stream management tools because they, um, were really good at showing us where we had bottlenecks. Um, and if quality gates were way taking way too long, uh, because when that does happen, uh, developers get frustrated with the testing and they start wanna wanting to break down those bottlenecks and, and break down those quality gates.

But you have to remember all of this depends on the industry you're in. If you're in, uh, banking, um, if you're in government, uh, you may be okay with, um, uh, having more than needed. Uh, and if you're in another industry, you may not care as much.

You still care, but you don't care as much as the risk averse industries. And I think that having the insights to understand how much time it's taking, um, is pretty important to know if you have hit that diminishing return point, Right? And sometimes we don't see that in the pipeline.

It's hard to see that. And that's where the value stream tools were really, um, helpful. Like one of the factors is also, um, what kind of things are you not catching right in your testing that, that customers do find or you find 'em in production, um, after you've gone through test cycles?

And is there a pattern to that, right? Do we have an area where we need to shore up our test cases? Um, you know, it's a funny way to say it, but it's not, it's not the test that passed that you're looking for.

It's the failures that you want failures to happen and you want to happen, have them happen as early as possible so you get a chance to fix it. I used to kind of gut feel software releases by how bad the, the, the bugs we were finding and we're finding some pretty significant, you know, really rough ones. And I could tell if we're, we're not there yet until we start to see some of those things, you know, as a product matures, you get less of that, but that's, that you, you, you have a sense of what you're finding.

The errors you're finding also informs you how your testing is going. Yep. Am am I just crazy or is that, is that it's kind of a sense you also apply raju.

That's, that's, uh, that's what we exactly talked about, right, Mitch? So yeah, that's, that's exactly will help you to understand whether we are finding the right, uh, um, errors are not, and we're fixing the right errors or not. Yeah, definitely making, Yeah.

I was like, great. I love those big bugs. Like that tells me we're finding good stuff.

Um, we're coming, coming up, uh, near the end of our time together. Um, you know, there's a couple ways to kind of approach how we close this is thinking about, you know, all of us have been working in the industry a while and shipped a lot of software and have experiences that we've, you know, you learned from your failures as well as your successes. Is there anything that you, if you rolled back time, maybe back 10 years or, or or so ago and said, boy, if I would've known what I knew now, this is what I would love to take back to myself 10 years ago and, and benefit from that knowledge, um, any kinda lessons learned?

Because so much has changed in how we do software now. If you knew you were gonna be doing microservices 10 years from now, right? You, you're like, oh, I might rethink about how I do this.

Um, Tracy, I'm gonna, I'm gonna impinge on you and ask you to go first on, on maybe a tough question. Okay. Maybe it's an easy one.

You Know, it's, it's a tough one because there's so many areas that we have, you know, fallen on our face over the years with, uh, companies I've been with. Um, and you're embarrassed by it. But I think it goes back to something that Raji reminded me of, and I I I always still forget it.

It's the broader integration testing that will always get you, um, especially if you're passing data across and somebody changes one little component, even, you know, uh, a memory boundary, uh, it, that those are the hardest tests to do and they're the most embarrassing bugs to find. Uh, so I think I would've told myself focus more on integration testing and really get it down. Yeah, the things you can't control in your code, right?

Exactly. Exactly. And I would, I would take it back, like, uh, I would would've focused, or if at all I would've owned a team, I would've told them to make sure that you're focusing on the configuration management and dependency because this was going to help you.

If you build this configuration management from the beginning itself, it is, it is then you, then you are not going to spend a lot of amount of work in the future. So if, if at all you keep updating that particular configuration management and dependency from the beginning itself, then you'll, it'll be easy for, it'll make everyone's job easy, even the developers and testers and even the product owners who is going to understand this particular, uh, product or, um, software. Excellent.

I Would add that, um, it's important to not forget about the human in the loop. I know we've talked about that in the context of ai, but at the end of the day, we're delivering an experience, right? Yeah.

Uh, you know, it doesn't matter if the functionality checks out fine and, and there are no issues from a test data standpoint. But if it's not usable, if it's not engaging, if it's not something that you would love to actually interact with, then all the effort is actually not going to yield the kind of result that you're hoping for. Because at the end of the day, we're still building software for humans, and we have to think about the application from an end user perspective above all.

And that relates to, um, you know, the functionality that relates to the requirements that relates to the things that we can automate as well. And if, if you really think about that way, let's say like if, if I was a new a, a user coming into this, like how, how would I feel about interacting with the software and does it want me to come back and recommend it to others? And, and I think what Apple and Google and other sort of, um, you know, app Store has done is they have really created awareness for user ratings.

I feel like we need this type of rating system for enterprise software as well, uh, in many cases. Uh, but I think that will influence and inform a lot of the automation strategies or test management strategies or value stream and other things that, that we use in the techniques and the process and tools, uh, to deliver software that actually was engaging and exciting, uh, for our in in customers. Yeah, I think that the word there is empathy.

Mm-Hmm. Uh, something actually, I learned from an interview on tech strong women, uh, women who, uh, is really into ux and she talked about how important empathy was for your end users and having true empathy for them. And, uh, that is a responsibility of testers.

Extremely. Good point. You know, something that, uh, I went through an experience where it was during an economic downturn, and I ended up spending about a year and a half, almost two years, part-time on the road doing, uh, we call it customer testing then, but it was user experience testing of software we were designing.

And I walked away with this sort of like, wow, I never would've guessed. It's the, it's the unintended uses of software that, that really stretched the boundaries. Like, you know, I remember developers saying, well, you're not supposed to do that with our software.

Well, people do, or you're not supposed to use it that way. Well, people do. And, and so it's kind of expand the bounds of the discrete definition of a test plan or a test suite or a design.

There, there are outer edges that, that, that gets expanded to beyond what we intended, uh, software, how it might behave or how it might use, and kind of keep an open mind and look for those, uh, what might seem like edge cases. But you never know. You might have a great product shift in, in what you product can do.

'cause suddenly you found a new problem that it solves you didn't know it was going to. So there's some really insightful things, you know, if you just kind of think of it as a continuous learning, not a, I'm shooting for delivering acts, and that's, that's all I'm after. So if, if that makes sense to everybody.

To me it was really like, wow, I had no idea people would do those kind of things. I learned so much from talking with people who were actually using the software. So, to your point, well, thank you all three of you.

It's been a ton of fun. It's always great to, uh, get in, get into the kind of details of this topic. And I, and I hope folks that joined us today looking for some insights in, uh, automating testing, got got more than that.

They got that, but also got configuration management, user experience, human in the loop, um, you know, thinking about integration testing, uh, more, more thoroughly. You know, so many kind of nuggets that I think any of us could pick out and use. So Raj, um, thank you so much for joining us.

Thank you, Tracy, as always. And of course, Martin, we really enjoy having you. All three of you bring such a great perspective.

Well, thanks everybody for joining us today. Thank special thanks to Tricentis again for sponsoring this program and bringing people like this together to talk about, uh, some really useful and helpful topics. So I hope you tune in again, and we will be back with another episode of DevOps and Band.

We'll see you soon. Thank you. Bye-Bye.

Thank you.