Test Less, Quality Over Quantity – DevOps Unbound EP 34

Hey everybody, Welcome to devops and band. We have another exciting episode for you here from Textron group. My name is Mitch Ashley.

I'm CTO of Textron group and also principal with our analyst firm text drawing research. We have a great panel, you know, this is one of the best things about this job is assembling just fantastic people both new and folks that we we know for a while to our topic today is we're talking about test quality our quality over quantity. How much testing is appropriate do we test last week and focus on the right kind of testing?

What's the what's the approach we should take we can automate so much but we automating the right things and spending our time where we need to we've got some experts in the testing field. That are on our panel today before I do that. I want to thank our sponsors that's tricentus sponsors devops and bound.

They've been a sponsor with this for a couple of years now and play an integral role in helping us put together these conversations and think of panelists. So even though the show isn't about their products specifically, they're contributing as a thought leader in in the space of devops and software and software testing and I think is a good example perfect example of a company who is also contributing back and through this kind of leadership opportunity. tv.

And we also have live roundtables that we do. com for the roundtables. So let's jump to our topic.

I'm going to start first by introducing our panel. So I'm just gonna pick you know, I'm gonna pick the new person Tristan is not been on and I found I found Kristen's book I said, this is a person we've got to have on because she seems fantastic looking at her book and what she's done. So Christian what you introduce yourself.

Sure thing. Hi. I'm Kristin.

Jackfoni. I am a principal engineer three for software testing at Paylocity which is an HR and payroll software company and as mentioned I am also the author of the new book The Complete software tester, which is now available on Amazon. It talks about all different aspects of software testing manual automated security testing performance testing.

So if you are interested in those things definitely check it out. I would highly recommend checking that out for sure. Congratulations on on your book.

That's the labor of love, isn't it? Oh, yeah. Next I'd love to introduce Excel welcome and again returning guest.

And tell us a little bit about yourself. Well, I'm a traveler. So I'm a Java Champion, but I have done all in my career from devops manager it so I might most of my sessions are about testing.

How when why and how to make it the most cost-effective? Excellent. Fantastic next I'd love to do introduce another actually prolific author.

Lee's been with us before and some some things welcome Lee. Thanks Mitch. It's great to be here and my name's Lee Atchison.

I'm a an O'Reilly author of my second book with the Romans coming out this week called overcoming it complexity and I also wrote the the book architecting for scale. I have a regular articles series on container. You're weekly participated Journal I write for infoworld I write for a digimonica.

I do Consulting for customers. I I spent eight years with New Relic and eight years with Amazon and AWS. And so I have a a lot of experience in the cloud and the cloud native and application modernization business process.

And that's the expertise to bring to my customers and the Battle Scars to show forward. I'm sure yes Sean. Welcome new guest Shawn from Christ dentist.

Tell us about yourself. Hey, thanks. I'm Shawn Jakes.

I've been In software for over 20 years must be on the vendor side. I started with IBM and a number of different areas there and then BMC Software GitHub for a while and then into testing with custom and they got acquired by tricentus. So really kind of seeing both the dev and the opsides of things from you know, from those different companies and you know from different roles like product management and strategy and product marketing.

So getting lots of customer interaction and really trying to understand what what they're paying points are Fantastic, how long ago was that acquisition? It was fairly recently. It was a February.

So yeah not too long ago coming up on a year. Okay. So under a year that's always a always fun interesting and congratulations and getting Acquired and then yeah.

Thanks a new world, right? Yeah. It was a big big change for us a lot of a lot of new process that we we hadn't had before.

So yeah. Here, you're joining a great company with tricentes for sure. Well, great.

Let's go to our topic here. So You know, I've been part of software teams my whole life and the the question of how much testing is enough. It's sort of like how much security is enough right how much testing is enough and in my earlier experience?

It was the linear waterfall process of all of your time got compressed out of testing and documentation that you just kind of squeezed in whatever you had time left and we've really done a great job of kind of rethinking that with devops and shifting laughed and doing a lot of things to automate things, but it's still I think a burning question of There's so many kinds of testing to do and how much testing what's appropriate to automate because then you got to maintain all that stuff. It's a real Challenge and there's even a school of thought that says, you know, what if we can deliver software fast enough we might be able to do less testing because we can fix it quicker kind of a counterintuitive thought there. So yeah like you to start out the conversation and just give us your thoughts on sort of how much is enough and how do you decide I'd love to and of course like everything else that we do it all depends on contacts and you know, someone developing a desktop application for in the healthcare industry.

He's gonna have a different set of of testing requirements and someone's building or SAS application. Right? I tend to focus more on the SAS application world and that sort of environment and in there.

We're we're really coming to the fruition that you know, the The key thing you have to realize is that the cost of a failure is dropping dramatically the faster you can you can fix a problem the lower you can make a cycle time. The less cost of defect has and the less cost of defect has the less valuable testing is in general and they're less you it must important it is to the application and really soon here now and in some cases, I think we've already it's really soon here. We're gonna be coming to the place where the learnings you get from seeing failures.

outweigh the cost of the failure itself And and once we get to that point every time something breaks and you fix it right away, you learn something about what your customers expected that you didn't understand or about the way the software work lead and understand you learn something valuable. And once the value of that is high enough. There's actually value in making mistakes and and there's value in learning from those mistakes as long as we detect mistakes early.

Text them quickly learn from the mistakes and move on. Failure is not bad failure is good and failure oxygen can be valuable to the building of the application. But although those things have to be true for this to be the case and that's not obviously the case with all types of applications.

But with the online SAS world of online applications, it's very quickly becoming that interesting perspective because if your velocity is such that you can react that quickly exactly. See your point about you can you can handle issues and I definitely agree with the failures is Rather than not an option. It'll happen.

It's always going to happen. So they're from it. So who would like to jump in next but your opinion counter opinion, whatever you'd like.

No, I loved your your answer early. And actually I one of my sisters it's failure is not an option. It's a fact and I agree with you.

There are like we learn a lot from failure. So they're exploratory tests to see what our customers want. We need that and when we fail on that kind of test, it's fantastic, but there are others which are more repeatable.

So those kind of failures are not adding anything to our development process. So for example, having the test to prevent regression tests still make sense because if it's a project that it's going to be in refactor for whatever reason you still want to be able to move forward. Even if you're not adding any functionality you to want to know when you grow things and the actual impact on your customers saying you broke this like three months ago and you break it now again, so What are we playing here?

So we have to strike a balance for for this kind another one the other topic that I want to bring and see the opinion of the of the panel is a developers should test our quality and Assurance team should test. Where do we meet? What are the separation of concerns?

I can definitely jump in to that question because of course. This is an area that I'm very passionate about. So my feeling is the whole team should be testing and the how that gets divided up is really going to depend on the team.

It's going to depend on the product. It's going to depend on the underlying architecture. But I I definitely think the model of making sure that the developers are writing the unit tests I think is very important.

I think API tests UI tests those can be shared by both developers and testers alike one model that I've seen that has worked really really well is having the developers put together the testing framework like, you know, we here's what are let's say our API automation solution is going to look like we've set it up. We've got a couple of sample tests in here and then the testers can start contributing. To that test Suite same thing for the UI tests you want to make sure and and I'm a big fan of having way more API tests that you're having UI tests because UI tests are slow and Flaky they're gonna fail so you want to have as few of those as possible, but you also want to make sure that you're reducing the amount of flakiness by having the developers really contribute their expertise to this is what a solid testing framework is going to look like and so then the testers can contribute in a way that's going to write stable tests.

But think that all makes sense one thing I just want to go back to is lately. It kind of put out there that you know, I guess if we learn from our mistakes and what and we're failing quickly, then we we can you know, we actually benefit a lot of from those failures and the only thing is that there's there's a big if in there if we learn and my dad used to say, you know, if a buzzard had a jukebox on his back would hear music from the skies. So I I'm not heard that okay tweaked it a little bit it wasn't on his back.

But anyway I'll just say that you know, what's really important is that we have the the data to actually learn from that and then we can we you know, so we understand why did something fail and we understand when failures failures are recurring, you know, they're similar types of failures or their failures that are happening for an underlying cause that you know, maybe it's not really to the test at all. Maybe it's related to the the network connections that we have in my test environment. Maybe it's related to you know, just the environment set up and you know, how do I learn from that information and fix underlying problems and continuously improve I still remember earlier in my career one of my developers saying well the customers the user's not supposed to do that.

Yes, they are and they're gonna do lots of things. We probably be so much easier without customers, you know. We not do that.

Well, you know, it's interesting because I kind of feel like we've gone from and I don't want to be derogatory in any well way but we were so focused on coverage right just do I have everything covered and this idea of eliminating failure to trying to be really smart about we're not only what we're testing how much we're how we're testing it and and how much we are testing like, for example, we know there's parts of code that rarely get executed maybe less of a focus on that as opposed to the things that are you know, the normative through the track, we always want to make sure that flows is best as possible philosophies like that. So I think you're pointing about the data Sean. Of it isn't just the test result pass or fail.

It's what does it tell us about? What do we learn from running those tests? Right even even if they passed right or do we know from it?

I'd love your thoughts on that. Say some more. yeah, so, um, so like just thinking about you know, it's patterns are hard to detect a lot of times and especially when you're when you're just you know thinking about You know, you're not using data to do that.

So, you know, we see somebody, you know in real life and coincidence become, you know, you know, we apply bigger meaning to it. But when we really think about, you know data and testing it's it's how can we attract data over time? And how can we apply, you know, like test results over time to really understand what the trends are so, you know one one technique is to to apply reason codes to failures and to Trend those over time for instance.

And so then you start to get patterns you understand. Hey, is it a bug and app that causes failure? Is it a flaky test that causes failure?

Is it a environmental issue or network issue and you start to like follow those patterns and then you say Hey, you know because I have so much of my failures or caused by flaky tests. Let's spend a bunch of time working on the flaky test issue or let's spend a bunch of time working on the environmental issue and then all of a sudden, you know, you're passing rates start to increase and you start to spend less time like troubleshooting failures. Really add anything to the value of the top of the application.

Yeah, that's a real critical point. I think the the worst time you can spend is times debugging problems that had nothing to do with problems with the application. That's right.

Chasing down those rabbit trails that yeah you end up not that resulting in that. How do you how do you are there? Some strategies that that can help, you know, you're on the wrong path and not not spending your time effectively.

I think it's important to to think about just the amount of time you are spending on something versus the amount of reward that you're getting for it. If you've got a lot of UI tests that are running in three different browsers or on four different devices and and you are constantly seeing failures day after day after day those tests are not going to provide value because everyone is going to ignore them if they get a message that says hey, you know, 20 of your 50 tests failed today. Nobody's even going to bother to look at them and similarly if you have a test, so here's an example I had this is years ago, but I was trying to test an email that an email was sent and so I went through all of this.

Trouble trying to you know, get into the Gmail application and make sure that the email had been delivered and that test was so flaky and I spent so much time on it. Finally. I realized this is not providing me any value.

I'd be better off just manually checking this myself and then a few years later because apparently the pain of that mistake was not great enough. I tried again and this time I thought oh, well, I'll use the Google API to try and get that email and the same thing happened the the value of it, you know, because sometimes the email was delayed, you know, when you're dealing with a time delay, you're gonna get flakiness. I realize that that's too much time.

I'm spending too much time on this. This should be just left as an annual check. Yeah, it's good Lee.

Oh some say so yeah. I want to insert a phrase here that I think we all are kind of talking about that. It's psycho time.

In fact, I talked a lot about this in my article that was published last week in container Journal called failures a secret of success, but it talks about cycle Time and healthcycle Time Has Changed and software development over the years, you know back in the real olden days Punch Cards it actually I did a few programs and punch cards. So I do myself there, you know how long it took you to take a stack of cards, you know, wait physically in line for your time to use the computer, you know, get those right in run the program send it back all that whole process of of making a change with so long and so expensive that you didn't there have any mistakes in that because you lose so much time and and as time has gone on we moved from that obviously to To prepackaged software where the cycle time to fix the defect that still could be years if you send out software to a to a company like Microsoft Word right up to a company in a prepackaged box and then found a bug and fixed it. How do you get to appear customers?

It could be a long cycle time before the customer buys and updated copy or get some updated copy and little by little we've been reducing cycle time. Where now in many cases cycle time is a matter of minutes, right? I mean for web applications, you can deploy a new change and depending how fast you're you're pipeline is you can get a new change up in a matter of minutes.

That's a world of difference to what it used to be and I think talking about cycle time and related to testing makes a lot of sense because the longer the cycle time the more valuable testing is to you. Yeah, I think that's really great. It just one one thing to add there is that you mentioned how you know that there's there's the you had to have those those cards perfect.

You know, you have to know that they had to be perfect and you know when you don't feel like you have a good Gateway on your tests, you know that you're really gonna catch things at the end. Then the developers have to be really perfect with their code too. Right and and you also so you can extend your cycle time at the beginning and the development phase.

If you don't feel comfortable that you're you're tests are actually catching away what you want. So, you know having that good kind of safety net before Things release is another way to help give the developers confidence so that they can actually experiment more and and not be so worried that they have to have a hundred percent coverage on their code before they release something so Well on that regard, I I am troubled though. I'm coming from the general world and I sit down.

I have sit down with the author of Jaco and one of the things that it's a Java code coverage. So he that is a tool that we use in the Java world like the fact or tool there is no project that doesn't run it and he his words is I regret this because then teens and companies. Equals one number with a goal and then when we start gaming the system and then they are starting to become meaningless.

So yes, I agree with you. Pursuing the skulls at some point. It doesn't matter on the other hand, even if we spend a lot of time in a test sometimes.

We need to decide as a developers. Are we going to Market? Are we going to stop it?

Are we or if it's if it's too complex, maybe this is a reflection of the architecture. Maybe we have to break down this. Class because it's too big there are many concerns.

So the complexity of the test is reflecting something about our projects our libraries or components. So great. Great Point.

Yes, so spending a lot of time on a test. Can even bring us some perspective on what we're going. To expand on that more because I can reflect back on situations where testing certain scenarios there's always the premium environment to simulate the test.

But just the logic to test certain conditions can be extremely. Convoluted and maybe it goes and then thought about it at the time maybe that is part of the software architecture of the design. if anybody anybody run into that before it's similar and and this is something I talk with my developers.

It's like if you're relying for example in throwing exceptions. And verifying the functionality assessed side effect see the problem with would I saying that you are using a side effect to verify something? So this should already tell you that it's we have to sit down and we think this test.

So yeah, I think some of the the modern application architecture patterns are trying to work toward that too right or the whole micro whole idea behind microservice. is that you you take the complexity out of the software and put it into the interconnection of the system of the component to make up the software so that you can Understand comprehend and therefore test an individual component much easier and simpler and it can be owned by someone and managed and operated by someone relatively simply. It doesn't remove the complexity of your hair application.

It moves it outwards but it does do a lot towards allowing you to build these components These Bricks these Lego bricks with you on the build up the application in an isolated Manner and have a higher level of assurance. It does Brooks themselves are what you expect them to be. One thing I'd like to jump in on which is something that that we mentioned a little bit at the beginning is thinking about the the level of risk involved and there are there are some areas of some applications where the risk is just so great that you want to make sure you know, you don't want to ever have a failure.

So I work for at HR and payroll company and we don't ever want to have someone not get paid because oops we release something that you know, what's it quite perfect or another consideration would be security like we don't ever want to accidentally open up a security hole in an API and not have that immediately detected because some malicious users gonna take that and try to get information about somebody's bank account or something like that. So we definitely want to make sure that those those tests that have the greatest risk if they were too fail or the ones that we are making sure to include Yeah, that's a great great Point. That's that's you know, it is really a risk reward trade-off, right?

And because it's you know, you will have defects in your software period every software has defects you will have defects it will have problems so we can spend our time and energy trying to pound them into Oblivion or we can take a look and say well is this important is this important is if a problem occurs here that we don't know what it is. Is that going to cause a serious issue or a minor issue? And so those are important discussions.

Those are all risk analysis discussions reminds me of excel your your point about failure is not an option. Actually the same should be dying in space isn't an option making payroll. We're gonna you know, they meet thousands of mistakes to try to figure out how to get the scrubbers to take the CO2 out of the air, right?

So just your payroll is is like, you know, we are gonna get payroll that's got to happen. So you're optimized for some things that are extremely important the risks are much higher if failure happens there so By dining space. I don't think that's gonna be pleasant for anybody.

So I'd love to hear you. Talk some more Christian about. So when you when you have that clear of emission.

How do you treat? Certain parts of your testing strategy around that kind of a goal versus other testing that you do across the application. Well at Paylocity we have all of our application areas are divided up into teams.

So so one team strategy is going to look very different from another team strategy. So for example that the payroll team they probably have, you know, a lot more API tests unit tests and and a lot more tests, you know, even going up the the chain into production and then something else to consider also is just The the test itself you don't want the test to fail in that it accidentally pays somebody who shouldn't be paid. And and so sometimes when you're dealing with something like payroll, you have to think about that and then you know, we've got other areas of the application that maybe are not as critical but are you know, the shiny new parts of the application?

For example, we've got a community social application that's part of our product offering and and so we want users to make sure that they're having a really good experience there as well. And there's probably going to be a lot more UI related testing associated with that, you know, because you want to make sure if people are uploading a photo that the photo is going to display and that kind of thing. So so you really you have to consider all of the factors and then another thing to consider too is when you're talking about continuous deployment.

You want to put the most important tests in your your CD pipeline, but that doesn't mean you can't test other stuff at other times too. So you can have like a complete sweet, you know, that takes two hours to run that's that's running overnight and and isn't causing anybody's build to fail or anything like that. So You can also give that a consideration where and when are these tests going to run?

You know, that's a great point. I hear so many companies saying that our tests we takes, you know, 18 hours to run so we can never do a ci/cd-5. No, no, let's talk about how we set this up.

Right you have that test running continuously. I would run once a day. You know, I'm a snapshot of what do they stop shot?

And then just anytime a problem comes up, you know reported back and and that's like a like a chaos monkey sort of approach, right? You just having something going off and running and trying to break things continuously and Reporting back into results. But what you do for your ci/cd, my line is more a matter of you know, it's a thing going to be kind of strophic or is it gonna be?

Okay, right and that's the sort of test you want to put there. It's a very different that's what happened. Actually, I'm going to build Fridays.

I just got some chills but he just fails over the weekend. Right? Right.

So you're jumping in I jump I jumped on top of that. Sorry. No, no, it was my bad.

But even even that if they are start failing your cacd pipeline. It's a little bit too late for me because as a developer, you should be even running some set of tests before doing the pull request before pushing because if something fails that it's critical at this point and your test either right the cicd or your full set of Suite of tests. It's going to fail in 24 hours that cost of changing context for the developers.

Sometimes it's huge. So you're all not only doing about service by not running some tests, but you're also paying an extra and over her time as a developer by not doing it. So having different types of tests run at the server at the developers environment then and the cicd and as Kristen said maybe you have more complex during the weekends.

Maybe once a month maybe during the nights. That that having that strategies. It's it's for me.

It's basic. You know Sean. You know tricentus is a very comprehensive and a test from a test Tool Company.

I'd love to get your perspective on this too because You know, we can create environments on developers laptops do pretty darn good job of simulating some of the environment or a lot of the environment and they can develop on the plane and you know, essentially run a good portion of the test. How do you look at that? The whole devops Pipeline and what testing to do where?

Yeah. No, it's a it's a good question. I mean, we you know, we we advocate for the the pyramid the testing pyramid kind of approach but I'll just say that you know, there's when you think about like the git process right you're gonna create your test, you're gonna create it into a branch you're gonna you know, create those tests in your or create those your code and your branch and you commit it back and you commit it back.

You should be running tests on your commits and then and then think about when those when that pull request is merging it back in then you run tests on those and then when you have a release candidate you run bigger suites of tests. So it kind of just Builds on each other at each level some with with, you know, just unit tests some with integration tests some with with the more extensive UI test all can be triggered from from the cicd pipeline and you know, and that's the way I mean two thirds of our Testament customers are running their UI tests in their CI CD. So I think it's it's, you know, it's all kind of relates together.

if I was to just kind of tie back to that earlier conversation, we're having a little bit more about the risk-based testing and and how I think it's Super important. I mean you can't test everything or you shouldn't test everything and so the tough part is you can you can identify, you know where Kristen says. Yes, you know payroll is really important when everybody be paid, it's it's how does how does other companies identify the right spot where I have to spend more time, right and if we can use some sort of you know usage base analytics to understand where the usage is or to understand where are the most likely failures are or the biggest impact of that something that helps us with data to better identify where we should be spending our time on testing.

I think that would be kind of the Holy Grail if you will of, you know, helping us to narrow down what we need to test and when we need to test it rather than just kind of relying on, you know, people's intuition which happens I think a lot and a lot of companies Yeah, I totally agree. You definitely want to think about who who is using your application. And what are they doing?

I was a earlier in my career at Paylocity. I was very surprised just because I didn't know I was in a different department. I didn't realize that we have millions of people clocking in and clocking out to their shifts every day using our mobile application.

And when I found that out, I thought my gosh, we need to start spending some more time testing punches punch clocking in and clocking out making sure that nobody's going to encounter any bugs. And right now we are actually rewriting our mobile application. And so that's where I'm spending most of my time in terms of planning and strategy for testing is we've got to make sure that the bulk of our users who use this product or going to be able to use it correctly with no failures.

Know I can't punch into my shift. what it makes me think of two Kristin is that There's data and then there's data that's important. Right?

So not to minimize people's vacation time that's important, but getting their paycheck right is even more important, right? Yeah, we can fix the we can fix the vacation days if it's wrong much easier, but it's a lot harder if somebody just can't pay their mortgage because we didn't get the payment amount right or the timing was wrong, right? So what your point about The user experience in the UI testing and people entering that data when you think about it's not just testing the app.

It's also making sure we're getting good data. And again the right experience the best experience people to get that data, whether it's uploading a an Excel file or you know, filling out a mobile app or a web page, right? The one of the things we did at Amazon that that I actually I talked about in my book in several things.

I teach my customers is the idea of service to use the the idea is any given piece of software whether it's a microservice a small application. Whatever it is you assign a service here, which is a level of importance of that how critical is the service to the overall application, you know the case of like a e-commerce company like Amazon the cart checkout process was incredibly important because it impacted the bottom line and impact the customers building the buy things. But the service that displays a little icon in the corner to tell you the sale of the day is important but not that important and the one that shows, you know, the you know, the the message of the day for for a new cut for a new customer walking in saying today's specials are so and so, you know that's even less important and the weekly report to management generation service and probably not important partly at all.

Are you believe is negative report? But that's another point. The point is you you prioritizing you a sign of physical label a categorization to each service that it's determines how important that is and you use that information you use it to determine sla's you do different use it to determine Staffing levels.

You use it determine interaction between Services if you have a high priority service that's talking to a low priority service. You better be sure. That's high priority service will continue to function even on both parenting Services family.

But the opposite may not be the case. If you have a low priority service that's depending on a high priority service. So who cares about those down at the high priority Services also, though because the high priority Services down that's what's critical.

so you can you can put processes and systems into place that take this risk and the importance into play and treat things differently and put your resources your risk determination determines where you put your resources and and make that work and that that's something that I've seen used in other companies as well to the very valuable tool to be able to to keep a system not just a piece of code but a system operating You know, I just just you mentioned the Amazon and just remind me just yesterday. I was trying to make a payment a medical payment and I realized the company, you know their businesses medical and that's their critical stuff. But but the same token they want to get paid.

I'm trying to make a payment on my phone and I entered my credit card information and I can't I can't press the save button to save my credit card. There's no save button doesn't work. I tried three times.

It keeps deleting everything. You know, I'm getting frustrated. They just I'm just trying to give them money and I can't give them money.

Right? So I end up having to go to my computer and do it and fortunately there was the the save button but it's like, you know, if it wasn't the healthcare that I was obligated to then, you know, I could have gone to another service and I could have bought it from somewhere else right, but it's just like you got to focus on the things that are are critical to your business. You know, son.

You don't want to give me started on medical software. I had a bad week with that this week. Okay.

I'm not sure where you're going, but it sounds interesting. You know what one of the the brake testing is something that's come up a lot with observability driven design thinking about how do you build in more Telemetry into your application in your architecture? So it's not just testing it under the most favorable conditions.

Right? Things are gonna work what happens when things go to heck and a handbasket right? It's that brake testing I think is almost more important than conditions are right.

So and what kind of brake testing do you do? Go ahead Excel? Yes.

I mean it is interesting that sometimes we have this idea of how our users are going to use our application and we set up our tests in a pristine in laboratory a title of ways. And actually we have the best computers. We are super close to award that the centers everything runs fantastic entrance out that were users are in the middle of Africa and they probably don't have even a sport access.

It's more phones nor have good Network. So the fact that we have this Rich amazing tests and they are passing is irrelevant. Another thing is that it doesn't matter how much test integration a2i contract you need insulin.

Whatever we're defining here. They are all bound by our imagination. I we are defining the tests.

They live in our heads before they live in the code. So and we're flawed as humans. So we also have this issue of Very short-sided and very limited imagination.

So that's why we start thinking about chaos testing like let's Do something and imageable or like we have an imagined or let's do this like what happens if I pull the block on this? Are we going to and then? The test should all not only help us to say you're up or you're down or this is not working as a specter.

But also give us an idea of this test. Like if we're building graciously because as we all agree, we are going to fail like we are going to fail at some point. Something is going to be down.

Are we reacting the right way? So this kind of very dramatic tests where we kill randomly things help us not only to verify that our flow. It's a respect.

But also, how do we react the chaos monkey style of testing. I completely agree. It's something I I emphasize a lot.

You're absolutely right is what matters is the customers operational environment and that operational environment is where their experiencing the software and you probably don't know what that is. No matter what that where you want industry. You're in where your software is.

It doesn't matter. You don't know what the customers operating environment is and you know, whether it's yeah, there's cell phone and on a 5G Network. That's probably pretty good but maybe not perfect or if it's a you know a a someone in World country that's the internet connection for a couple of minutes and then it goes away or whatever the issues they run into this or you know, imagine, you know, chaos testing in the country like the Ukraine, right?

Are infrastructure comes and goes appears what that means to software infrastructure. It's the operational environment of the customers really critical and very a lot fewer pieces of software than should take that. I say anything because the end of our time here.

Let's do this Sean. You want to kind of wrap up for us? I'd love to get your thoughts on sort of break testing and conditional the environmental factors of how do we handle failure and handling gracefully?

Yeah. I'll just say that and maybe not directly answering your question here. But the one thing that I've noticed is that there's a lot of testers who have developed it kind of as an art, you know, they believe that you know, you really do a good great job of putting their mind and the user's perspective and and they they think about the way somebody would use the software and but again it's an art and I think it takes it takes a lot more skill and and we need to we need to come up with ways in our software and and the processes that we using around our testing so that we we're not relying on individuals to be you know, especially thoughtful and they're testing that we can give them guidance that helps them, you know understand where they need to think about testing.

So, you know so that they are thinking not just about the happy paths, but the negative has and the you know, the things that can happen in, you know, if everything's not going it should go and you know, so it's really comes down to like, how do we how do we help those those people and those teams, you know broaden their their thinking and and approach testing from a more analytical perspective as well as the art side of it. Fantastic, thank you Sean. Well, let's wrap up with this Christian.

Tell us the name of your book again where people can find it. Yes. It is called the complete software tester and it is available on Amazon both as a paperback edition and a Kindle edition.

Overcoming it complexity available on Amazon Kindle on paperback and it will be available sometime next year in and audio version. There you go. You can listen while you're driving.

You're flying on that airplane. Yeah. All right.

Well, thank you. It's for a great panelist today and thank you for everybody tuning in to our episode on testing how much testing is enough? Well, if you versus quantity, it's been a great conversation.

Thank you to our newest panelists and also our return analysts and of course a big thank you. Shout out to the tricentious team love working with you putting this together and thank you for sponsoring the show. So be sure and check out another episode either upcoming on Textron TV.

com for upcoming round tables that will happen on various topics around devops. So appreciate you all joining us and tune in again soon. We'll see you all.