Step Up Your Game with TestOps – DevOps Unbound Roundtable

Hey everyone, welcome to a live Roundtable edition of devops Unbound. Devops Unbound for those who may not be familiar is a bi-weekly or is it semi weekly? I always get those two confused.

It happens every other week video series where we talk about relevant topics in devops. We've been doing devops and bound. Oh for almost two years now.

And one of my favorite Parts about devops I'm down is about once a month. We do a live version of devops and down what we call our devops on Down Round Table. And why do I love it?

It's because besides having the great guests that we always have on our devops on battles. We have you our studio audience out here with us participating and when these things work, right? They're golden because as good as our panel members are.

our Studio or our alignment studio audience, but our live internet audience. They really drive the discussion. they really keep us on our toes.

We really understand what it is. You want to hear about what it is. We want to do to do this.

So, you know, I always like to make sure the chats working for everyone. So I'm gonna ask everyone if you want to type in it while you're here in chat say hello from wherever you're from because that's another thing. I've Loved this ever since I've been involved in the internet since 1996, which is it freaks me out that it's truly is worldwide, right?

It's not down the block or down the hall or down in the city or even in your state. It's from all over so Mitchell and I are both in Tech strong headquarters in Boca Raton, Florida today. I'm in studio way Mitchell's in his office here at Tech strong, but sign in where you're from Raleigh right, North Carolina, Nashville, Tennessee, love it.

Los Angeles excellent, excellent. Excellent. Everyone ourseling Minnesota.

I've never been there, but that sounds good, Portland, Maine. A lot of American cities in here where my friends in India are in Europe. Come on, I know some of whistlementers.

Well, that's Mark. He's here with us sign you wherever your farm. Let's just make sure that channel here.

Yeah, and and thank you for joining. Speaking of chats as I was saying what really makes this interesting is your comments. So as we're speaking if you have a question or comment or a thought or you'd like to throw something out at us that you'd like to see us explore.

You can put it in chat. We'll see it. We may answer you and wet sweat in.

Straight on chat or we may do it. You know raise it to the panel and do would be a video someone here from West Boca. I love it.

Let's spoke High School right older son went there. Welcome. So use the chat for that also though, if you have a particular question or comment that you do want to raise to the whole panel.

If you look in your Communications window to your right, it says chat up top and then public presenter. It's such a go one level over to Q&A and you will find The Q&A panel or window in there and when you put it in there, I know you want me to ask it of the panel. So I will elevate it.

To the panels for their comment on the video session. So use QA for that use chat as well. Either one is really going to work depending but in chat, you may get a chat back is the bottom line.

I also want to remind people that towards the end of today's event. We're gonna be announcing for winners of an Amazon gift card. We give them out at every event.

And we'll announce those winners at the end today one other thing. These are life and anytime you doing live on the old over the internet. Stuff happens right you may see panel members pop in and out.

Maybe they've got a bad connection or they're coming in and out. I see Mark corned beef minor dropped here. Orange County, New York, I love Orange County, New York.

Been up there many times doubling Island. We were in Dublin last week for the Linux Foundation event. Anyway, let me introduce you well before I introduce you to our extreme panel.

I want to just give a quick shout out to our good friends and tricentes and I mentioned we've been doing devops on back for two years. We've only had one sponsor in two years, which I sent this they have stood behind this from the beginning to the middle all the way through to today and Beyond. com.

Now, let's get to our panels. So I'm going to start off our panel with someone who I've had the pleasure of knowing for. Look, I'm doing devops calm me, you know, nine years.

I probably know most of that time. He's from he's well, he's he's important. Oh, very harder now my friend Mark corned beef.

Hey, Mark welcome. Introduce yourself. Thank you.

Okay, I'm basically Mark hornbeak. I do consult. I have a little Boutique consulting firm called engineering devops Consulting.

And what we do is Consulting with large Enterprises on devops Transformations also do teaching and training and devops topics as well as ride content and things like that from my little office in Puerto Vallarta. I'm happy to be here. I've spent a lot of my career.

Thinking about and working on the testing aspects of devops. So continuous quality assurance. That's also another good name for it.

I think it's also an esteemed author check out Mark's book on engineering devops. So that is also one Awards of IEEE for test engineering and you're being humble mark That's not like you but welcome proud and happy to have you here. next up I want to introduce you to harid Patel a read if you wouldn't mind introducing yourself.

That's my thank you. I'm harid Patel. I'm based in New York currently in Portland.

I'm a product guy. I love creating products and building things that help solve engineering software problems in the world that we live today. Every company is a software company.

So I've been fortunate enough to build tons of products in the past couple years, but looking forward to help in the devops world and the desktops world. So excited to be here. Thank you Karina.

I don't know if I caught the company though or would not oh, yes. I'm actually work for it. I sent this right now.

Okay, actually, thank you. Next up. I want to introduce you to Jennifer Velazquez.

Hi everyone. I'm Jennifer alaskas. I am the IBM America's devops leader.

I work with organizations that have Enterprise systems to basically devops them and I have the luxury to have that job because it lets me work with everyone and follow everyone's Journey testing is a passion of mine because I feel like if he automate things that don't have things tested, it's your risk with no insurance policy. So excited about the topic today. excellent and welcome next up.

I want to end this I was saving this name for Alaska. They do my best we have summon. Copper in corporate so they help me.

Yeah close enough. Hi. Hello everyone.

I'm Suman gopinath. apparently the chief for a set of products within ibmz systems. I don't like to say devops products because devops is not about products.

It's it's about everything else. I would say automation. Thought of things and a big part of my job is to help customers understand that devops is not about technology but it's about all of your Enterprises systems.

It doesn't matter if it's on ibmz systems it can it can all go into your devops platform second part of my job is because I do have all of these products is to make sure that we have continuous everything within each of these products because and otherwise you're just preaching Absolutely someone thank you. Thank you for joining us and now last but certainly not least anchoring the panel as he always does for me my co-host here on devops Unbound as well as our CTO or text strong and a principal analyst a text dog research. Mitch Ashley.

Hey Mitchell welcome. Welcome, welcome panel. Welcome Allen glad to be here anxious to get to our topic.

So let's jump in. Yeah, you know what? Let's get to our topic.

I realized a lot of talking we haven't done it. So look, for those of you who registered to attend today's event. The title is step up your game with test stops.

right and you know test stop Heights testings focus on operational effectiveness and continuous Improvement, of course devops workflow Pipelines. Blah blah blah blah you can all read that on the web page, but I'm here in the format. We're here to find out what exactly is test stops.

How can it help you? And how could you do it better that's gonna be the focus today of our Discussion it always helps us to frame who the audience is. If I want to know.

Right. If you work in test stops, please go in the chat here. Let us well not in test stop.

So if you're working in testing, you're a tester you're working in testing. Let's hear from you. If you work in testing going to chat and let us know you work investing.

If you don't work in testing, you're going to chat. Let us know you have nothing to do with testing either. So he's a good night a good.

Way to see you know, who's in the audience while you're all doing that though Mark. You're the esteem test out testing the expert here. What is test stops mean to you?

What? How would you define desktops to our audience? Well, I'm not gonna give you a succinct definition but I was simply say that you know test stops to me is very similar to what I call continuous quality assurance.

In other words. You're it's the tight integration of strategically looking at, you know, the test skills processes Tools in order to accelerate continues delivery safely. So it's very much related to The Continuous delivery movement devops.

But if you look at devops and break it down every stage in a value stream has bottlenecks that have related to testing so you got to get testing right? I often say, you know culture is the door to devops but continuous testing is the primary key to that door. So to me, it's the key to continuous delivery.

It's the primary key. There are lots of other keys, but that one's essential. Good good, answer Mark.

You know, I love what Mitchell wrote here in chat, right as an end user. He tests every software he touches so he's definitely a tester. I wasn't really maybe the testing I was looking for but Hurry, you work for you work for trisanus worldwide leader in continuous testing marks since that's all about continuous testing.

What do you think about test stops? I feel testops is slightly similar to devops but very different as well. It's about the ability of running automated testing at scale in my mind and the collaboration with building software testing and helping testers test the software and also the operationalization of deploying and managing all of these different pieces that testers have in their tool sets.

But again in all it's about continuous testing and helping make sure that you're enabling the ability to run complex test automation Suites much more seamlessly and figuring out how where the model X are. excellent Jennifer Simone, I wonder if you have any additional thoughts on that. I know.

sumont I'm Rhode I always tell any organization I'm working with it's the most complex piece because when you think about your software delivery, it has to have a Target to go to when we think about testing. You've got the test where you've got the test you've got the application got the infrastructure. You got all of the data to support it.

So holistically, that's my insurance policy and I like it attached to everything I do and if I don't have that part automated then ultimately I haven't hit the core element that a cicd pipeline is going to speed up and help us with our agility. So Mom, I I agree. Right that that part of what Jen said, right I come from writing business applications.

That's what I did for many years before I got into Automation and everything else right and any large business transformation any transformation existing app anything if you don't have a good reliable test bucket to fall back on it's it's you're just hitting or missing things right? But but test stops to me is is really to keep it simple is to consider any part of test. As just like you would do with the development of code.

I mean everybody kind of emphasizes on the fact that it's the application code. It's your product and everything that needs to go through their life cycle, but Test your tests. Make sure your tests go through the same life cycle.

Make sure your test go through the same rashbody that overall governance process is really to me desktops. Yeah, yeah, I don't disagree there either, you know for me to understand that she got it kind of look. At the evolution of testing for me, right Mitchell and I co-founded a company 20 years ago.

He was VP of engineering and CTO. I was I was a little bit everything sales marketing Biz Dev product it we did a lot of things. You know how start of life is.

but we outsourced a lot of testing and we did a lot of testing in-house but this before there was devops. It was it was. It was just some silly thing.

We did at the end of our engineering cycle that just made us miss deadlines, you know in many ways there wasn't a rhyme or there wasn't a good enough Rhyme or Reason to it. One of the great things about companies like tricentes for instance is they brought this concept of continuous testing automated testing. I mean Mark Hornby, he's been a test engineer for probably longer than many of our studio audience has been alive, right?

That the move to continuous testing the move to automated testing really brought some rhyme and reason. to testing So to me that was sort of the the era the dawn of the era of you know devops that's things. when I think about test stops though, I think about taking that to the next level think about really making a lockdown process methodology in place That allows testing to keep up with the speed of devops with the speed of business today.

And to me that's the difference between let's say continuous testing and test stops. Yep, that's what do you think? There's a very interesting question though.

And I know it's not directly related. But I I find that a very very relevant question on why is testing the most underfunded area for companies, right? It's up in our QA section.

Yeah, I and and I think it kind of a little bit ties back to what you talked about Alan that there is a progression. So there is you start and then you go take it to the next level and so on. and there is also this notion of testing being.

Oh you created the testis move on. It's it's a it's a wand it's a fix. No, it is continuous investment.

Sometimes you have applications products that may not have any kind of automation. So you really need to invest in that and that's gonna pay off in the longer run. And so I think more about the fact that it's I wouldn't say it's underfunded.

I think the expectation behind it's not a one-time investment versus it's a continuous part of everything else. I think that's sort of my my take on. This comment about why is testing underfunded?

It's It's about having the right expectations and putting in time more than let's say I've in for this it I've invested this amount for this period it's it's not just that it's a continuous investment of time. And I think that's where the expectation is where we have to set for customers to you know, get going I think the other aspect of that though is because it's not a deliverable most want to funnel the funds if you will into what is actually going to be a deliverable at the end of the day and because it is a supporting role in the quality element. It's just seemed differently in terms of the allocation and the like Fair anyone else on on you know So I'm gonna come at it from the other side of the table.

I never ran a testing team. I never ran a development team, but I was an executive that had a fund at that thing or approved the budget for testing. And I know from an engineering perspective.

It's much like security you ever talk to a security guy raises his hand and says they give us too much budget. I think I don't know what else to buy they give us. No every security person I've ever met in the world.

Bones and groans that security is underfunded and that's why they can't do a better job. It's the same thing with testing. No one raises their hand and says, geez I have extra dollars in my testing budget because we didn't spend it.

Right as an executive sending at the table. I think we give too much money to testing a lot of times. Right true.

Yeah, prove me wrong. You know hurry. I'm wondering you're tricentes.

You have a different view. What do you think about that? No, I think that's true.

I think like every organization and team always requires more funding and be able to kind of grow and from a testing perspective as well. I feel there are these instances where we aren't able to highlight problems with our tools and our Integrations and collaboration. Mostly it's about process and that's where people sometimes confused between test management and desktops.

We're test management is primarily about problems with defining testing organizing it all of that when you look at tests, we never talk about technology problems like how quickly can we test this little piece of code and make sure it's providing the proper risk coverage. Are we focusing on and comparing test coverage to directly code coverage? So it's like it's figuring out.

What is the problems and surfacing it, which I feel sometimes a lot of teams struggle with. Actually, I don't know if you guys can hear me, but I had a comment sure. Yeah, I think one of the problems is that.

I think a lot of Executives see testing as a non value added activity. Yeah and don't realize that you know, it's at the root of almost all of the delays or a half of the delays and a value stream typically and there's so much return on investment if you can accelerate testing, so it's always any automation project. I've been involved in with testing has always paid back very very high Roi but it's not always apparent to the executives why it would be because testing in itself is kind of a non-value edit activity, right you get to test for if you get to test verdicts without having any You know expense you would do it, but don't you think the area of devops and that the idea that organizations are now on devops.

Journeys is bringing light to that topic IE how much time is being spent in testing because previously I saw organization after organizations say I've got you know, here's our deadline. We're just going to squeeze testing we're gonna squeeze testing there was no recording of that. There was no reporting or other because they didn't have value streams like they do now.

What are the issue I think is we're not bringing forward. The value that testing is delivering to often that's going to just recorded in terms of how much testing God done or what percentage of our quality Gates. We did we meet but we're not bringing forward and I think it's, you know part of our own fault to not not bring forward the real value which is shrinking the time which is saving money, which is real dollars.

And you know, it's the time element. I think that is where the ROI is and so what we really need to teach people and do is To get to the result of testing quickly. It's all about testing verdicts frankly.

But testing is irrelevant if you don't resolve it to a verdict, so we should be teaching people and really thinking about how do we get to vertex quickly at every stage of the pipeline? It's all about verdicts the result of testing. If you don't get a verdict, there's no point in testing.

So how do we accelerate getting to verdicts? That's what I believe, you know part Mark. I think when things it's really we're kind of hitting on a different areas is that I'm gonna paste our link to some free research that folks can get it's the role of testing in devops.

It's actually where sponsored by Tri Sanchez. That's not one putting it out there but it's a very recent study. There's some interesting things about it and when we start talking about devops and doing automation, you know, we can talk about the testing that we're doing in the quality improvements that it's making in the outcomes.

The other thing that we could I think we can Hance maybe we don't enough is how automating testing it both increases coverage and increases speed of delivery of software right now. We don't we don't necessarily want to automate everything but we testing doesn't have to be a bottleneck quote unquote. Like I think many of us probably felt at times either.

We are feeling that way or or we thought dust was that way but I think in a devops world, it's a much better environment to highlight not just what we're testing but how it's improving getting software out. in really good software right one thing though. I know so metrics is as a rabbit hole and I let's I for now.

I do want to go in there right because you can Spend hours doing metrics that have no meaning and that's a different. We need to get the right metrics, but personally for from from our own point one way, right and it's not the only way but one way. to get management look at testing one of the things we did is to have an actual okr which was based on that.

We're going to bring down the number of Cases and and when we use when I say cases that's something a customer would raise after a product goes life. Right and we will not have or we will have only this many but indirectly for you not to have that many you need to have good testing but you when you represent that. That sort of becomes.

Oh, yeah. That's that's a valid point rather than I say. I want to build more test Automation and I want in you know investment for it, but I kind of related to what does that mean?

Agreed, you know? I kind of started. I feel like I kind of started this thread but I when I said is an executive I feel like you spent more than enough fantastic, right?

And so and then Mark responded saying look you got to be able to show value in a lot of times very hard to show the value and testing. I have a friend Andy Alice. He was Chief CSO it Ackerman for 20 years, you know and you and these thing is choose metrics that matter, which is I think Superman very similar to what you're saying right choose metrics that matter the problem is Matter to who when and what?

Right, and and I may choose metrics that matter to me as a tester that really don't manage matter to a manager or developer or security person or or what have you. You know, I think that is an area where users kind of rely upon companies like I sent this or IBM's to say hey, what are the metrics that matter? What are the metrics that matter mark look you doing this a long time.

What are the metrics that matter? One of the things in that area that I'm always talking with clients about on their journey is what matters today may not be what matters tomorrow so that it is when we think of metrics you have to continuously improve where you see or measuring your value. So it may be the number of defects or issues that are raised with a product.

Especially if it's a new product, but maybe we're we've trended down and now we're in some not only area. So now maybe it's what resources were consuming and how much you know, how long is it that we're spending in test in relationship to the other things? all of those things are going to be based off of what that organization is trying to improve when we have so many metrics that it becomes noise.

We negate needing them. We want to keep that succinct where it's value at. Good coming in and out.

I'm sorry Jennifer. I think you're right Mark. If you could hear us to respond, what do you think of the metrics that matter?

Well, I don't want to do an advertisement because I'm doing it webinar on this next week on metrics the matter and believe it or not. But you know clearly the metrics need to be taught. There are so many possible metrics.

They need to be tied to your business strategy, you know, the strategies are generally going to be in six categories of benefits. So there's going to be like agility qualities efficiency stability security and satisfaction. So what what is the core of your strategy and make sure that the metrics align to that strategy you can't you can possibly measure everything but very few people do but you want to really zoom in on what is it that really matters quite often.

It's agility right lead time frequency the things that really matter that are going to if you focus on that, we'll get you the other things anyway, but you know, you may have a different Focus depending on what your business strategy is. So make sure that you have metrics that align across the organization that all the levels, you know, according to whatever your strategy is, so And that does require that you tie those different parts of the organization together in some kind of, you know, Common data Lake and make sure that you can then serve that up to the different stakeholders in a way that they can action it. according to the strategy Sure.

And yeah, I want to add something on there as well. One of the things we talked about from metrics was I think it's a time to start talking about testing slightly differently where typically when we talk about our test results. We always presented as a binary Choice like yes or no Pastor failed 87% of tests fast doesn't mean anything that needs to be some talk about confidence levels.

Like are we confident in some of the tests that we're running and I think some of that discussion changes the talk about how things are successful and how we can operationalize overall testing. So that could be an additional metric we add on the list actually. Yeah, I mean because 87% of times past 13% fail.

That's what that means. Right and you know 13% is not as big as 87 but you have 13% failures out there. You have a really or I'm reminded when you know Dragon Dictation for this came out years ago, and they said they were 90% 87% effective and it sucked.

I mean, we're all honesty having that we type one out of every 10 words was not fun. I'd rather just type it already. So you're a hundred percent, right?

Everything you need you need some context around that. Suman you you had jumped in here as well. The other thing though, I'll you know, I've heard this across the board and testing.

it security is You know the the people doing the work have one set of metrics they use. Right because they get it they they're cool. They know it but now we got a dumb it down for those idiots up above us, right the managers and the board and the sea level folks.

They don't get it. Yeah. I don't know how that they got those jobs, but we got a dumb it down and we got to use a different set of metrics that make sense to them.

Is that true? Number one? Not that they're dumb.

I don't think they're done. Translate Right and that often times that's the hardest part is you become managers in your career. You find that that's one of the hardest things you got to do is be that translation layer between the people do in the work and the people you report to Right, I think.

I think that forms the key because and and you need to sort of think about how I personally to me. I've spent a lot of time thinking about why should this matter and what is the ticket? I have like customer satisfaction is a great tool to use right, but it may not it might it may not be A greater to though it should be then sometimes sales.

So we have to use the right tool to drive the metric forward. That's that's really something that I I would suggest because there are always a set of okay ours that an organization will all will measure and if you can try you try your metrics to those okrs, and that that needn't be The job of I don't know a developer or it should come out from the pipeline itself. It's not a manual task of saying.

Oh this is what it is. But if I can tie down my metrics to say but this is a measure of this which is my okr. I think that's that's a good way to represent it.

I on the metrics though. One of the and I think probably in Mark's session that that's probably going to be a great insight into that. One of the issues I have with metrics is and if somebody has to spend a lot of time collecting that metrics then I don't know if that metrics is even worth, you know measuring right if it's automated it has it has to be automated and it cannot be subjective.

it so that's Yeah. Sure. Yes, we hear you Mark.

I'm sorry. Oh, sorry. I don't want to stomp over yours.

Go ahead. No, no. No, it's not.

I was just I'm I'm sure your webinar is gonna address some of that right the what metric could be subjective. But the way you look at a metric just cannot be subjective then if you lose the purpose of the metric. Yeah, I would only comment that remember that metrics are not a goal metrics are a measurement towards achieving a goal.

Yeah, and honestly a lot of folks what I noticed as they spend a lot of time just kind of creating the you know, the glue wear the technology that to make the metrics and then to make them visible but it's equally important to make invisible. What do you do with them when you got them, right? So I'm actually spending a lot of time with the client right now explaining like okay, if you get this score and this metric here are some things you can do to make improvements towards whatever their particular goals are.

It's the job isn't done when you just present the metric and that's true for testing along with other dimensions, you know that we care about but you know, it's all about how do you use the metrics to make improvements? agreed Agreed I think one of the other elements when you think of the topic that you brought out up Alan in terms of the layers in the organization and what someone may be focused on there are going to be certain metrics that I would say are not those that are tracked at the product level because they are for my role and responsibility to the team. But that's another Trend that I'm seeing changing in that the team is now seeing and having a common goal with the product instead of I have my Silo my piece and as soon as I deliver it, that's all I really care about.

If we don't meet the objectives of the product that we're delivering. We've all failed. And that's where we start seeing a more holistic view in terms of the metric and Trend and what action we take.

Because that becomes important to our subject area and we haven't had that in the past at least not with the organizations I've been working with. Fair enough. Hey, I I got to just interject real quick.

I want to announce the gift card winners Lizzie a victor e Melissa a and Jorge D. Our team will be in touch with you. You're a lucky winners today.

Enjoy it. By something nice on Amazon there. You know Steve out in the audience wrote in something about here that Google acts that way.

I don't necessarily believe that proof should determine if you can pull back or not. Right, we're having a conversation and yeah, so part of a big conversation there. Testing less sometimes to share with Sharon Mitch.

Yeah, there's a school of thought I think it's small but growing which is the more you increase velocity testing. Everything becomes less important because you can respond quickly. To fixing things and I'll give you an example that I think might be about example.

I might want to get an MVP product out to Market and and having it perfect run perfectly is less important or near perfectly. There's less important than getting customer feedback. I want experiment.

I need to know what people think I want to know how they're gonna use it. If there's a few errors and bugs along the way I'll take it unless it's not, you know, not so bad. It's not gonna work.

That's a pretty clear example, but I think there there's a growing school of thought of delivering it higher velocities gives you some flexibility on testing. Not everybody believes that but I think there's something to it. So that my comment was on.

You cannot stop following rules because you know, your accident rate is zero, right? So that's why there was a Google comment on the fact that that's the whole I think that we're all good, right? but I think that gets back into what are the what are the goals of the product that you're delivering if that is in the leniency of what you can do then maybe a smoke test suffices, but if you're brand is at risk.

We can't we can't have that risk. We've got to have the amount of testing now. What's the right amount especially as things continue to get growing complexity and the amount of tests.

We have continued to grow because we have now engaged in automated testing. That's something you've got a triage and figure out what's the right amount. It's an art to some extent.

Hopefully if you can go that fast though from an agility standpoint. Maybe you can get all of your tests we did. But that's I'd say Case by case that we have to look at.

Yeah, Mark you made a comment in chat here but defects are not a good measure of quality usage is what's important, right? So look, we're the title of this stuff up you game with us with desktops. Why why is defects not as important as as usage?

Simple analogy you have a line of code. I can think of a million reasons. It doesn't do what I wanted to do.

I have a line of code with a lot of system with a lot of bugs, but it does everything I wanted to do. It's not about defects. It's about usage.

Does it do what I what your customers wanted to do? You can find a million defects. How many of them are relevant to the usage so don't test just to find defects test strategically to find out whether the product meets its requirements from a user to you, but that kind of change our whole metric.

A whole metrics argument on its side right? So we really don't care about how many defects we want to know. Is this thing being used and of course until I release it to the public?

I may not really understand usage that well, right. So how does that work in a devops environment? That's why we do continuous delivery.

We will have a pretty clear idea of how things are being used and whether it's usable as we do the continuous deliveries, you don't wait for the Big Bang and then find out that you had the wrong idea of usage. So this is where devops I think fundamentally adds so much value to the world. Is that you know, you you can get fast feedback and of the actual product very quickly without having to worry about doing a whole lot of useless tests.

That really don't matter any way to find defects that don't matter. you know, I'm a strong proponent of of saying a lot of their coverage metrics are wrong, right they're focused on code defects. Nobody cares about the code.

They care about the product and the service. So it's a security person though. I think it's important right because you may not care about that code until that defect.

In the field your bank account money or your coinbase account or what have you? Right. So I think I think defects do matter.

I I really think as Steve wrote in his thing here. It's a question of risk management. Right defect in my code gonna what risks does that expose me?

Just security is a different category. It used to make a mistake of thinking security as a subset of quality, but it's not it's a completely different level of Consciousness, right? You have to think about the fact that really you No, you know what?

It's not any I mean. It's not a level of Consciousness either. It has to be built into everything we're doing today.

Aren't you know what? I meant it good. What I'm trying to say is the month the mindset for a security is a little different than the mindset for Quality because you know have to think about you know, what is a value to protect for the or even beyond the life of product potentially.

That's a different mindset. So certainly for security. There's no such thing as a hundred percent security just as there's no such thing as a hundred percent quality, but if we're talking quality, you know defects is not a good measure if you're talking security, you know, it's all about just mitigating risks.

So you're going to do is right. It's not necessarily mitigating. It's managing it.

You know, we got a question here from Melissa. How can we make the minds and it's right in line with what we're talking about Mark. How can we make the mindset change with leadership when they want to do regression tests with the same stuff all the time for example, and and when might not be relevant anymore or won't help identify real failures.

It I have a question or the other way. Is that is that really a leadership thing? Right?

I mean on what to test? Because your regression test bucket should be something that again goes through your continuous enhancement process. And why do you go to what's the role of leadership she leadership in here telling you?

Oh, I need more regression testing. No, I you know my my thought is leadership wants to know. Hey, what's the quality here?

What's the usefulness? Yeah. I agree.

I don't know how far in the weeds they get. I think it also goes back to what I was mentioning earlier is the confidence level leadership. Maybe might just be looking for a yes or no, everything faster failed or I'd be successful but I think it goes back to confidence level in our testing if we feel as testers and just managers that all the confidence level in what we just tested with our selective regression build.

Then we should be supportive of that. So I think it's a discussion with leadership to change that thought process in my opinion. I think it comes down to risk.

I mean how much risk am I willing to take on as the executive responsible for this portfolio this product that's whatever. Very their name is is tied to it and depending on what kind of thing didn't get tested or didn't get caught from a security standpoint. That follows them forever.

So that's where you have those that are risk adverse and they're probably gonna push the regression full regression buckets others that are a little bit more entrepreneur. They're gonna go after and say if I have those core use cases or the like Done. Then I'm gonna take the risk on those other elements that we've done.

Well always in the past. I don't think those need to be tested. I mean timing the audience I think said that well Jennifer too real simply use risk to focus your testing.

Well listen in the audience had another question or another statement which was to avoid the liaison and deployed this classic, right? Why did we decide to do this testing on that testing or making decision to avoid delays on deployment? Right?

And that's always been the lament of testers. Right, they the developers the bosses the board. They want to get, you know back to the Phoenix projects.

They want to get the software out. And they won't give us enough time to do the right testing. We want to avoid delays on deployment in a world of continuous testing devops and feedback.

Is that still? true No one um, there's a follow-on like which says better testing will reduce the risk and I I noticed what you said though. You used the word right testing and I think that's a better word to tell that the right testing and going back to what you should test is.

Yes, it should be on use cases, but that needed necessarily be. Measured by the most number of uses alone. It also be it should also be What is the impact of that right?

So it could be features. That get get that get turned on very rarely, but it probably impacts an entire. I don't know stock market.

So it what is the impact of that functionality? Also adds on and I think I would sort of agree with you Ellen and what you said? Yeah.

Anybody else on that right testing versus the leg deployment? I'm gonna join in I'm gonna join in on this and I'll use an analogy in the in the SRE world and other part of devops. A lot of discussion about is five nines the right measure this is reliability up time the right measure.

Well if it's the wrong software, it doesn't matter, right? Yeah five nines, but if it's not doing what it's supposed to do. It doesn't matter same thing in testing right?

It's not about Precision don't don't apply an engineering discipline to it to drive it down to you know finer and fine. You're fine your levels of measured quality per se it's it's are we delivering the right functionality at the right time at an acceptable level of quality risk Etc. It's all about what's acceptable because nobody has the resources to do it perfectly and it can't be done perfectly.

There's no such thing. Of software that we're talking about. It's all going to vary in terms of what those answers are to all of those questions.

Yeah, the moon land Rover's a little different than the software running on my way and I would also Say when I go to do something with my bank account, it better not zero out. I mean that's to me is you I'm just saying so that kind of testing. I always want to see in a system.

Sure enough. I don't see Zero out either. It's questionable is almost asking it rhetorically, I think that Anything we do that delays deployment.

has to be justified there has to be a really good justification if it's going to mean I'm releasing software later. I'm releasing functionality later. I'm not moving as fast as we can.

I'm not staying we should never do that. I'm just saying it's gonna be a damn good reason. But I think you have to look at what what is that length of period that we're talking about?

People to run your complete Pipeline and your test in an hour two hours a day. How is that impacting the end user community? Did you did you lose market share?

Did you lose significance by having it at the end of the day versus in the hour? That's all going to come back to what was the product in question to begin with the Automation in bringing automation to our delivery to drive efficiency is radically changing the game. And I think it from whether we're talking the testing the security whatever they are all options.

We got to get the right balance in so that we get that efficiency in agility in the delivery. But we balance it with the risk that we're willing to take. and argument I may have with you on that Alan on the real on the delaying of for release is Again, it's not to do with.

The faster time but it's to do with if I have a new product or a new functionality. It might be better to go on a beta mode to get faster feedback before I do a full release out and in that perspective, I think delaying a real release so that the beta tells suppose a beta tells, you know, you're not there yet because that's not What is useful to a customer? the delaying release to get more feedback from that Niche set depending on the product where it is that I think might be a valid reason and of course all the beta Cycles are still going through your automated pipeline process.

Exactly. That's the appointment of the beta is exactly I mean it I understand your classifying it so that we have a limited audience in terms of the visibility. But again what you're doing with the bait, maybe you set different profiles for your automation as a result of it being a beta and not something that's formally released.

actually guys, this is why I at the beginning of this talk, you know, I was saying that we have to approach this strategically and what is important make sure you get The things tested before you deploy that are really critical to whatever your strategy is, but then don't forget that there are some. Strategies also for deployment that allow you to do testing in deployment or at least as part of the deployment process. So, you know, probably the best practices, you know feature flag deployments where you can turn a feature on our offer select group of customers to test in production.

You can also look at Canary testing. That's a little more sneaky. And blue green and all that but you know some of those tests you you can allow strategically deciding right deciding which test are you going to do in deployment?

Don't slow down the pipeline allow it to go ahead as fast as possible. Just make sure you don't, you know start with something that you know is already flawed. So, you know concentrate your tests accordingly throughout the Pipelines.

So that means that you really need to know what tests you have available for every transfer transition of those pipelines and organize your tests accordingly. Depending on what you know, what's the likely risk of that particular change going through to deployment? Require some intelligence and strategy.

Okay. Well, if you like if you like everything that everyone's saying goes back to testing is an art it's not something that is easily take it's not exactly like, oh we make this change go test this one piece out. You have to think about so many different scenarios the business impact the customer impact the user impact the security impact so figuring out which tests we do is definitely an art So are you people go ahead Mark?

I'm sorry. I've got to take exception of that. Me too.

I was going with what I'm glad you did first dad. Why was that I think one of the most dissers to the entire industry in the world was the art of software testing the seminal book by Glenn Fred Meyers back in the 70s where he had some actions that said, it's a conflict of Consciousness for a developer to test his own code instead. He should have been saying no we need to make sure developers know how to test and equip them with the means to do it.

And that's an engineering problem. Not a problem of Art in my opinion right here, but my issue is when you tell me it's an art you're saying it's an art not a science. And if it's not a science.

It'll never be continuous and automated and done enough to keep up with. the science of devops if you will or signs of software development. I think what you're citing hurry or many factors that we need to put into some sort of formulaic.

You know equation. that allow us to dial in No, I like 100% agree with that and I say that is an art because we're continually evolving how we test and we're learning new things which will help us build that science. So it's a combination of both but at the crust of it you will always have new things that are adding to your automation to your operationalization making it complicated.

I'm gonna come to your defense hurry and add to this. I think another way to interpret what you said about it being artists. You can't take the human element out of testing.

You can't just automate it. You can't apply AI hereistics to it and say that's it. That's all we need because there's humans on the other side using it and they're gonna do stuff that we never anticipate and we need the creativity and The Thinking Beyond, you know, Engineering Process and discipline.

What else what do we need to think about that? We aren't and that's where the humanists come in and also think of new ways. Proving it.

So I think that it's that that's the artistic part of it is the creative and the expression that we as people bring to it as well as what technology can do repetitively extremely well for us. Did you any man you okay? You feel good?

I got what I'll take what Scott is saying it's both an art and a science. Yeah. Hey guys.

I just got this that boys in my airpods are telling me they're out of juice. So I may lose you here. We're about at the top of the hour.

So I'm gonna use that as an excuse to to sign off for today. I want to thank our audience, especially a lot of the people to Easter in Scott and in some others who really participate and appreciate it. Most of all marks on hurry Jennifer.

Thank you for coming on and making this a great discussion Mitchell as always. Well, you did great big extra shout out to our friends at tricentes. This was one right in there wheelhouse.

So I was happy to have this out here for them. Thank you. All we will be back with our regular kind of about some bound recorded panel and another two weeks and then we'll be back on another Roundtable within the month.

So check that out. I think Mitchell put in the questions if you go to devops and bound You can check out past episodes of devops around and stay on top of the latest. But for now then this is Alan Shimmel a texture.

On behalf of texture on TV and devops and down. Thanks for joining us everyone. Have a great day.

Thank you. Bye.