Strategies for Performance Testing in the Cloud – DevOps Unbound Roundtable

Good morning, good afternoon, or good evening, depending on where you are in the world and welcome to today's devops Unbound webinar brought to you by textrung and tricentus. My name is Cody J Brown. I'm the host of tech strong learning.

We have an exciting panel ahead. But first I have a few housekeeping notes to touch on first. Today's session is being recorded So if you miss any of our discussion or you'd like to share it with a friend the on-demand recording will will be made available shortly after we conclude the webinar today.

If you have any questions for our panel members today, we want you to go ahead and put those into the Q&A tab, which can be found on the right side of the screen and if you have any further comments for our panelist or you just like to let us know from where you're tuning in once you use the chat tab for that. Finally at conclusion of our webinar. We will have a drawing for 425 Amazon gift card.

So stick around to see if you're one of our four lucky winners. Onto our topic today. We'll be discussing devops Unbound strategies for performance testing in the cloud.

And my expert panel today consists of Mitch Ashley principal of Textron research Alan shimls CEO of tech strong group. Hendrick rexid Cloud native advocate for Diana Trace Brian Cole director of customer engineering for tricentus and charity Majors CTO of honeycomb and it is my pleasure at this point in time to let our panel have the floor. Thank you all so much for being here with me today.

Thanks, Cody. Hey everyone, this is Alan shiml CEO of tech strong group. And thanks very much to Cody for the introductions.

I'm going to give each of our of our panel members a chance to introduce themselves further in just a moment, but I wanted to go over a couple of house. Cleaning or ground keeping rules here because this isn't your regular webinar. Right.

This is devout. Some bound is a bi-weekly video series that we do here on Tech strung TV, and then once a month or once every six weeks on Tech strong learning, we do a live version of it. So this is our sort of normal devops Unbound except we have some very special guests in addition to our panel members and that's you our audience and how good or interesting this panel and this Roundtable are will be directly related to how active you are audience.

Are you guys drive a lot of the conversation you guys drive a lot of the activity Cody mentioned that we have a public chat section and feel free to use that. People already are we're just getting participant names and not their real names or what. Have you.

But if you're going to say hello say hello from where you're from Ice. We got somewhere from London, right? So someone else from New York, but you know, if your guy always get a kick out of thing where in the world people are on these things and you can feel free to chat in that public chat, of course, whatever you put in there.

Everyone sees everyone can answer including our councilmember our panel members Or other attendees there is another section of your chat bar on the right hand side, and that's under Q&A. And if you put something in the Q&A, I just our panel members will see that and we will try to bring it up. To the to the whole panel and and you know, it doesn't just necessarily have to be questions.

If you have comments thoughts your own opinion. We you know, this is an interactive session. Let's let's be interactive.

um with that being said now though, let me let me introduce our panel Cody had introduced her last but I always liked to introduce her first. I'm really happy to be back on board here with Charity Majors. Who's the co-founder CTO at honeycomb and charity will tell you a little bit about honeycomb and herself.

Hey Charity. Why don't you give him a little background? Hi.

Yeah, I feel like the odd duck out in this panel because I don't have a history of like two decades of performance engineering. Although I've done a bit of database performance engineering. I I'm currently the co-founder and CTO honeycomb where we do observability for complex distribute systems.

A lot of people are stealing our marketing language, but nobody actually does what we do so you should try it out very generous here. Absolutely and charity you're never in our duck out of a good fact that the discussion. I know you well enough.

That's true. I have your opinions for everyone and and quite frankly you have a lot more performance. That's the experience than I do.

So, yeah, okay. I'll take it, right? Yeah, you're good.

No feelings of inadequate inadequacy here. Let's next. Go ahead Rico coming to us Henry.

This is the south of France and Rec. Welcome to devops some down a little background on yourself. Thank you.

So, my name is Henry correction like you mentioned and I've been working 15 years a bit more than 15 years. In fact in the performance industry. So I've been almost a colleague of the other.

A council member here, Brian Cole. So I was working from the Otis and now since one almost one year. I've been I'm working for directories the observative platform.

Similar to a honeycomb so we do both provide observative for corporate Enterprises. And here I'm very excited to talk about performance testing and how we could also validate anything related to the cloud native space. Good and thank you Henry last of our regular of our panel members is Brian Cole.

I should mention for those of you who may have signed up in days a week in advance. Brian was a not last minute, but you know recent addition to our panel stepping in. So Brian special big.

Thank you for my pleasure strapping in and if you wouldn't mind giving the audience a little background. Yep, so, my name is Brian Cole. I am the director of customer engineering here at tricentis and I have spent the last 24 years dealing with performance testing tools and software Solutions primarily at software vendors.

So a lot of proof of Concepts a lot of helping people realize the value in giving strategy and guidance advice. Excellent and the last member of our panel, but certainly not the least is my co-host on devops. I'm down.

Long-time business partner Mitchell Ashley. Hey Mitchell, introduce yourself down always good to be here co-hosting with you and love our panel. By the way.

We may not have baseball. We don't know we're gonna have a Brian is our pinch hitter. So thank you Brian for yeah, that's Mitch Ashley.

I am principal with our research organization text wrong research and also CTO with Alan for Textron group. So great to be here with everybody. Thank you for coming here.

Thanks for joining everybody. Thank you. So as I mentioned just for anyone who may have just recently joined in.

Thanks for joining in. People from Togo and India, you know, if you have questions comments put it in chat put it in the Q&A if you want the panel to comment on it and and we're gonna get started. So we talked about performance testing in the cloud.

right two of you have been doing it extensively through more than 10 years. To mean that there's two aspects of performance testing in the cloud one is look when I used to hear performance testing as an executive a company Mitchell and I worked on together when Mitchell would come in and say well the engineering team has to do performance that we're done with coding. We got to do some performance testing but my hair was on fire I had hair back my hair was on fire because man that just seemed to me like Ching I'm writing big checks performance testing was hard.

It was expensive to scale it out to because she back then you didn't even have the virtual Ability, you don't wait until you're done writing code to do performance testing because you that's what it was. Thankful after sharing my good laundry, but but that's what it was back. Then it was like oh God, we just spent two three months coding and now we have a release due out in a month and a half or two and my we got to do all this testing performance testing.

We we don't have enough phones. We don't have enough browsers. We don't have enough anything to really kind of testing and you know the idea of having the cloud well wasn't the cloud supposed to obliviate all that I wasn't now I had burstability and and no elasticity and all of these things.

Should I still have bottle necks the flip side though is look I've got applications and that run in the cloud. Right. So, you know, I I where where are my performance bottle next anyway, so, you know the bottom line is the clouds had a profound effect on performance testing Ash has devops and shift left in all these things charity you spoke up.

Sorry. I play you know, I'm gonna throw it over to you. What do you think?

I think the performance testing is just another kind of testing and it's not even safe to the end. It's something you need to be doing all along. I also think that you know, my big brother bear is testing and production because you know, you can do as many synthetic tests as you want.

You're still gonna Not Gonna Learn f*** all until you actually put it in front of real users. So like I think that the the real, you know, Frontier of testing is feature flags and you know, Progressive deployments and anything that gets it in front of real users, even if they're like a small segment of users as quickly as possible as soon as possible so you can start finding the the real, you know, unknown unknowns in your code. Okay, Brian henrique that is that Anthony you or you agree?

So I would I would say that first of all, I just want to remind that performance testing. It's a big name that includes lots of various type of tests. So saying testing in production, I would say yes, but for a specific type of performance test activities, if you apply heavy load you when you design your tests you design them in front of specific objectives that will cover specific situations or risks that you have identify.

So if you want to I don't know just break. It just destroy things. I will never do it in production or you can probably if you're immature Enough by unplugging your database because you want to do read and write in your database.

So if you start yeah you riding a lot of assets in your production environment, then it could be also for the business perspective a lot of disaster because you will include a lot of date and the data that doesn't make sense for your for your business analytics for example, but yes you can do in production but it requires a lot of maturity special from the load perspective. If you do just send a big test. Yes, obviously, it's a way of get a signals to see how How response but for me testing the the performance and how many concurrency your services are able to handle that needs to be taking at early as possible like mentioned before before.

So every time you have a new service you can start doing very simple basic tests that will identify the the limits of that particular services and then one you move move in your Project Life Cycle then you can Complexifier a bit your tests by including more aspect because you're connecting more services together, but I think for me don't improvise in production, that will be my my biggest share for the for the audience performance is crucial. I mean, I mean look around you if you open your browser and you go to your social networks that you your prefer social networks, and you doesn't respond. You said what happens here?

It doesn't respond in half a second. So people are expecting performance from you. So don't do load without knowing what you're doing in your production work.

Yep, and even the low is the new down, right? Exactly. So after about four seconds you lose about a third of all of your audience after about five seconds.

You've lost about half. So if you're you know, I'll let you talked about that big paycheck towards the end. There's a huge cost to having a slow performing application.

If you're gonna especially on things like an e-commerce site potentially the cost there is dramatically higher than whatever you're going to spend a performance testing and I've been where you were as well or in those early days of development went for about eight months and now it ran a little bit long. So we had six week QA cycle and that ran a little bit long. So we've got two weeks to performance test and I was that guy asking the question what happens if we find a problem Because performance isn't a feature to be bolted on it is an inherent characteristic and attribute of that application.

And if you pile a bunch of features on top of a poor architecture, that won't scale properly. You might need to undo a lot of work in order to remediate that problem. It's potentially very costly which plays to henrik's point of do this continuously.

That is the continuous performance testing message of embedding it intentionally into your ci/cd pipelines constantly stressing starting from the individual components and services that you're developers are creating. com as an example. I'm a great candidate for them to turn stuff on.

I'm an Amazon Prime member. I don't shop terribly often. I'm not running a huge.

To transactions, but I'm probably loyal enough that I will put up with a feature failure or some sort of system error. I'm the target audience for this feature deployment into production for that subset of users to make sure everything's working while the system is under that traditional user load. Exactly.

I have to say I didn't realize that I was so ahead of the curve. I've been accused of testing my code in production since almost I started. That might be a different problem.

We want to talk about maybe we're production. It would have worked in test. Right?

No, so certainly things have progressed a long long way. I I totally agree with you know, we're talking about the impact of performance and how that affects users experience the business all of the above and the incremental Progressive kind of approaches that charity that you mentioned. It seems that gets a combination of how we architect and configure our environment and software and how we test things and also how we use resources differently, right?

Because we're not only testing for one environment. We have be testing for multiple environments or current a future maybe going back and figuring out recreating something so we've got a lot more flexibility, but there's also complexity that goes with it too. So I think it's the linear approach we were all talking about nothing is linear anymore.

So we're just going to throw that out and think about how we test these different variables and feature flags and all of that. I think any capture a charity. Captured replay is like the it's like the gold standard of tests.

Right? Like I've written this piece of software for three different databases. Now where you capture all the traffic for 24 hours, you take a snapshot of the database at the start and then you replay it again.

And again, you tweak the variables right? And that's where you can just slam the discs right tweak that read I/O versus the right I/O and and it's super fun, but it's not as you know, that's probably the best way to you know, flush out that Nona knows you're not never gonna really address the Michael Jackson problem, which is like the the problem when Michael Jackson died. He was only gonna die once and it took down Google because nobody was prepared for that, you know, and I don't know.

Those are your unknown unknowns, right and like another really interesting idea that you know Netflix does and and you know people who have reached some level of maturity like the novel maturity where you haven't instrumentation you have enough reliability that you're like looking at second order problems. It's just a run like 10 or 20% of your traffic as as test traffic all the time because then you always know that you have overhead if you can turn it down, right if there's an unexpected spike in traffic just run it all the time. And of course then you then you need to make sure that the traffic that you're sending synthetically is that distributed evenly across all of the shards or partitions or whatever, but I think it's a fun trick.

You know, so let me let me. This early part of our of Our Round Table here. We really kind of focused in on Testing, you know shift left testing free performance versus testing performance.

And yeah look strategy. Number one. They're not mutually exclusive you can do both.

Right strategy number two and it's something Brian. Yeah heart hit on I had on it. Is that for too long performance testing was kind of the The last you know, the last kid in the bed in the little one said roll over and that's the one that gets knocked out in the bed, right?

Because first we did the code then we had to do all these other testings and we almost started doing performance testing and production not as a strategy that was beneficial and the way to do it. But as a strategy of us crap, we don't have enough time we get it done before and this thing has to go out the door tomorrow and okay, we'll do the performance testing. You know on a live audience, it's the best way to learn anyway, right, but that's not really it's that that's you know, the Henry's Point that's improvising.

That's not strategy that right if we're gonna do performance testing and a live environment. We should be from the outset saying we're doing it alive environment for all the reasons Charities talking about Right not because shoot we ran out of time and this thing has to get out the door. And I think that's been one of the biggest differences.

And maybe it's the devops and the continuous testing kind of mentality taking hold of saying we are going to do performance testing in a production environment doesn't mean we're not going to do performance testing pre-production. Well, we're gonna do some in a production environment and we're not going to improvise to Henry's point of view to Henry's point. We're gonna this is what we want to do.

I mean think about Brian. What was the first time you heard the term feature Flex? Oh at least six years ago at this point probably a little bit longer and I can probably count on one hand the number of companies that I've interacted with who are using feature flag deployment into production today really very rare.

Wow, really? Now yet, I know. You know from our Tech strong TV interviews.

I mean if you quite a few companies that are very involved in feature flags and you know, and some of our kind of feature flag is a service if you will and the numbers are in the billions. right Henrique, I'm wondering I don't know. How much are you seeing sort of feature flags and let's call it post-production performance testing like that happening.

So I think it really depends on the visual the the maturity of the the company and which business areas. They're working on obviously when you come to e-commerce, those most of the the retailers are quite aware and on that approach of you losing feeder flax for all the business that are more critical like financials banking or stuff like this. I want a pretty most of the banks want you that's really flags for father, please for the for the the one that I know same thing for insurance company.

So I think it really depends of how agile or how Fragile I can Name It We I am and how mature I am to implement this and once I have that then I can take advantage of this and then I think the devops movement has pushed companies on that direction and I think the the notion of blue green deployment is it's not a video flag, but at least it help you to test with a minute with a small population your release and validate that's and secure it before deploying to all your users. So feature Flags or plug-in deployments. I mean it's up to you but is that you have to to measure if what all the the news this release is in is adding to your business or to your users before pushing it to the 100% population of your quality or customers.

I would say And also I would just want to bring one topic up. So we were mentioning our performance testing on the cloud. There are obviously things related to the cloud itself because we rely like Brian mentioned at the early beginning of this of today's event that cloud.

Yes, you can scale. That's great, but it at the but he has a consequence the the notion of scale the notion of power you pay it. So 20 years ago, we were measuring.

How how fast is my service responding? Now we can say how expensive. Is my service running because if I have a service that requires one node, or a couple of PODS to run and then the next release it requires 10 times more than potentially.

I've increased the costs. Of this of managing this application in my production environment, but with 10 times more just because I have a my service requires more more power to to serve my users. So that's also an angle.

So yeah, there's a lot of kpi. I think behind the notion of performance especially in the cloud that we didn't have 20 years ago when we were just testing basic on premise sort of applications. That was it.

Okay. Charity, I was going to say something and something from the audience. But before I do I thought you sounded like you wanted to say something or no?

No, okay, so You know. It's funny Brian. I also didn't know.

the term feature Flags told maybe probably four years five years ago, but I knew blue green it's something you know, and I and I'm far from a Developer Mitchell much more than I'd say. I've no blue green. One of our folks in the audience says that you know, they've been working on the internet banking platform, but they've been using feature toggles as he calls it.

For going back 20 years. So I I don't know maybe the term feature Flags is a new term what what's represented behind it, whether you want to call it blue green something else I think is something that we've always Or at least for as long as I aware. Was something that we've tried to put into you know, it may not have been called performance testing but we did blue green thing to see what a customers like, right.

Did they like a square Edge or around Edge on that button? They like it in this font or that fine. Right?

I mean these These are the you know, the that's how I always thought of it at least so I agree with the the audience member who says it's been around. Guys, I want to focus for a little bit on pre-productions I think though. Right.

Is it still a pre-production performance testing? Is it still a thing with the cloud? If so, absolutely but there's some Nuance to the conversation that needs to be probably talked about.

So when we talk about an application in the cloud, there's two main types that we're discussing here one is an application that you are putting into a public or private Cloud. You're deploying the software you're running those AWS or is your servers that's your application. Absolutely.

You want to performance test that those are very different than software as a service applications. If you look at something like a work day or a sales force or some other system that you are subscribed to those contracts often include language saying you are not allowed to run performance test and that system. Those are multi-tenant applications those underlying compute resources, maybe serving multiple customers.

They don't want you running a load test to bring down that environment or degrade it which may impact many of their customers they're responsible for Dividing the service and that should be written into the contract as well. So if you are looking at signing one of those contracts look for the performance guarantees, they should be in there and if they're not add them you have a expectation of getting the service that you're going to be paying for. But when we talk about Cloud it's this idea that I've heard many times of I can just scale my infrastructure out if the application begins to get slow.

I can build an elastic application that will begin grabbing additional compute resources. It will provision additional machines. And I can scale laterally that way that's a great way to have your costs kind of spiral out of control in that you lose a lot of the benefit enter.

Well, yeah only works for stateless application. It sure does and also just to add yes you can scale. But adding an extra node keep in mind that it could take between spinning up the node or spinning in the extra pod getting the container all the assets to make the application ready to take some traffic.

It could take almost four or five minutes and when you're in a spike situation four or five minutes could be eternity. It could be just stressful and you can wait for that extra resource to come in but it just too late and then you have what we call the waterfall effect where everything's falling down because the node is not able to take the the traffic anymore than you stressed and you know, they're just come in and then you are waiting for another one and Jen you just struggling. So even with scale the elasticments they are obviously mythology behind the scene to validate how you want to scale to scale up you nodes because they are usually the policies based on thresholds specific thresholds that you can determine on your cloud provider and this needs to be tested to validate that the policy that you're going to use in the case of scaling up and you know is able Check the traffic in time because it's great to pay for a not awesome elastic service.

But if the elastic service that you have there is not helping you in peak hours. Then it doesn't make sense. It's interesting.

There's a SRE angle to this too. It's not just you know, when we don't have enough performance. We add, you know more capabilities to to meet to meet the demand.

There's also trying to break it trying to find where the breakpoints are what those things are that we can go back and address from an engineering or configuration infrastructure perspective and performance. Testing can fall also falls into the realm of the sres as well as test Dev and that whole cycle so performance testing, you know to Charity's point about doing it early in the beginning and as we talked about very in production and very various phases. It's also an ongoing life cycle element of this.

I think it's a mix of early performance engineering will be from a devops teams. And then close it to your production is gonna be the sres but the SRE when they do performance testing they want validate the same objective the same risks because you're fishing for a reliability. So you're gonna probably mixed load performance and Chaos engineering to validate critical situations.

So yeah to that point, I think there is a part of the traditional Big Bang load testing that we knew a few years back that are now in the hands of the sres and now the short simple load test is in the hands of the teams the develop teams. Because we got a question or in the Q&A section from our audience and question is around. You know, what about the impact of things like blockchain for instance?

Right it blockchain is of course a potential fly in the ointment. It's or maybe it's a friend and not a foe. I don't know.

I mean, what what would you how does blockchain and that type of technology impact? Would say performance testing, you know, it is impact across. A wider Spectrum I get but you know specifically to testing here performance testing.

I know it has blockchain has been performance testing or performance of blockchains has been an issue and there's various now ways of Designing and building them to be more efficient more performing but I think that was an early question. Maybe it still is relevant of you know, how how fast how much can we put on dependency on blockchain as part of our transaction flow in response time to customers versus asynchronous activities that could happen on the blockchain? Okay, anybody else any thoughts on blockchain because I have another QA from the audience.

If not, so I think there is. Depends on the the nature of your blockchain network. Because again blocking is just a network.

So the blockchain network if it's a private one, then you own the network. Then if you know that you have one node that sits in Paris and the other one is in Sydney and you want to make sure that the transactions are mine in the right time then. Yes, you're gonna test it.

but and if you do public blockchain It's like testing peer-to-peer. Do we really test peer-to-peer transactions? That's that's a questions and it's more difficult more expensive.

So I think testing the private so the hyperledger networks. Yes, you can definitely do it in as a load testing with low testing for sure. Sure, so technology perspective.

I'm not certain that the performance testing Solutions out there care very much. Right. It's an API.

You call those invokes the the service this is very much a Telemetry and monitoring issue trying to figure out is it actually working properly behind the scenes? Yep. Some got another question here in the QA section.

And this one to me is really look not everyone in our audience our performance that thing or even testing experts. We get Cyber people and devops people and Cloud native people and then and it's really goes back to basics kind of thing the stress testing and load testing come under the umbrella of performance testing or are they are there huge differences between the three of them? So from a absolutely.

Yeah, there are differences between them. I would say that in all the hundreds of customers. I've interacted with the terms are often used interchangeably synonymously and wildly divergently from one another and that there isn't a set standard that's out there.

If you're asking for my opinion a load test is where you are attempting to put user traffic or load your simulating users on a system based on reality. They're going to go at the same rate as normal users. You're trying to load it up to what you've observed in your current systems as expected load and normal conditions performance testing might improve or increase those numbers as you're looking to kind of stress out and start to get towards the edges of that stress testing is really where you're trying to go vastly beyond that.

So if we use many years ago into it had a performance issue around Tech season because they budgeted for multiple factors of additional load, but that was that turnover year where almost everybody started to e-file and instead of getting three times the user traffic as they expected. They got 10 times the user traffic so stress testing is designed to try and find those but there are countless other terms that you'll bump into around endurance testing negative testing volume testing. Which in support of Charities argument I particularly hate the term volume testing, which is you should run your test with large amounts of data.

Yeah, you should all the time never as an explicit type of test. That should be the norm you need large volumes of data that are going to exist in the production environment. It's like imagining you're holding a water bottle.

And the goal is you unscrew the lid and all the water needs to come out. So you hold it upside down you unscrew the lid if you have a thimble full of water in there comes out really fast and you go great a response time is half a second and then you roll it into production and load up a full water bottle. It takes a lot longer for all that water to come out and suddenly your expectations are now not in line with reality.

I can't emphasize how important having a large volume of data while you're performance. Testing really is you should absolutely be prioritizing that fair enough right. So let me go to a new subject within the it's not from a question in the audience.

It's my own. Kind of question, you know part and parcel. Important part of the whole devops experience is the concept of feedback loops.

Right and look it's one of the it's you know, it goes to the heart of observability If You observe a tree in the forest. But no one gets any feedback didn't really fall. Hey, I mean, it's the same thing with testing.

How do we make sure we have the proper feedback loops for our performance right in the world of continuous testing pre-post production. What are the feedback loop? You know, what how do we make sure we we've got those feedback loops.

Yeah implement this is this is where I feel observability is so important and and I often say to people who are thinking of doing some sort of chaos engineering like cast engineering is something where like you really need to be this tall to write this right and this tall is defined by having enough observability to tell what the f*** sorry what you're actually doing because you know, I've seen people who have like done can't chaos experiments and then a month or two later. They realize there's some there's some like Refuse there's there's some versions out there that are still old versions or some stuff that they you know block using a network and it's still out there in the wild because they didn't have enough information to clean it up afterwards, right and and by observability, I think you need the ability to drill down to the to the raw Rose to the rot. That's to see like, you know, this is why you try cardinality.

This is why you need High dimensionality to be able to ask the questions like after I made this change what changes happen and where so that you can see. Oh this notice still issuing, you know above average number of errors or oh this Shard is still, you know, returning these weird weird things or oh, you know, this this instance is still, you know, running this old version or something. If you don't have the ability to ask these really fine-grained questions to see what you actually did then you don't have a feedback.

No. I don't disagree Henry you want to say something? Yeah, so I would say that first I think devops and SRE relies on observatory in general.

So if you want to be successful in your devops any way, you deploy your application through your devops mythology and how you manage your environment in your production. You need obviously observantage. So by observity, I mean when you're testing especially when you do continuous testing, you don't want to spend three hours to run tests and then suddenly say, oh, I don't know what happened.

So let's rewind the test. I'm pretty sure if you say that to your manager and say I don't get it. Why are you losing six hours just because you just forgot to pick up those metrics.

So so yes OBS required. So you need to first of all measure how your environment Works technically How the of course the user is experience it has what is the experience from the user perspective? And also if there is a problem in the code, if you have a debility to drill down to the code with traces, then you will save a significant amount of time that's from the pure pre-product staging environment approach.

But then when you go to production you have deploy something you want to make sure that between release one and release two, you're always up to speed that you're always good. So I think with observity that's that's it's natural you need it and I think to be able to automatically validate and be smart in your Automation in the continuous testing approach. You need to implement what we call SMI and SLO.

And yes, so we practice that will help you to validate your your test results. From at least that they are early stage, but then yeah the advantages that you have you took the time to Define your SLI to define those SLO and then you can use those SL in SLO in your production environment. So then you keep track on all ways the same objectives from the from a pure deaf perspective staging whatever and to the production environment.

and it plays into that conversation about Continuous assessment essentially the propagation of information from one team to another and very often. We don't see a lot of operations involvement in the pre-production space. The devops term seems to be Mostly unidirectional Dev and Ops and that really should change.

There's a huge amount of value that an operations team can contribute and provide in any performance testing activity most notably for their own comfort with what's coming down the line and it's about to be running in their production environment that they would surely they would like to know what this thing is going to behave like this provides a great way for them to be gain to gain that insight and access to here are the metrics here are the performance characteristics here is how it was performance tested and hey that matches our experience in production or it doesn't and maybe we should provide some feedback on how their tests are being structured and created a lot of value in involving the performance team and without monitoring its Of limited use to know that there's a problem. I would say a good analogy is looking and seeing smoke over the hill, you know, there's probably a fire as opposed to knowing which house it's at as opposed to knowing that it's in the kitchen in the oven because of the following reasons. They're they're observability provides tight guidance and targeted response mechanisms and leads to the practice of performance engineering not performance testing, which is run the test get some results and then provide actionable feedback on what should change in order to improve performance.

That's everybody wants to get there. I have another technology like Brian. Yeah, you used to to compare performance testing without observability or monitoring at that time.

It was monitoring with it's it's exactly like listening to a horror movie at the radio. So you hear people you hear people screaming in the radio, but you have no clue who is kill. He's been killed.

So we are in 2022. So just purchase a 4K screen and then you will see the actors being killed on lively. So that's obserably I find it interesting that all of our analogies are so dark.

I'm tired house fire or movies. I think it's really good point. You know Alan I thought it might be interesting to explore.

how can observability, you know having much more insights into the sort of internals instead of thinking of performance testing is an external factors IO and throughput and all the normal things and being able to drill down is really really you know like We're so used to you know, our monitoring or observing or office metrics being so low level the you kind of need to translation layer to translate it back into the language of like endpoints and apis and stuff and like and that's something that I think is changing and can't change fast enough. Right? Like you shouldn't need a wizard to interpret to you like oh this changing in your code resulted in you know, these you know, because there's like what four different kinds of memory usage and in Linux and like all these things and respect Park you shouldn't actually need it to have it translated for you.

You should need to have to be expert and you should be able to trace it back to oh I made the change in my code. So, you know this endpoint the errors or the latency is Raising rising or whatever. I think that's super key.

Yet sometimes sometimes it's not it's the fields of the view are not as clean as that though, right? You know very much. So yeah, and that plays into that the Box message right of I want to have that pipeline of changes the developers write some code.

They check it in in series of automated steps begins at that point. What we want is to have the ability to link back. This performance test was executed because this source code was plugged into the CIA system and built and deployed and then tested and that source code change made things worse or better.

Yeah and be able to correlate that properly it's very challenging to do because software is very weird, you know, you touch component a and suddenly component are as much slower. Yeah, okay when nobody knows why exactly but you start to find these very bizarre relationships between different components. But sometimes I see this like in talking to people in the future flag space, right you start running multiple feature Flags in releases and you're seeing results and can you ask isolate?

Excuse me? Can you isolate? This is a result of that.

But there's other feature flag that we had set to this didn't have any impact. what's called feature flag a didn't have any impact on feature flag B because they were just two different things anyway, and and it can get it can get Cloudy quickly nobody you, right? It's very red.

You can draw a straight line from one to the other. I when it happens. It's fantastic like the odd out.

It's just moments wearing that just goes right. Hey guys, I I hate to do this but we are where we're 45 minutes and we usually try to keep these sections so about 45 minutes, but I wanted to that's Cody if you see because I know we have Amazon gift cards to give out, right? and Cody if you want to announce some winners or did I get you off guard?

No, no, we're totally ready. So our four winners for today's $25 Amazon gift cards are venkata e Timothy Q Sandra S and Frank J Excellent guys before I give each of you a chance to say goodbye to the audience. I want to first of all shout out to our good friends who try sentence.

They sponsored devops on bound. We're not always live like this is as I mentioned three out of four always just pre-recorded. com.

And there's some good ones up there. Check it out. We do have next month or the I don't know if it's a month or the next live Round Table Topic open and that'll be on March 15th.

So it's actually just a couple weeks out and it's can Automation and AI ml solve our devops hiring problems. and I don't know if anything can solve our devops hiring problems, but we'll we'll try to tackle that one sign up for that. It should be a great life.

Should be a great one charity. I see you you're showing up to you. You always show up Charity.

But anyway that let me let you know the chat thing. All right? Let me give each of you a chance to say goodbye to the audience.

So we'll start off with henrique if it's okay. As first of all thank you for to listen to this to this great discussion a two it's too bad that it's only 45 minutes because we couldn't probably stay for three more hours. Anyway, if you're looking YouTube performance I would recommend to to channels in YouTube first is a perfect bites.

So it's a well-known podcast that is turning to YouTube. So check it out. There is a lot of tips around performance and the second one is my YouTube channel about is it observable.

So if you have if you're looking for any tutorials related to observability in general with opens open observerty Frameworks, check it out. I cover a lot of Open Source Frameworks. So if you were looking for observity check it out Channel.

actually and we'll check those that charity. How about you? Uh, you can find me on Twitter at missy Tipsy.

io. We do a lot of blog posts about open Telemetry and observability instrumentation in general. This has been super fun.

I would I would happily sign up for a three hour webinar to talk with you all about like my sequel low testing and and I'm gonna be load testing and be a ball. I don't know if anyone would like to watch it, but I would have fun. Well, you know audience you could vote right everybody's like no, no three hour talking about performance testing.

I learned one of the things I love about being in our Tech space in the plate where we play. There's no matter how weird or different you think you are. You're not in this right because there are people who are into it.

So I love that about you know being in in Tech is there are people who really do want to talk about this stuff for three hours. And and those are that's our audience, right? That's those of the people we're trying to reach and maybe a little weird but that that's the fact.

Brian how about you? So thank you everybody for attending today. If you do have an interest in beginning your own performance testing Journey or you're interested in a very modern take on performance testing.

It would encourage you to look up the tricentes website and see what we have to offer. Excellent again. Thanks actually sent this Mitchell.

Why don't you take it home? You know, oh great companies that folks in the panel are representing and thank you again to try centers for sponsoring. I think one of the things I take away from our conversation is Yes performance testing performance can be designed in as part of you know, our application architecture and devops processes, but it really is a holistic thing.

It's part of the life of Because you know nothing happens to you put it in production. That's when you really find out. Yeah, and now we think about you know label your performance testing SRE.

Maybe you'll get more funding for it or do something like that. But I think we have an opportunity to really make this part of the the ongoing life of software. Because things change to what are fun than performance testing is where you find all the things that pull your own like.

Oh my God, I had no idea my code is doing this. Like what's more fun like that? Let's break it right exactly that I can do the three.

Yeah and hand it over to somebody else to fix. It's great. But that that used to represent 60% of the people in what we used to call the info second industry.

Why did you get into info Seco wasn't for the money it was because I like to break things. All right, and that was my best friends and security. That was what they were all about.

Anyway Henry does a question for you in the presenters chat if you can answer that. The charity, but guys Mitchell. Thank you.

Thank you Brian. Thank you Henry. Thank you charity.

Thank you for watching this. It's been a blast do check out devops Unbound on techstrom TV. We'll see you all March 15 for a live Round Table until then, it's Alan shimmo for Tech strong and devops on down.

Have a great day. Everyone. Bye everyone.

Thank you.