Living at the Intersection between Testing and Observability – DevOps Unbound Roundtable

I believe we are live. Yes, we are. All right.

Hey everyone. I hope you're here. We are starting on time.

So we'll give some folks a chance to log in and come on in here, but welcome welcome to devops Unbound and this is a Special edition of devops on band we do them about once every four to six weeks and we call it our live Round Table edition of devops Unbound. And my name is Alan Shimmel. I'm the CEO Editor in Chief Textron group.

com and Security Boulevard container Journal. digital cxo Textron TV as well as many video series much like this devops and bound before we go any further. I want to shout out.

Our sponsor for devops on bound is the good Folks at tricentes. And Chase sent us. is Has been partnering with us on devops on Banda from oh, I think two years maybe two floods here.

And they are terrific partner. Also, obviously the worldwide leader in continuous testing you check them out, but many thanks to try sentence for sponsoring. For those of you who are here for the first time devops some bound really we you know in every show and it's usually every two weeks and as I said once month to six weeks we do these Live Events.

We we look at interesting topics in Frontiers in devops, right and it's very little devops related that we won't put under our Focus. Um, but today today's show is kind of right right in the mean of things and especially with what's going on in the world today. I'm going to introduce you to our panel in just a minute, but word about these live round tables.

The reason we do them is because of you our audience. Right, we could we could go on zoom and record these and show them to you and it's all fine. We've got we've got seven people including me on this show and we have no problem talking for an hour about observability and testing which is what we're going to cover today.

But what really makes these shows special. Are you the studio audience in these shows when they run, right? You guys drive?

The discussion and you drive the discussion by using the chat. Function and Q&A function in your browser windows for most of you using a browser. It'll be on the right side of the browser.

You'll see that communication panel and if you see chat by default is lit up and usually public is lit up you can you can talk to anyone right in there. Right? Just feel free to chat away.

I think many of you already are it's always good to see where everyone is from, you know, because the Magic in the internet, right? It's a worldwide audience. So feel free to let us know where you're logging on from there make sure the chat works.

So that public chat is great if you want to talk to other attendees as well as Some of our panel members the Q&A tab next to chat up on top. There is great. If you have a specific question or comment for the panel that you'd like us to address put it in there.

I check it all the time as there's the rest of the panel and we'll we'll present it to the panel and let's see what we have to hear, you know say about it. So use chat use Q&A. And and if someone else posted comment or something in chat, and you want to answer feel free to chat away and answer that as well we do and that's when these things work best when the audience is involved.

So, please feel free to get involved. I'm coming at you today from South Florida, Boca Raton. It's good to see you all from around the globe.

Let me introduce you now though to our fantastic panel that we have for today. I'm gonna ask, you know, tell you their name and ask them to kind of give you against them to give you their own background. So this way I don't have to read their bios, and you can really hear what they're about.

Let me first introduce you to Brian Cole. Brian is a director of customer engineering which I sent this. Hey Brian you want introduce yourself?

Sure. So, my name is Brian Cole. I am responsible for evangelism and advocacy for the neoload product line in particular but performance testing and performance engineering generally and today's subject is near and dear to my heart.

I also chair the performance advisory Council and the theme of this year's event is observability and testing you can't do one without the other and expect to have a good outcome and I'm super happy to be here. fantastic next up. I want to introduce you to Adriana.

Vallela. Adriana is with a senior developer advocated lightstop lightstep. Excuse me.

Hi Adriana, how are you want to introduce yourself? Hey, I honest to be here. I'm Adriana.

Vallela. I am Beast out of Toronto Canada senior developer Advocate at lightstep. I'm fresh into this role just started in April.

I was previously managing and observability team at two cows. So got really familiar with observability and I'd say after my my first love being devops, I think observability is now my second love super passionate about it super passionate about the topic today. So I'm really excited for the discussion.

You know every time you see two cards I'm taking back to my early days. Of playing on the internet and what a Powerhouse I mean, I spent a lot of time on two cows in those days. I know mmm next up.

Well, thanks Adriana. We appreciate you having you here next up. I want to introduce you to Bjorn Edward Edwin.

I'm going to principal consultant with latario. And you know, why don't you give a little background on yourself. Thank you, Allen.

Hello. Everybody be on I've been here. I'm principal consultants for you.

Also. I don't know. Are you guys hearing I'm not yeah, we are.

Okay so good. Thank you. We all have been here from Principal constant from the actual.

I also had our dojo and I I can hear I think we're having a little difficulty here. I did. Yeah, I can hear Bjorn.

Yeah, I can hear him, too. You're not reaching South Florida Beyond. Sorry can't get from Jacksonville.

Alright, UK you I'll be flexible because reconnecting. What should Beyond and dig three you should probably Know My Name by now. So that's Rio.

I also lead our Dojo or enablement practice. And one of my passionate thing is basically helping teams and organization to get to devops. Right right.

It's not just about bringing all the people together or using the right tools or buy a tools in the market that says devops and you go down. What's the right way of doing it and pushing them to the right culture behind it the behaviors behind it. So I I wanted to like push a lot of people towards the the right triangle of people processing tools and I would say that's my passionate thing.

My background is is mostly from application delivery testing doing more from the Legacy or full side and pushing them into more modern. Practices more devops we are working and whatnot. So this is going to be a very passionate topic for me to see that's that's where I'm actively pushing my our teams and customers to go towards continuous testing observability to reach that last flagpole of devops, right?

So it's very easy to start the first one and we want to go all the way to the end. So looking forward to for a great discussion today. Fantastic, you're 19 here.

You know, I apologize. So back to my old customer support script and just restarted. Thanks.

Please work. Anyway, last but not least. I want to introduce you to I always think she's a rockstar.

She'll tell you she's not but I see it. Her name is Charity Majors CTO co-founder. Charity, why don't you I didn't mean to embarrass you but God give us your background.

All right. Now, I'm not a rock star as a classical pianist. So oh.

Family quotes. This was the Rockstar of the 1700s. So anyway, and yeah, so co-founder CTO of honeycomb that IO um Ops engineer.

Previously of pars Facebook Linda lab Etc. Oh and co-author of the brand new observability engineering book. It just came out in hard copy last month.

So if you haven't gotten it yet, you can get it. You can also get a free copy. If you're following the honeycomb honeycombayo Twitter account.

We post like fairly frequently. So very cool. Hold that book up again Charlie.

Let's see it. There it is. Very cool.

Congratulations going on it you are. You can get a sheet of stickers. If you have the hard copy.

I'll mail it to you. If you leave us a little review on it's not I'll mail you the stickers and you can decorate. Very cool.

Very cool. Hey, all right, so that that's oh last but not least my co-host right here. It's Mitchell Ashley Mitch.

One of you introduce yourself. I was pretty bedazzled by all the stickers too. So I don't blame you.

Yes, congratulations charity. I know you've worked really hard with the other authors on your book. So fantastic.

It's a great book. By the way. I've read it.

It is always here. We're like, okay six months to write three and a half years later. I considered a must read so go check it out go grab the books, but you can get with free ones.

So anyway, which I actually see deal with Textron group and also working with our research organization so in Going back to the early 2000s podcast co-host with Alan of many conversations. So good to be around again. What a wonderful crew we have here on.

Absolutely. Thanks and thanks for coming along with me Mitch. We make it work here.

Just hey before we start again, you can chat to everyone in the chat window if you use the Q&A window, no one's put anything in there yet, I believe but those are for questions right here to our panel and and feel free to jump in. So as we've alluded to today's episode is living at the intersection between testing and observability. And I guess before we kind of talk about living at the intersection.

Between testing and observability. We should Define what we mean by testing and what we mean by observability. I think testing is a term that's been around for many of us for most of our careers and before it's fairly easily understood, right but, you know there obviously many different facets to testing this pre-deployment testing.

There's post deployment testing as well. And many different flavors of that testing observability on the other hand. Is the more recent term that's burst on our scene and Charity's book.

And it's probably a little bit less. Clear maybe to everyone in our audience. So charity now that you wrote the book.

It's kind of you know, there's no there's no getting around it. You're the expert. Why not you define observability here in terms of our discussion today.

Observability is the ability to under to look at your system and understand the unknowns. Not just the No No, the thing you could just questions you knew that you needed to ask up front the ability to understand what's happening inside the system just by looking at it from the outside of the system without having to ship new custom code to understand what's going on because that implies that it was a note unknown that you knew in advance, right the ideas that you don't know in advance what you're going to need to understand and therefore you need to be able to ask any questions interrogate your systems as figure out what's going on in the inside. Just but from the outside There's a lot of things that proceed from that by the way, if you accept that definition, it means that you have to have a solution that handles high cardinality high dimensionality like explorability all these things that like the previous generation of software that like monitoring or anything is based on the metric just actually doesn't have so I think a lot of the confusion out there is because you know, we Define this new technical standard that has an actual definition then everybody's like, oh, well, we do it too because we do Telemetry we do monitoring we do like educate anybody who does anything about visibility now says they do observability and I I'm bothered by that.

Well, it muddles the water. I think we're all probably a little bothered by that. Right?

And this is always I mean, this is not an uncommon problem in text. I'm think it's popular and all of a sudden everybody. Everybody wants to be in there.

Um, I've seen it a few times in companies Mitchell and I started but but never you know, and I guess you know when you go through the Gartner hype Cycles, right? It's that You know when it falls into the trial of disillusionment, then they all run away from it, right? But right here at the peak height, you know, everybody's involved.

Yeah. Adrian I know you know in talking before we went live today you you spoke about you know, and and even in your introduction to the audience how observability is really, you know become a major focus of your career right now. Yeah.

Yeah. Absolutely. I it's funny.

I stumbled into observability through Charities tweets a couple years ago and it took me a while to like fully grasp. What observability was until I was thrust into the position of leading and observability team at two cows. And then I made it my mission of like making sure that like we needed to do right by the company by Doing proper observability and not just schlepping a bunch of tools together.

It's kind of like the devops thing right you you use a tool. Therefore you are devops you use observability tool therefore you have observability and I wanted to like I wanted to cut that like right away. Nip it in the bud before it became an issue.

So my mission was like really making sure that the company had a good understanding of what observability was and once we had the fundamentals, then we could focus on like practicing observability Yeah. You know and we have a question from a participant who wanted to do we have any metrics metrics to quantify the observability. I don't know if I call it the observability it's observability and I I think at the end of the day, it's all about metrics, right?

That's how you know, we use to well two ways to use the word metrics there is little and metrics which is just like Numbers about how things were going and there's capital and metrics which is you know, the building block that the last generation of Technology was built upon which is a number with some tags appended to it. And then you have time series databases, you know, that that you know aggregate them over an interval and so forth and my argument is that no tool is based on the metric will ever qualify as observability because you you have to predefine your custom metrics up front which means you have to figure out in advance what questions you want to ask of these complex systems, which means that you are not quite you would not capable of asking new questions and understanding different, you know, emergent qualities or outages or anything. So like part of part of what I would argue is the foundation of observability is moving from metrics-based tools to Capital and Metro space tools too tools that are based on the building block of these arbitrarily wide structure data blobs where every Dimension it can be a high cardinality dimension absolutely good stuff.

Let's move, you know. I think we've you know done our best to kind of lay out there. What observability is for purposes of today.

Let's Brian. I see you gave a nice answer in the chat as well, but I wanted to Really kind of Define the Nexus the the intersection the connection. Between observability and testing and hey you're with Trade Center.

So I'm gonna so when we talk about observability were in particular through a testing lens. I'm representing tricentus. We are focused on managing the quality process through your devops tool chains and ensuring you have a good understanding what's happening observability is a core component of that it the systems have gotten so complex that it's impossible for any one human to understand the entirety of the system anymore.

It's a thing of the past. And two Charities point if you look at a performance testing solution like neoloud and you say great it has built-in monitors. I don't need anything else.

Those are so limited compared to what true observability actually means. You have to know ahead of time exactly what you need to monitor to understand what the system's going to be doing in systems. Unfortunately are extraordinarily complex behave very strangely under load conditions as they're being developed and have all these interrelated components on third-party systems Cloud providers cloud services that can have a negative impact on how your application performs and scales that without this understanding and the ability to ask questions and really figure out what the heck just happened as I'm running this test.

You're gonna be in a cycle of constantly rerunning tests and fiddling with your monitors desperately trying to understand what happens a true observability. Mindset is what's required to effectively manage the quality process. Excellent.

I want to push back a little bit though on the I hate not. I hate metrics but the anti-metrics thing or it's more than metrics you're on you you put a comment in in the chat, you know kind of plus one and on charity. We should avoid metrics driven delivery at all stages of the software.

development life cycle You know what? So I I come from a background where we were taught in technology. We everything could be measured and we measure everything.

And and based upon what we measured is how we manage and do. Kind of bumps heads with the would stay away from the metrics driven delivery. Yeah, so it's I think the subtle changes measuring I'm all for measuring I think we wanted to know where we're going.

We want to know how we're going. So that's measuring this is like having driving a car and you have all the panels that's measuring to me. What we want to avoid is put a limit at the end of it saying this is what I wanted to go after and this is what I want to accomplish without a big reasoning behind why so the metrics are in my point of your leading indicators to an outcome a business outcome or value that we want to accomplish it out of this even putting a system like observability as more teams are going into the devops ways of working.

There is a specific reason why this is pretty important because you are incurring technical that you're incurring systemic that as we go along and we need to have this intelligence system in place to make sure it is telling you where the failure is and why this failure is happening to us. That's the right. And rather I would not I would avoid things like I wanted to get in my like the common thing that I always get is.

Can we get something like 100% rely with your 100% code coverage on things like that? I think those are something that we should avoid to hit and being the being the only driving factor to go towards a specific outcome. Right.

I'd like to add to that in terms of because I agree with Bjorn's comment. It's it's really measuring not so much metrics and How do we measure successful observability and the way I look at it is? Do you have enough information like is your code sufficiently instrumented?

Such that if you were to fire up your observability system and s*** hits the fan. Do you have enough information to tell what's going on? And I'd see I'd say that is successful observability and I think tying it back to testing then I think testers can help tease that out in the in the testing process.

Hey is the code sufficiently instrumented based on what we see right now, so they they can help identify maybe some instrumentation that might might be missing during the testing process. Identifying those gaps. I completely agree with all this and it goes it's just as prevalent in the QA space as well.

I've had conversations with customers who are showing off their metrics-based quality dashboard, and I look at all the metrics. They're using and I say the easiest way to turn all of these things green is to stop running tests. Just don't test anything and all your metrics go green.

That's what we're talking about here the met the quality metrics were collecting from those systems under test. Yes, we still need them, but they're not important in and of themselves. It's the understanding of why they're representing what they're doing at any moment in time that really matters and being able to trace that back to how your coat is behaving what's going on on those systems.

Why is it doing the thing that's doing right now? I think often like observability and understanding systems gets kind of and I understand why we get drawn back to like outages and things failing and everything because that's the flashy but like we should be understand proactively understanding our software every day, you know, instead of you know in the old days. It was like you'd set up these monitoring alerts and then you'd wait for the pager to go off.

You know, you never like look at your code. You wouldn't look at your instrumentation so much as you'd wait for it to tell you something's terribly wrong. But that threshold for paging people should be pretty high like a lot of things have to go wrong before human gets woken up in the middle of the night which is why as you're writing code kind of Shifting it left and and focusing more on, you know, you you always want to think of your observability system is like a production Rebel right?

You want to be able to write your code look at it through the lens of your instrumentation in production and just have that be a very tight feedback loop as you're changing things as you're as you're rolling things out and and as Brian was saying about about testing, you know, I think Testing is being like the very first step towards gaining confidence in your code. Like it's just it's like basically saying does the logic work right have we regressed at all? And but like your code is not like code alone is nothing.

It's it's nothing. It's useless unless users are using it in production on it, you know, the infrastructure the state of things the current you know users are using it like all of these things are part of the system along with the code and testing offline can't Encompass that which is why you see my shirt says test and prod or live a lie people get really like upset about that saying I tested prod but everybody tests in prod, the only difference is some of us admit it instead of going no. No, we don't test the production which just means you're not actually looking at production.

Yeah, well, let me let me let me and posit to you charity is testing in production really observability. Absolutely having good observability just just means you're closing that Loop right? You're making sure that the code you that you wrote is doing you expected it to and that nothing else looks weird, right?

That's all it is. And the thing is that until recently people haven't actually if you don't have actual observability, if you only have monitoring tools based on the, you know, capital and Metric I understand why people don't look at it much because all you get is these very blunt answers right? Did the errors go up or down?

Does that correlate to users doing something different or my code doing something different? You don't have any way of knowing unless you have you know, high cardinality high dimensionality like raw rope-based systems will let you deploy your code and see. Oh this change that I made is failing in these specific ways that are specifically different from the last version that I rolled out.

Absolutely, you know what one of the one of our participants put in here some people testing product and production by accident and prod by accident. Beyond, you know you you deal with many different customers so you're probably in a good position to answer that one. Yeah, so there a couple of things here one like 100% agree that the the fear against testing in product.

It's always in product to charity respond. There's no value the engineers and developers. Don't get pay check if that product doesn't go to market.

Right? So it's like putting a very simpler term our goal is to put that in front of the customer and how what are we to make that as the best experience as possible is where we want to go and Testing observably these is all the tools that you have in place to make sure that road to the customer is as smooth as possible as predictable as possible and and you're driving towards it. So within in terms of testing and the whole mindset, I'm I my honest opinion is that testers needs to get into the role of this.

I'm I have to get into this continuous testing phase Force. I think that's the first step before we jump on to that next shiny object of observe. Really I can like I can go past this one and rely on my obso really to to give me all this feedback so I can be lighter or more drive more risk based testing.

No, I think there's two compliments each other. This is just basically How are you getting insights of where the system can fail and what the failure is at different stages of the stlc Observer? It is closer to where the production where the customers are facing testing is throughout it starts from quality of your requirement to quality of your core practice and quality of your testing of Our quality of infrastructure or platform behind and all these things.

Everything should should come together to ultimately tell that so if you are testing it sooner in the cycle testing is production should not be a big deal if that is not happening. Then testing in production is a nightmare. So, how do we make sure we we step up to that level right?

So that's that's what I would say. Most of the customers are fearing because the this is the goal to go after but let's break down the goal saying let's make sure you as a developer you have the best test return to catch that logic the method that I've written start from that and build slowly on top of everything so that testing leads into better practices and observe really gives you the most confidence to put that in production. There's also a tie-in from an observability metrics perspective back to testing to help inform what you should be testing.

There's so many different aspects of executing a test case that drive the system. If you're getting observability metrics layered in that aren't being triggered by your test cases, you clearly have a testing Gap. There is a deficiency in your test plans that you need to address and it plays to a larger problem perception problem.

I guess I would say that exists in the industry that quality assurance is somehow the responsibility of the QA team and it is not they do quality control which is the verification of things are implemented as as for everybody does QA including the operations teams, including the business analysts including the application owners. It happens throughout and it should be continuous and it should all be scrapes together in codified into a view that lets you actually understand what's happening and observability is the verification half of that. Yes my test ran why did it run successfully was this test valid anybody who boasts about I have 9,000 test cases.

I'm a little wary of these types of metrics because I might find 7,000 identical test cases that you're running over and over and over again without actually testing anything new in your system. And these are the the general trends that we start to see I've also begun to Socialize the message that the shift left is potentially starting to get a little fraught with the development Community. They're increasingly.

They keep hearing it. Everything should shift left. As a that's not what they really hear what they really hear is my job dude, you do it now.

Yes, you do everything you're responsible for performance functionality building it. It's everybody has a role to play in this absolutely and the work the developers are already doing should be called quality because they're doing a lot of that already today and treat it as such as a first class citizen in these types of quality discussions. If that shifting left, the thing that I always think about is the the research that you know, Facebook and others have done that show that the longer it is.

From the time that you write the code the the cost of finding fixing bug goes up exponentially, right? So like you type the bug you backspace great that was super fast, right if you can fix it with tests. That's super fast.

What's what's this in production? You know, if you're watching it if you're if you're like poking around making sure that that's still pretty fast. But you know, if you let it if you let it go past that point once you've swapped it at the moment that you've written that code, you know exactly what you're trying to build.

Why what you tried that didn't work with the function names are you know, how it all works. This is you switch your focus some other part of code it all starts to drain away out of your head, right? You swapped something else in you swap that out and the cost of fighting fixing that bug just keeps going up and up and up and up and up if it's six months later.

It might take someone, you know, three weeks to reproduce it and find it. So like finding is early as possible. Like that's what I think of when we talk about shift left because it just makes it so much cheaper.

Absolutely. I'm gonna want to talk about another aspect of the of the connection and intersection here and that is you know, look fundamentally part of devops is feedback. Loops.

Right and when we talk about observability, it's kind of the ultimate feedback. right, and and so Have we have we adequately? Made these loops and maintain these Loops so that the findings right the intelligence that we getting from observability makes its way into the next round of testing right improving that code right making a better product satisfying customers.

They're so much that goes into that. Yes. They the idea here is when we talk about devops there is the buzzword Bingo you slap the word continuous in front of it.

Now, it's part of a devops tool chain. So it's integration continuous delivery because he was deployment continuous testing whatever and assessment part of that continuous assessment continually getting information and feedback from what's happening in operations what's happening during your release and delivery time lines and feeding that back into your agile backlog and really focusing on continuously improving everything God I use the word continuously again the the messaging Here is that I see in a lot of spaces on the Internet is when we have a devops discussion. It's Dev Dev Dev and Ops way over here.

And that's really not the case. There's so much value. There's so much knowledge and information contained in the operational excellence of running these applications that can inform and structure your development organization.

They can be leaders from a devops tool chain perspective by providing things like composable infrastructure. Nothing's more infuriating to an operations team than managing a cluster of databases and getting a new application that runs on the next version that you don't plan to upgrade to these types of very basic into interactions between all these teams can really lead to a lot of friction and they're really easy to get around if you start communicating and I don't see that enough. I'm sleep you are you again in your row Consulting?

I mean, how do you how do you instill this? How do you you know grafted into the organizations you're working with? Now I have a magic wand I wear it.

That's what you could borrow that one. Yeah. No, so this let's break it down, right so Just intersecting two things feedback the prop it's it's the process of feedback loop and and the process of Shifting that so shifting left.

It's it's making sure to char respond to making sure we discover this sooner rather than later. So to do this one around later we have to do this actions sooner rather than later to get the feedback. So when when that is when that starts if the organizational this is this is handing off a work for another person to do something else and office like all this after all after the seven steps of the eight step is the operation step.

That means there is application knowledge. There's domain knowledge is fading away at the end of it and the operation just gets the system knowledge How do we make sure that it's across the book the knowledge of the system is key. The knowledge of the application is the key and also the knowledge of the domain what what the customer is expecting from the system to do it.

That's the biggest challenge to bring everything together in a place. That's the core of shift left. You're making sure that knowledge is kind of in the center of the software delivery where you could utilize the domain information.

You can utilize the system information and make sure you are able to build and test and deliver a very observable application at the end of it so that it delivers very customer. So that's I would say that's from the from the customers the challenges from from the culture mindset. We have we have a ton of experience from where we deliver software for so many years.

I think we're really good at it. And I think that there is also hindrance to make sure we are we have to break every year on some things to learn new things to implement it in the current world. That is the second challenge in how people and teams kind of approach it.

Like how can I make sure there's a there's a need to unlearn something here and you could use the learning aspect of things. So they need to slow down to speed up ultimately to speed up and improve on things this tying it back to like Enterprise red tapes around I cannot I cannot have a day dedicated to this one because I need to hit it that line. I need to accomplish that just break it up.

So, how can we make sure we bring the leadership along to make sure they see the value of that and how can we give the conference for the teams to say you can actually this is how we should be approaching it and whatnot. And my last point on that is playing it back to the feedback loop side. I would say feedback loop starts in every process.

The culture the thing that I'm I work very closely is the shorter the feedback loop is better it is if I could if I don't understand a requirement that the business is giving me my feedback loop starts right there. I need to ask all this clarifying questions to go so that I'm not building a system on top of assumptions. Now, let's take you the next level when I develop in my write my unit test.

I don't make any assumption. I need to test it slowly X into that the last leg of it which is very you're putting into a system. The system is having observed observability characteristics to provide you that level of feedback and if the user can provide a feedback, which is ultimately what the devops loop is good, you're getting feedback from all the aspects of application people and the system to get into continuous increment And Brandon City containers were born absolutely but it's more than that too.

It's context. I can't just give server logs from how something's operating in a production context to the development organization. That's not they don't have the right context to consume that information and get value from it the translation of the information through the systems you're using to present it in a way that is Meaningful and impactful for the stakeholders who are viewing it ideally in the system of choice that they already use today not I don't want to force a Dev team to start logging into an APM solution to begin to see what's going on in their observability space.

I want that information published directly into their CI system or whatever else behind the scenes. You know to your point Brian I the context in the environment that something happened in. Because that's also changing right we think about continuous delivery of smaller and smaller right code.

We're using SAS services and cloud services. And those things are changing. Right?

Our infrastructure is code and software that we're running on all those are variables and more a lot more in addition to our own code our own systems. Those things are changing and whether it's hours or days or weeks from when something happened or minutes, you know, just giving someone a log those conditions may be entirely gone, you know, if there's a serverless, you know, kind of oh the worst case scenario for developers, we have this intermittent issue in production. Fix it.

Yeah, no data for you no guidance to offer but it happens every so often and they have zero ability to replicate it in any of the QA systems. So maybe the idea that we have an environment business static here and we can draw the line exactly. It's just not there anymore.

So you've got to be able to collect the data the context and draw the threads with two Charities Point earlier without understanding what questions you need to ask ahead of time that's have that's observability and it's very very key to having a good quality effort. You know that we have a comment in the Q&A area participant asked or really more of a statement. So we need to have an observability strategy.

Is it by project program at the York level? Where everybody? Yeah, I'd take my observability where I can get it.

We're gonna coach on that right? You wrote the book again. Tell us you know, how do you set up an observability strategy?

Doesn't great question. And in fact, I just finished writing and ask we have a column called askmus Ali where people write in and ask their questions and and I just finished writing one to answer the question of what doesn't observability team do right because I think a lot of it starts with the build versus buy question because a lot of teams are used to running the road, but frankly, I think in this day and age especially with Cloud native stuff, you know, every company like engineering Cycles are the scarcest resource that you have and the more you can send your spend your engineering resources on your core differentiators. The thing that make you you as a business the better right the more you can Outsource, you know, all the other stuff the stuff you have to do or get the stuff that you want to do the better.

So like also like you typically when you're running your own observability stuff, you're ready on the same infrastructures, you're running all the other stuff. So when something goes down now, you typically have two problems, you know have the stuff to observe the stuff that you want anything bring up. So like, you know, obviously I'm biased I'm a vendor but you know, I I think Usually your strategy figure it starts with you know, figuring out who you want to use.

This is where I think that an observability team is really well placed to do what I think of as vendor engineering which I think one of the most challenging and underrated skills and Engineering which is just you know, they sit there you there between your organization engineering team and all the vendors out there and say and they decide what what to use right what will meet their teams needs. They invest in that they you know, they handle upgrades and and clients and and they provide a supportive set of tools to the engineering work so that it's easy to do the right things and hard to do the wrong things. Right?

Maybe they write libraries or or Integrations that mean that you know, if you just use a template to add a new service and a bunch of stuff happens magically, right? Maybe they need to write their own, you know integration or or you know implementation of something because it makes it that much easier for their but the end of the day it's about you know, it's also about like having an observability culture of you know, blamelessness and blameless postpartums and making it okay to ask questions, you know, you know, okay to okay to fail and you know, let's figure out how to, you know, fix it together, but I think that Convincing Engineers to instrument their code using Hotel setting good patterns, right? You don't want to work at a company that has 10 different teams and 10 different ways of logging right here 10 different libraries, you know, providing some consistency and some guidance.

Everybody in your organization shouldn't have to be an expert in observability. Right? They should they should know enough to do a good job, but it's it's the it's the job of your observability team.

Also as much just said it's it's also a you know, your observably teams to be those deep experts and to make it clear to to leadership why it's so vital to to everything that they want to get done in the world, right? Yeah. I think that there's a real role for ad I think they play the role of developer Advocate and Champion within their company is as much as anything but it's really just about you know, being those that those domain experts doing a lot of Education doing a lot of guidance and writing writing the code and to make sure that you know, That you're doing good job, and that not everybody has to become an exp.

ERT no, but that's an interesting point and an important point and that is Someone has to be the champion. Someone has to be the advocate right look in my mind having I mean Tech a long time one of the biggest changes I've seen is you know where we used to hire sales people to sell. to customers in a large number of organizations that role Or not that they're replaced but we don't have as many sales people.

We have Advocates. We have evangelists right we have Devrel, and so forth and the same way we have sort of let's call it external devrel who are selling. com is for devops Advocates within an organization to be successful.

They needed air cover. right if they didn't have an executive who was sponsoring it or at least you know advocating giving it some air cover the chances of it being successful and well taking root word swim we see over and over and over is that there's this breakdown of communication between engineering leadership and the rest of the org because It's for headcat and to hire people and it's so hard to get money to like to make it someone else to pay vendors. Right?

Like it's easier to spend to a quarter million dollars and hiring two new people than it is to spend, you know, a tenth of that much just spinning up a new vendor which is insane. Right? Like the inside the incentives like aren't really you know, there's there's a break to the communication.

I think most engineering leaders don't understand how to make the case in terms of like cost benefit and like and like actual you know costs are just going to battle for sums of money and and this this results in so much duplication of efforts which duplication of work so much distraction so much, you know, just like yeah, there's a lot of ways that it breaks down and I don't know go ahead. I was gonna say like, you know, I I worked at a large financial institution here in Canada for a number of years doing like basically dead devops. Advocacy.

We were running like a coaching team to get people like excited about devops and and this stuff. The only way we could be successful like we wanted obviously like people to get excited about devops through like a Grassroots movement, but there was no way we could absolutely be successful unless we had that leadership support from up top because you know, if you if you don't have someone that ultimately says they'll shout shout do the devops where thou shalt do the observability. People don't have that kind of incentive.

Unfortunately. I was I've seen it at a previous job at this financial institution. I saw it at my and my last job running the observability team.

We were lucky that we had support for for observability for doing things the right way because otherwise You know, we wouldn't have gone in the right direction. and how quickly we forget because when we were doing agile transformations 15 years ago anybody who's switched over to Agile for the first time those first two or three Sprints are kind of a disaster. They never went.

Well, there was a learning curve everybody needed to figure this out. And as teams got proficient at it the value really started to come out of that change in process that change in mindset that change in tooling that needed to happen. This is what we're talking about here as well with observability.

It is a fundamental shift and how you're using a lot of your existing technology and acquiring some of the little gaps that may exist many large Enterprises have the tools. They need already. You don't need to buy something new.

You need to change how you're using what you already have. You're not going to come back to you. Yeah, so from from touching I want to give a quick feedback on the leadership thing.

I think this is this is being a bottleneck or a constrained at any change like any technology change your process change. That means we need to use that. I believe there's a key role by the leadership.

So recently based on these all this lesson. We started doing something focus with the leadership first defining otherwise, let's build a rallying cry and why we are doing this change, right? And so that you get them all excited.

This is why everybody should be excited to do this. Everybody should be excited to come to the work come to the office or being remote work here in this new culture to you have to drive the excitement behind the wine too. So before we touch a team as the champion team to introduce all this new techniques or new tools or whatever that what are we trying to accomplish it.

The first thing that we started doing is just bringing the leadership community and defining the problem statement and defining the charter around why we are doing That phase of a lot of confusion and alignment and all those things and let's define a chart of why this particular organization is monitored of observerally or devops or go go Cloud native or whatever that objective could be. I think it's important for us to Define it and get that alignment first. So once that's done The Grassroots movement will be very successful the teams are ready to do it because if I could relate back to my career times, this has been an extra work that I always did have after my 95 Channel like a cool learn about them.

So don't know about cloud and come back and show to my team members into what it is. How do we get away from that and make sure this is part of your job and you can you get paid you get recognized for improvements and continuous improvements around that. So once you have the leadership support and they then we can start with the Grassroots moment and the leadership can go back can change it down to their to their team leadership and whatnot so that they are appreciated about doing it.

Otherwise, there is no incentive for any of this teams to to continue doing or challenging and existing status girl, so they have to keep the lights on and they don't get anything extra by just doing this something different. Hey, I introduce this instrumentation in my code. So that next time in this incident happens.

I can know for it. How are they appreciate it for their holiday into device for that. So I think the leadership support is very big key, so they have Book it might behind that so I would I would push hard for having a conversation with the leaders or the sponsor bringing them into the room.

Let's work together. Let's define the charter the start from there. Now, you have a clear goal to go after and you could be agile the small agile Small Engine where I'm going to try this for three months.

Let's see if it's gonna work. Let's define the outcomes what I'm going to accomplish with three months and let's go all in on that and if it doesn't work, we'll come back to the drawing board. Let's figure it went wrong for the next step and so on so forth.

I think to add to that it not only like, you know bringing the leaders into the room to get buy-in but also like bring them into ground level like going back to Brian's comment about the the agile Transformations that we saw in many large and small organizations. Often times you'd be in situations where you know, like you'd have these agile teams stood up and they got to a good Cadence and then all of a sudden some exact sweet swoops down and basically crafts all over the entire initiative and then all that good work is like gone to hell right? So you need to like constantly keep that you need to keep that conversation going right so that those Executives actually know what's going on in the ground so that they don't run interference whether whether it's accidentally or on purpose to like, you know stick with the mission, right?

The greed the greed. Hey guys, I need to pop in here because you know, we're we're 52 minutes in we give away four Amazon gift cards on every one of these round tables and I have to announce the winners so that the team doesn't get the inevitable. Hey, I thought you said you were giving away gift cards.

So let me put on my glasses so I can read well, I can't read that far away. But it looks like the windins are little a Judy. I'm gonna say yes, Marie L and Christopher with a k m.

And our our webinar team will reach out to you and get you guys folks your gift cards. Thank you very much for Attending today's Roundtable guys in the time left. Right, I we don't you know try sentence is a pleasure to work with us as a sponsor.

We I have to encourage them to to promote their company on these things, but I want to ask Ryan. I want to ask charity. I want to ask you on it and Adriana we have time I'd like to finish up with you.

Tell me specifically in your organization. What are you doing in the case of the three with not Adriana? What are you doing for your customers to to strengthen that intersection between testing and observability whether it be product-wise or you know Consulting wides or what have you and then they trying to talk about it.

Maybe what's happening if you are allowed to well actually, yeah, you're like that to you. I'm still think you guys talk about it like step as well. If it's okay charity.

Would you mind if I asked you first to kick it off? Yeah. Absolutely.

Sorry, what was the question? Exactly? So what and honeycomb what else specifically can you tell people a point people to a product or service something that will help them navigate that intersection between testing and observability.

Yeah, I would recommend that you start with something that we call build events, which is a great Library. It's a it's a thing that you can use to use our free tier to instrument your build pipeline your cicd pipeline so that you can see where the tests are slow or where things are flaky or where or whether like it's getting longer over time or you know and which is really because I think this is critical part of this is that feedback loop, you know that it be like the rebel that the feedback loop of deploying be short and tight and you know, so your tests are running for days or whatever and you know, I think tea it's really smart for teams to set like an upper bound like we don't want it to take longer than an hour to deploy, you know, I recommend 15 minutes or less but like shorter the better so instrument your pipeline as a trace and you can see you can see where all the time is going. It's brilliant.

It gives you a really nice like dip your toe into the world of observability and get something really materially useful back. Excellent Brian. How about you?

Yeah, so working at tricentus we Advocate strongly for continuous testing for the digital business as people are migrating towards Cloud platforms towards container-based architectures new types of challenges are coming in particularly around figuring out what the heck is going on as containers pop in and out of existence. We have a completely automated low code or fully code list set of solutions designed to help you test all of these different aspects of modern application delivery stacks and are tightly integrated into an observability ecosystem through our Partnerships with a lot of the leading APM vendors, but as well as the open source Solutions like Prometheus and open telemetry. Excellent.

Excellent Beyond how about you? Yeah, so I think we touched about the instrumentation the tool sign, let me share another perspective. So if you feel that you are at an intersection of performance testing and observability, you're not you're not a choice to pick either or it is not intersection where you want to go in One Direction Another is that is a point that you have to operate on both the sites.

So both both the systems would need to be in play your continuous testing should be maturing. You should get into more closer towards more Automation in testing world and getting into that more like a shared responsibility plus observe it as a choice. So it's normally there are so you make sure at any given point so this I've seen this train from when there's more automated testing the unit testing doesn't have to be there.

So it's it's not everything has a purpose in it and you have a build on top of that same applies to Performance testing or testing and obserably this is it's not me there are so make sure you challenge that second to any of those instrumentation if there is a black box solution within your company challenge that I think the purpose of all those things is not to make it as a black box to make that information or measuring visible to everybody. So this is something a team is providing a service within your organization. I would be very curious.

I could I would say go challenge them. See I wanted to figure out how can I read this laws or how can I see this progress towards it so that you're the developer Engineers doing the work gets that inside while doing the works. It's not like giving to somebody in just getting getting some feedback that later points later point of the stage.

Actually, thank you. Adriana how about you? So, um, I would recommend so when one thing that I wanted to talk about briefly is something called Trace based testing, which was originally presented by Ted Young in kubecon, North American 2018.

So be sure to check out the the talk recording because it's kind of a it's a cool concept. The idea is basically using your traces to help Define your test cases and there is there are two products out there that now allow you to do that one is called Trace test, which I think came out in May of this year. And then there's another one called malabi and both of them allow you to basically take your open Telemetry traces and Define test cases out of them, which is super cool.

It's it's a really good way of bringing testing and of having that intersection between testing and observability the other thing that I wanted to touch on. briefly is this idea of basically, you know observability is is a tool not just for sres not just for developers but is a tool for testers and so, you know, As a tester wouldn't it be great to like not just discover bugs, but also figure out what's causing them and observability gives them that so then they can provide the additional feedback of okay to developers. Okay.

I know what's causing this or if they don't know what's causing it. They can go back and say hey, you know what you're missing some instrumentation in your code. This is what I think you're missing.

excellent guys are almost out of town. We have a few minutes left Mitchell. I wanted to give you the last word here to wrap it up.

I posted this in the in the chat. It's just fascinating them. Just I just learned so much listening to our panel the thought I had was, you know, we don't talk about observability and testing together a lot.

But they're like a Reese's Peanut Butter Cup. Hey, you got observability on my testing you got testing on my observability. You know, they go great together matter of fact, they go hand to hand.

You can't you can't do it one without the other really so I think I think this will help all of us advance that conversation in our organization. Absolutely. We are at the top of the hour charity Bjorn Brian Adriana.

Thank you so much for being on our panel today. Mitchell is always thanks for being the the anchor leg here. And and you know holding this together most of all thank you to everyone who tuned in to watch this or is watching it on demand afterwards.

We invite you to check out the next devops on down live round table as well as our bi-weekly videos on it on anything related to devops. It's always great. Again, also thanks to our good friends and tricentes for sponsoring this and making it possible until next time.

It's Alan schemel. We're out. Bye.

Okay.