Advancing Testing in Continuous Delivery: Unveiling Challenges, Strategies, and the CD Foundation’s Initiatives – The CD Pipeline EP 8

Hi, everyone. Welcome to CD Pipeline. CD Pipeline is a joint, uh, production of Techstrong Group and techstrong tv, as well as our good friends at the CD Foundation, which is of course part of the Linux Foundation.

And it's a once a month show. And, you know, once a month's, not enough, frankly, but for at least one time a month for about 40 or 45 minutes, we take a deep dive, we delve deep into topics that are germane to CD and CI I C D and software pipelines and, and what the CD Foundation is doing to enhance and help organizations and developers who are, and not just developers, but DevOps people and SREs and platform engineers and people who are in involved in delivering software. So, with that being said, let me introduce you to this month's show.

This month's show of CD pipeline is advancing testing and continuous delivery, unveiling challenges, strategies, and the CDs Foundations initiatives. And really, I couldn't think of a better panel to discuss this than our panel right here. Let me introduce you to them.

Now, I want to first introduce a first timer to any of our text Strong shows, I believe, and his name is Oli Lenmar. If I mispronounce it, I apologize, but I think I got it right. Ollie.

Ollie is the, yep. Ollie wears many hats, but he's here wearing the hat of the CTO of Test Cube today. Ollie, welcome to our show.

Why don't you give people a little bit of background? Awesome. Thanks so much.

Um, yes. Uh, great to be here. I'll, I'll do really, really fast background.

Uh, worked with testing for many, many years. I was the founder of a project called SOAP P, which was a big API testing tool for many years. And then fast forwarding from there, worked at a company called SmartBear, which is big into testing and quality assurance tools.

And then fast forwarding from there was, uh, part of storing test cube, which is, uh, an open source tool or an offering a project, trying to cater to tester's needs in the modern cloud native world. Uh, and based on kind of all the experience with stuff I've done before, and fast forwarding from there, test Cube joined the CD foundation earlier this year. And we've also, uh, an enhanced the CD events specification, uh, for, with testing related events, just based on our experience and the needs that we've seen in our community.

And I'm glad to be here to talk about that. Fantastic. And thanks for joining us.

Next up, I want to introduce you to a frequent guest, not only here on CD Pipeline, but on many of the Text Drunk Show. She's also the co-host of Techstrong Women, and she's a good friend. Tracy Reagan from Deploy Hub.

Hey, Tracy. Welcome. Welcome, Allen.

Thank You. Thank you for having me here. Yes, I am the, uh, c e O of Deploy Hub.

Uh, prior to that, I w worked for a company called Open Make Software. We had a product called Meister that worried about managing the bill process. Today, deploy Hub is an evidence store.

So all of the evidence that we're getting generated from testing, from security, from the DevOps pipeline, from the supply chain, or pulling it up into one store so we can start seeing an organizational security and use profile. And I'm really into CD events. People, you know, I think people will say, don't take, don't take Tracy's name three times, otherwise she'll appear and tell you CD events.

Very cool. Um, our third panel member needs no introduce introduction to anybody from the CD or C I C D world. He is the, uh, founder of Jenkins, sun engineer, uh, formerly of Cloud Bs, and now CO c e o of Launchable, Inc.

It's, uh, my good friend Koa K I'm gonna do my best, Koa k Koa Kawa, or kk, which is what I always call him because it's easier for me. Kk, welcome. Just so we get it right, Koosa cake, say your last name for us.

Oh, um, it's actually a very common name, Japanese name, um, which, which kind of tell that I'm, uh, I'm from a common or humble or origin here. But, um, yeah, so, uh, but you know, I, um, I've been passionate about the developer productivity for the past, I guess 15 years or so. As you said, I probably am best known as a guy who created Jenkins.

And so I, you know, I unleash a lot more automation. I can, I don't take all the credit product, but I at least a lot more automation to the world of the software development. And as I was talking to the, the software development teams who is using those things, I realized that the next challenge that picked my interest is, you know, that because of all these additional automation created a lot more data.

Um, and then that's creating a challenge for q people, right? Say suddenly, I, I used to where, where people are used to running, you know, like a test every night and seeing the results, now they are running like 10 x a hundred x. So how can we help that?

And also the more data is actually opportunity, right? Because they, in places where you have a lot of data, you have opportunity to apply machine learning, ai, I thought, well, this is the worst challenge and opportunity. So I'm trying to solve the mess I created in the world in some sense.

So that's, uh, that's what I'm up there. Excellent. KK may have come from common, uh, background, but of course you're one of the giants in the field, and thank you for joining us today.

Last but not least is my co-host here on CD pipelines. She's another woman who wears a lot of different hats and she'll tell you about them in a second. Uh, Lori LaRusso.

Laurie, welcome. It's good to see you, my friend. Oh, it's always good to see you, Alan, and thank you to all of the panelists.

So, yeah, so I am the open source program manager at J Rog and the outreach chair at the C D F. And so this panel is a big boon for us to have kk the founder of Jenkins, which is the first, uh, project donated into the C D F. And then to have Tes Cube, one of our latest members to join.

And then of course, Tracy, who has been a supporter from the beginning. Uh, this is just really cool and to talk about testing, which I think when you look at the CDFs mission, which is to, uh, help, uh, release fast and securely, we don't have testing in that. And so this is gonna be really interesting to see like where that plays into the entire pipeline and into, uh, the process of delivering software, uh, fast and secure.

Absolutely. And Lori, you're also C N C F marketing Chair. Yeah, I'm the marketing chair of the CNCF F Yes, just, I'm just all around everywhere.

I'm Trying to keep you humble. I'm trying to keep you humble. I should also imagine that Lori's usually globe trotting around the world somewhere when she joins us on these, but she's actually home today with her two puppies, and they're misbehaving a little bit in the background.

So if anything untoward happens back there, just know that that's what, what it is. And we're good. Yes, That was a box of tissues.

Now. I, yeah, that Was a box Of tissues. Lunch, Lunch happens.

It happens. But, um, hey, you know, this is what remote work is all about. So guys, look, I, I think KK kind of hit it on the head, right?

We, we, we, we jumped headfirst into the CI and then CD and ci I c d as we call it, kind way of developing software, which is so germane at the core of what DevOps is. And very quickly we realized that the quicker we release software, the quicker we have to test software. And the more we release, the more we have to test.

com 10 years ago, you know, there was, there was a school of thought that said, no, no, we don't have to test more before release. Let's just release it. And, and, you know, we'll get, we'll get feedback from the customers and we'll get those feedback loops, loops going and, and we'll just release it again better, right?

And so we call it the kind of the, the, the, the forever beta cycle. Like forever young, you're in forever beta, and you know, you're releasing half bake software to, to customers counting on them to tell you, you know, to do your testing for you. Well, that's not really a formula for success either, or maybe it is.

I I don't think it is. And, and so, you know, this whole idea of continuous testing and marrying that testing cycle to our release cycle and our software, you know, s DLCs and our, so, you know, the whole way we're doing pipelines really kind of grew up around the idea of continuous testing, and we've gotta match the testing to our development patterns and cadence. Um, I mean, Oli, is that what's behind Test Cube and what's driving you, or do I got it wrong?

No, it's, it's a great, uh, great point. I think what we are also seeing, if you go back 10, 15 years, uh, when, when you release an application, it was often a monolithic kind of release, right? You, it was all baked into one big process, uh, and then at the end you deployed something.

But today, applications and infrastructure are so fragmented. There's all these components being built asynchronously, right? This team is building this, these backend APIs, and this team is building that, and this team is using GitLab, and this team is using Jenkins, and it's all very fragmented.

And testing there becomes a real challenge, kind of how do you have a coherent testing strategy across all these toolings, across all these unsynchronized workflows, right? Uh, where teams are releasing things, maybe not as coordinated as they were, or when everything was kind of baked into one big thing. I think that's kind of where we at least, uh, wanted to come in with Test Cube to kind of solve some of those challenges, but definitely also see the events and the testing events, and see the events to help coordinate more asynchronous, um, testing activities that could be part of those workflows and pipelines.

So I think that's, that's kind of the, the big challenge. It's, it's, I mean, and every, I think big challenge is that every team and every organization will be doing this a little bit differently, right? There's, uh, it's something that often has grown, uh, over, you know, many, many years.

Uh, very few are kind of starting from scratch to put this into place. So you al also have to be able to adapt to what they already have, and maybe you have some older teams using, you know, uh, software that's been around for a long time, and then you have other teams using more modern software or newer software. And then once again, I think CD events and these kind of initiatives really help to try to glue these things together and try build, create coherent workflows where testing can be a, you know, a, a vital part of that.

So I'd kinda like to stop you here, and maybe Tracy, you can kind of fill us in. So you've mentioned CD events, uh, quite a few times, and because I'm part of the cdf I know what that is. Uh, but Tracy, can you talk to us a little bit about CD events, interoperability and why testing should be a component of this?

Oh, absolutely. That's a great question for me. Thank you.

You know, I think the, the way to describe what we're doing with CD events is disruption. Uh, we have, um, relied on workflow files, uh, for many, many, many years. What, when did you start, when did, when was Jenkins born?

Uh, But it's been around a long time, and there's probably somewhere in the ballpark of 30 to 40 million, at least 30 million Jenkins workflows. Um, so we've, we've come to a point where disruption is needed, uh, because we, as o Oola just described, we now have all of these, uh, a decoupled environment. We, instead of application teams, we have feature teams, which creates the need for more and more workflows, but static workflows are starting getting in our way, uh, and they get in the way because now we have new tooling, we have new, more, um, uh, better tooling than we had when, uh, we, when everybody's first started, uh, adopting Jenkins and building their, the, these workflows.

So disruption, agility, um, and the need to be able to drop in any tool that you need in the pipeline because you have now a need to say, I need an sbo, I need signatures. CD events is the way to build that agility into the pipeline. And all you have to do is think about cloud events because it's pretty much, uh, a cloud event, uh, methodology.

Uh, and what we're doing at the CD events, and what Ole and his team just said was, define the, the use cases for each of the events that could occur around testing at a high level. Now, keep in mind, there's all kinds of testing too. When we first started doing Jenkins, we didn't have so many different types of testing tools.

So CD events is the way for more teams to be able to adopt more tools in an easier way. Thank you. Okay.

And can, and I think also, Oh, go ahead. No, Go ahead. No, I think going back to what, what, what Alan was saying is that, uh, you know, while back testing was done in production, I think now we do testing at a, at a lo at a many different phases in the pipeline, right?

There's obviously, there's unit testing, uh, closely to your build, but you might, and then you might wanna do functional testing, integration testing, but you have load testing, security testing, and all these kind of things that might happen at different phases depending on how you assemble your applications. And once again, these are often decoupled, uh, workflows. And once again, CD events can kind of really help tie that, make that possible, and tie that together.

And if you're building a new tool that you know, fits into one of these niches, uh, if you adopt CD events or tie into that, it'll be much easier for, uh, someone to, to actually put you into their workflow and not, uh, be dependent on the static definition somewhere. It's an adoption issue, right? It's where it's allowing people to be able to adopt better and more tooling in a faster way across the enterprise.

Agreed. Cool. So, so kk, I have a, I have a question for you.

So, um, interoperability seems to be, uh, and city events seems to be like the hot ticket, right? And so you as a founder of Jenkins and now launchable, like how do you see this shifting in the industry where we're telling developers they need to shift left and they need to add security, and here are things that they can do, like, do old school like farms have to have new testers? Like do they, does that need to be a job role?

Is this something that the developer can do? Like how do we sh help shift the mindset of, it's not just deploying fast, but it's deploying fast after you've tested to make sure things are actually doing what they're supposed to do, and that you're not going to create some big malware issue or something crazy, uh, getting released into production? Yeah, I, I mean the, so the, the testing did spread over, I see.

I think, uh, maybe the Alan painted the earlier fixture where the testing, I, I remember the world image testing was a gate, it's almost like a manufacturing invent really, of, uh, oh, one side, we produce a software and then they check things and eventually they give us a green light, and then the product shipped. And that was, uh, you know, that was, uh, 20 years ago or so at, at this point. So it had spread to the left and to the right.

So, um, and a lot more testing does happen close to the upper, a lot more testing does happen in production. So it's, it is a thing. And then, uh, but, but it still remains like there's a middle part of it is still like the covid testing, it's to end testing, whatever that takes a long time to run.

Um, so I guess I'm, you know, then that's the part that I still somewhat, I guess, still interested in, you know, like this cause the frequency goes up. And then all I talked about this asynchronous nature of testing, it also has this side effect of like compound effect of, uh, when, when things start breaking, it's, it's much harder to track things down to where things are going. And that always felt like a very labor knowledge intensive work that only like exporting the team could do.

You know, like the error message just is one eye. And some people who know what they're doing to say, yep, I've seen that before. I know this is usually what, what happens.

And then, so that's, you know, how can we, that that's a part that still fascinates me. Uh, and then like I continue to push the test the left and right, like what are the techniques and like a tooling assistance we can do. So there's a lot that excites me in those spaces.

And that is the reason why, um, we're having discussions about artificial intelligence in testing, right? Yeah. There is a disrupt we have disrupted ourselves, uh, which is why we have to disrupt the CD pipeline and we have to disrupt the way we test.

We have to disrupt the way we deploy because we have moved into this highly decoupled architecture now. And what we have to start thinking about is blast radiance. If we update a single service, even though it may, um, maintain its a p i contracts, that doesn't mean it didn't change the way it was deployed.

Somebody said, I'm gonna change these key value pairs, and then it gets deployed out and it may not work in a mi in a Kubernetes environment. It may not work in all namespace, namespace the same or clusters the same. So right now with the way we are managing most of our testing, we're, we don't have the intelligence to say, if I have one service that's changed, what is its, what is its glass radius?

What is its risk of, uh, score, so to speak? And building in more intelligence into what those clusters look like, how those parts and pieces fit together are super critical to both the testing steps of the CD pipeline and the deployment step of the CD pipeline. And the, remember testing isn't just done in one place, it's done in multiple places.

So we have a lot more complexity, and this is the why. This is why we need to start thinking about gathering data, starting to understand models, what, what is normal, what is outside normal so we can have these trends that we could begin seeing and the testing tools can start producing that for us. Yeah, it used to be that the test output outcome from the test or something humans looked at directly, and by still by and large, that is a case.

But now the volume is going up to the point that I think that's starting to be impractical. I'm starting to see the emergence of having machines look at this data for us to do some sort of a pre-processing, and then it could be as fancy as ai, but a lot mundane, a lot more mundane things could still cut down the amount of the cognitive workload that we put on the developers a lot. So, um, so that this shift kinda excites, excites me, um, especially in the context of this, you know, the large language models, which is a lot better at dealing this, this way in the world of text, which is fundamentally the output from test is kinda all about, right.

It's like a world of text, megabytes of log file. And I'd just like to pick up on one of the other questions Laurie was around, is there a new role there or, uh, when it comes to testing and, and managing tests, uh, in like this de kind of decoupled environment and we're trying to shift less testing left to developers, but I don't know, uh, I, I've worked with a lot of developers over the years and it's a tough, the tough, it's an uphill battle. Uh, so I think definitely, uh, that tester, uh, developers are more maybe quality conscious and that testing has been, you know, more integrated into their immediate tooling.

But, uh, I think the kind of testing we're talking about, talking about here, the decoupled testing, a synchronous kind of testing related, uh, events that might happen as part of your workflows, I definitely think there's someone needed, uh, in especially, especially depends on the size of your organization and team, of course, but someone that is responsible, uh, for understanding and understanding the blast radius as you were talking about Tracy, but under also understanding and, and maybe, uh, managing the quality and testing related, uh, things going on in your, uh, infrastructure. And I don't mean unit tests, but more the synchronous functional integration end to end, you know, all the kind of other tests that you might be running as part of your workflows. So I, I do think just like there are build engineers, uh, there are platform engineers, uh, maybe I don't know what the, what the, the title would be, but I do think there is a need for someone that kind of, you know, has the bigger picture of how are we running tests in our organizations, how are those linked to our CD workflows and are how are we managing the metrics we're collecting from all of those tests, and are we making the right decisions based on those?

So, uh, long answer to that question, but I do think that there's a need for a role there, even if devel even if we are shifting left. Yeah. And there's always QA engineers, but oftentimes QA engineers are, uh, seen as individuals who, who are really responsible for, for managing end user testing.

They may have what we used to call model office testing, and they were responsible for the testing tools. But some of the tools today are really about automation, um, and building the, building the, an automated testing suite into the CD pipeline is very different than putting a user in front of a screen and saying, did we get it right? Did you find anything?

And oftentimes, QA engineers are people who are doing or participating in that role, and DevOps and engineers and SREs, they all kind of assume the other roles. And it would be great if organizations saw that there was a, a individual who was like a DevOps engineer, but was a, a quality engineer who worked on that side of the house. So, you know, I, another one of the shows we do here on Techstrong TV is called DevOps Unbound, and that's sponsored by tricentis who is pretty big in the continuous testing space.

I've learned a lot hosting that show over two years. I, I think Tracy, to your point, yes, there used to be a QA engineer who who did testing. That's what QA engineers do, right?

They do, they do QA testing, but a lot of those user, you know, you have companies like Apple Tools for instance, that do automated, uh, visual testing to test like what a user experience would, you know, how they see an interface or what have you. I think what we've seen is those, the, the quote unquote QA engineer has moved from a person who does testing to a person who scripts automated testing, right? That's where the money is now.

That's where the emphasis is. Now, I want my QA engineers not to actually do the testing, but to set up what tests should run when in a continuous environment. And, and that's a much, that's a different role than the role you are talking about Tracy, which I think was the traditional QA person.

Um, where I see the problem though is as part of that shift left, we're saying, Hey, if we can automate this, let's just shift it left, like we do everything else, right? And the developer could kick it off as part of his, you know, as part of his, when he commits to pipeline, it's like a Rube Goldberg machine, you know, we, we put it in there and a ball drops in here and that kicks off a test. And when that, depending what that test comes up with, it kicks off something else.

And we work our way through the, you know, the, the machine guess, and no, yes and no. I think what we have found is that developers like to develop code QA folk, like to make sure we have a full testing, uh, coverage, right? That when you talk, when I talk to testers, they want to know, right, what are we responsible for testing, right?

And do we have adequate coverage to have a high degree of, of certainty that this thing meets our quality standards, right? But we're moving so fast, we've gotta do that in an automated fashion at some, I I, you know, is this, this to me calls out for ai, but I've always thought testing in a DevOps world calls for automation, right? It's the only way we were ever gonna do testing in a DevOps world.

Um, but, you know, is that the way it's going down again, Lori? No, I was gonna say, and I think that leads to this whole idea of integration, right? Like how when you're, when it's both on the left and on the right, when it's before production and while things are, uh, in production, like how do you really integrate this?

Like I know you, like, you both have products, but like, how does this, how does this actually work? Like in the real world where when you have all of these siloed operations and then something goes wrong, like how, how, how is, how are you all with testing, bringing it back together and resolving things in a fast way so that the pipeline stays secure? Well, so, so, uh, I so many thoughts.

Just when you said those things, cuz I've been a lot of test, uh, testing conferences and of course well aware of tricentis and, um, I mean there's, and I, I don't wanna turn this into a testing discussion, but there's, I think, uh, automation is great, but it's definitely not enough, right? For, and I think, uh, the best testers are those who can just sit down with your application and play without round with it and find what breaks, right? I had a friend who always, when he was on his flights, he was always able to break the, the, the video screen in front of the seat in front of him.

He was always like, I don't see why, what I could get this working. And, and, you know, five minutes later it crashed and then he had to ask the, the, the people to reset it for him. So, so I think, uh, what I'm trying to say, automation is great and I think you can use it as a safety net and to a certain degree, uh, as, as you can define, you know, measurable metrics, uh, for some kind of quality of your product.

But you should never ignore and downplay the, the value of exploratory testing or manual testing or whatever you wanna call it. You talked about QA engineers, uh, uh, uh, and I think one of their big skills is of course they can write automated tests to validate, you know, that if I do this, this happens, et cetera. But it's also really thinking outside the box, what, you know, how could people misuse this application or these APIs or whatever else they might be testing.

Uh, so I know it's, it's a little bit of a tangent from what we're talking about, but I think it's just such an important thing to realize that as much as you're going down the route of automation of testing, which is great, don't forget about actually putting someone in front of your application to play with your app or whatever you're building and, and, and try to, you know, provoke it to do things that nobody else would've thought of. So, sorry, there's a side note, but it's, uh, passionate about it because it's just so true. And I've been in so many situations where everything's been automated, but then someone just comes along, what if I do this?

Then click over here and then don't do that and everything breaks. And I'm sure we've all seen that. Yeah, I have that exact, it, it happened to me.

I, I was, uh, working for, um, u p s, we were building their hub and feeder system and we tested and tested and tested. We had some of the users come in and test, and then we were gonna do it in a, in with the actual, uh, warehouse guys and the truckers. And they came into the room and, um, kind of said, Hey darling, I hope this is gonna be a good day for you.

And one of 'em, uh, started working on it, turned the keyboard over and sat on it. And at first the whole system was like tombstone immediately I took him maybe five minutes, He sat, I didn't test for sitting, be sitting on the keyboard. Yes.

He talked about the, you know, the, the, I guess the never, uh, the manual testing. So like, let me talk about little bit about automated testing. Cause I think the making the automated test easier has undoubtedly like a key front of innovation for the past multiple decades.

And that I think that still continues. And then the automation in testing is hard, the more stuff is coming along the way. So, and then as you know, the computer got more abundant.

I think the value we can get out of automated test also increased a lot. So I think that, you know, I think that the, the shift is definitely in my mind, shift is happening toward that direction. That doesn't eliminate one side completely, obviously.

But, so that's, I think, um, I think it's the little surprise that either a lot of like innovation around applying, let's say AI to, to do those hard testing or like making it low code, no code to test. You know, like I haven't yet seen any level of result. I have enough tests, so I feel comfortable with my quality yet.

So yeah. Yeah, there's so many different kind the, it's kind of testing, but you're right post k, the, the, the l the lion's share of the testing, we really have to build in and make it automated. As Alan pointed out, developers have, they wanna code, they don't wanna set and run tests, they just don't.

And the more we can, the more we can automate through the pipeline, the better off everybody's gonna be. And you know, pen testing is not necessarily done in a test environment, it's done in product production. So there's all kinds of different flavors of testing.

Now we have to think about it. It's not just the QA engineer sitting there with the truckers and making sure that the system does the trap. It goes so far beyond that.

And that is the key here to our discussion because with so many pipelines that we have to update, it is a barrier to adopting new and better testing, uh, methodologies and testing tools. And that's, uh, really, I always bring it back around to the CD events project because if you're one of those developers out there writing scripts to automate, why don't you join the CD events project? So you never have to do that again.

And that is the whole point. So, Lori, I I've got a question for you, right, as the c d F person here. You know, we've spoken a little bit about CD events, but what, what specifically is CD Foundation doing to advance the state of testing and continuous delivery?

Well, I think as we've highlighted, C events is a project that is just wide open for testing to be brought in. And I think all of our projects are so well-rounded and working on top of each other to make sure that they are staying at the forefront of what is happening. Um, we talk a lot about security, which is why it's nice to talk about testing because that's part of it, right?

And I, and again, I think it's part that we don't highlight enough. So, um, so to know that testing is being brought into CD events and other projects are being brought into CD events, I think this is where, you know, it's just that snowball effect. And before you know it, it's not something that we're going to have to like say, oh, is testing, like, no, no.

0, like that was already taken care of. Now we're just working towards more projects, more adoption and more other ways of checking out what's going within the pipeline itself. So, um, I don't know, correct me if I'm wrong, but I think that's kind of what the value of working in open source and working for a foundation where it's vendor neutral and it's everyone coming together try, try and solve a common problem across the board.

You just start getting all of this, uh, ingenuity and inventiveness and people get very excited. Like people have gotten very excited on this call, the more we talk about it because you wanna make sure that your piece of the pie is included, right? So I think it's just becomes a much bigger pie, lots of different flavors, you know, was like a shepherd's pie as like kind of everything thrown in.

Um, so that's kind of where I think the C D F is because it is all about the pipeline and delivering like software. All of these different components need to be added in and looked at in each of the projects that we support to make sure that we're not just promoting things that are missing a big, a big piece of what the pie should be. I, I just wanna make it real clear for our audience then, right?

Is CD events one of the projects under the C D F? Cuz I don't think we've come out and said that. Yes.

So actually what's really cool, and again, um, if you are looking to join a foundation or your board, as Tracy said, and you're really interested in certain things when you go to, uh, a foundation, there's not just projects. There's working groups and SIGs within the projects. So CD events came out of both the events SIG and the, uh, interoperability sig and they realized that they wanted to do more and they came together and created CD events.

So it's been, I think two years now, or just under two years since its inception, um, the adoption rate has been really surprising and exciting because again, people wanna make sure that they're not missing out on this new, and it's not a new idea, but interoperability, when you have all these different things that are siloed, how do you bring them all together and how do you keep an eye on everything to make sure, if I had a break over here, what is it doing over here? Is it doing anything? Maybe it's not and that's great.

Like let, we've got one thing to fix, but maybe it is, and if we don't have CD events running, how are we going to know? And I think that's what, what makes this project very exciting. Um, and I think why we've had such a strong adoption rate and a lot of different com contributors, um, from both, you know, uh, companies like Apple, Erickson, fidelity, and, you know, adoption from other projects like Argo and Flux and Harbor, um, it's just really exciting test Cube Jenkins.

Jenkins has a plugin. Like everything is sort of moving in that direction As, as ALA was saying. Okay, so what is, you know, well as you guys are talking about what the open source foundations could do in the testing space, one other thing I was thinking is that testing space is actually incredibly fragmented with all these different tools, different things.

Yeah, it's good. And uh, yeah, it's kind of, well partly it's by necessity because every domain of testing is different filling. But as I, I remember distinctly like, you know, writing Jenkins and like one of the very basic things we wanted this to like, uh, you know, process the test reports, um, and then it turns out that there's no standard and then there there's kind sort of defacto thing, but it's like you, everybody's trying to shove slightly different things into it and like it's because nobody is like taking care of it.

Like you can't, it still can't do even simple things like attaching screenshots to the PE reporting must in them way. That's nuts. So maybe, you know, I I I I think sometimes it's like, um, these mundane things, it's not particularly sexy, but those are what, like I really show up the, like, you know, the, the practice here.

Maybe, maybe I should, maybe I should be in on that one one cause it's, uh, close to my heart. Um, yeah, Yeah. Come back.

Totally, totally agree with that. I think that was one of the drivers for Test Cube was that, like you said, there was no standard for test report formats. Every tool was doing it differently.

Every, you know, as you said exactly that, right? Everyone was adopting the J unit XML format, which was like, I don't know how old. And uh, just, and obviously we can't go to every vendor and say, Hey, you need to change your format, but we can try to build a, some kind of layer, trans layer where we try to harmonize that data into something that can be coherently used across multiple tools or, uh, testing tools, C I C D tools, et cetera.

So one of the things we're, we're, we're trying to do, but just going back to what Laurie was talking about, CD events, I think, uh, cuz uh, what what's been so great, at least for us contributing is that that project has been so open and, and, uh, we on, on the receiving end, right? It hasn't, there's been no, no real pushback. I mean, obviously on some of the technicalities, but it's been extremely easy for us to have a discussion and really like to get other people, especially in testing world, to have a look at what, what, what is in the CD event specification regards to testing.

Because as we said earlier, everyone does testing a little bit differently. So the spec that's there is probably not gonna be the right one for everyone. Uh, but we'd like to be able to capture those nuances and maybe, you know, expand the spec or add some, you know, abstraction layer or whatever.

Going back to what KK just said, we don't want to end up, you know, just defining just one standard that a handful of tool uses. We wanna end, end up with a standard that is hopefully easily adopted by a lot of tools. And that's obviously, uh, based on the community engaging and, you know, telling us, uh, or, you know, submitting, uh, improvements.

So this asked me like we need a testing working group or a testing sig to come up with some reference architecture, like a basis from which others can use. Just throwing that out there. Just a CD events testing working group.

And you know, I wanna say to follow up on Alan's question, the CD foundation has made a an effort to brain testing into the conversation. You know, when you think of C I C D, you think of build and deploy. Build and deploy, build and deploy, build and deploy and testing becomes an afterthought or something that somebody says, well maybe we should stick testing in the middle there somewhere.

But testing really doesn't go in the middle because it's happens at every place. So we still think about build and deploy the CD foundation over the course of the years. And with, uh, fat's, uh, Che's leadership has really said we need to bring, uh, we need to bring testing into the conversation.

So he is making an effort to bring more testing tools into the, the CD foundation so we can begin having this conversation in, in terms of how we have CI I C T and CDs. But it's a, it, it does ha it has to be a conversation and we have to really think, think about it, it, and that it's the same way for security. You know, we, all of that is managed through the pipeline and all of that has to have a conversation.

Um, and testing teams and testing tools and testing open source tools and testing commercial tools all need to come to the table and really start talking about it. I was just saying, and these are separately the key to bring these vendors together to the tables. Cause I think every vendor listen to their users.

Um, so that, that's, that's the, um, and then I know CDF cares about that. And guys, can you CD foundation's also pushing for new end users Fidelity just, uh, joined. So those, and some of their input on and how they're managing the pipeline, it's been really interesting Hear people.

Um, okay. But you know what, luckily we're, we're, we're at our time limit here. I wanna thank OSA K and, and Ole and Tracy for joining.

Laurie and I, of course you could find out more information about all of these, uh, goings on at the CD foundation website. And we're gonna have a link to the CD foundation website cuz we figured out how to insert links into our Vimeo here so you'll be able to click on that. Thank you all for joining us, have a great time and we'll see you at the next CD pipeline.

Thank.