Toward More Efficient QA Triage With AI & Data | DevOps Experience 2023

Uh, hello everyone. Um, I'm Koska. Today I wanted to talk about, um, the thing that excites me most right now, which is this, you know, the age old activity in software development, which is triazine, but which is a different phase.

It's AI and data. So, um, let me start by introducing myself. I'm ska, I'm probably best known as a creator of the Jenkins, the popular open source CI system.

Uh, that was what, at this point, more than 15 years ago, and then I was being c T O at cloud base where we, you know, turned kin into like a beer enterprise. That's been, what, 10 years ago. And for the last three years, I studied a new company called Launchable, that's a startup where we are working on the triads in the qa, trying to modernize the qa.

Um, so in, you know, in those capacity, what we've been saying is that this, you know, DevOps, infinity, this is how we improve the software at the moment, right? That's how, you know what, that's what I've been passionate about, um, pitching to the world. Um, now if I, if I look at this a little more carefully, uh, like going to different teams and seeing and talking what they're actually doing, my observation has been that it's really like this po this part, like testing that's clogging the effective, uh, delivery pipeline.

In other words, you know, producing changes is easy enough, quote unquote, easy enough. You just type it in. But building enough confidence to those changes so that if you're comfortable putting it in production, well, that's a different story.

And then the project gets on, like this gets hot and hot. So that's the part that I sort of started thinking more about, and that's what the part, that's the part that I wanted to talk about today. So, in other words, like you have automated the tests, that is a challenge that its own, but let's assume that already happened and you've been running the test, you get the results, great, then now what?

Right? Because if you just let the test run, then that's not, that's as if you're not doing anything. The, it's the, the action to what that test results say, which actually improve the code to the software, right?

It's like, you know, just like the exams don't mean anything. If you're just taking it, you know, in that outcome, it's your reaction to the score of the test that makes a difference. So, so what happens after these test run, that's what I call the triers, and I generally like a, classify them into these five different buckets in the order of the sequence.

The first one is about, you know, getting tests to run and then, you know, easier said than done because as we'll see later, you can be efficient and effective in doing this. And once you get the test run, you know, it, often the results contain noise, most notably, let's say fakeness. So you need to feature away this noise.

Um, and of the remaining program, you need to analyze those programs to figure out what actually happened. And then the right developer needs to be like poked so that they recognize and they accept the problem, they know what to work on, how to fix it, and then they work on the fix. And then finally, you know, it usually takes time until that fix shows up and it gets verified in the place that the was detected in the first place.

So that time lag creates interesting, um, the, the challenge that can be, or in efficiency that can be addressed. Um, so the reason I think this problem is interesting, at least to me, is that this kind of triad problems, the test problem deals up over time. It's like the, you know, it's like that, that you developed the acquirer over the years, um, when your life is successful, right?

The project gets bigger, you know, you start to like accumulate more tests, the code base gets bigger, so the test takes longer to run. More people get assigned to the projects, which means they are driving more workload. And then so slowly, and instead of these things start to slow you down, and if left unchecked, it could crumble under, like the project could crumble under its own weight.

So that's it, that, that is the challenge that a lot of, if anything, like a successful project to face. Um, but I think the, so when I think about, well, how to tackle this, I think the data is this, this is where the data creates an opportunity, right? Because if cout the industry, like everywhere where the, like the, the amount of data went through beyond the critical mass, um, it created, it, it created opportunity to turn that data into the intelligence like actionable insights or the, you know, AI that makes some of the decisions instead of humans.

And that scale, the, the scale of data is what was driving the, like a create opportunity to make like the next level of efficiency gain. So in the software development space, i, I I, I firmly believe like one area where the massive data went above the critical threshold is the testing. Because what automated tests run a lot more frequently in the processing lots of data.

And then, you know, I'm not alone in saying this because evidently, like a lots of development teams out there are starting to tap into this data to make a quantitative decisions, uh, to approach this problem. So let's take a look at, you know, some of these practices that I've seen in wild or propose, uh, that would, uh, as a, as a source of inspiration. So what can be done?

Okay, so the fourth one that I wanted, the, uh, the ti was this about getting tests to run. So again, know the, so the problem, the challenge in this space includes things like, alright, like my tests are too big, or what takes too long to run, or because of those reasons, the frequency of tests run are too low, like 90 number, the nightly test or even weekly test. Um, and then that, that creates a slow down the feedback cycle.

So developers, discovery issues to date in the game ing delivery schedule, um, or that the testing capacity is limited. Um, if, let's say you're in the embedded space, you might need like the actual physical computer, um, or the devices to run the test. So what are the kind of things people are doing to cope with this program?

So I think the are pretty clearly like effective tactics here is to prioritize tests. Meaning, you know, they can recognize that not every test is equally important and meaningful. In fact, I'm pretty sure in any place where they have enough number of tests, you probably have a good number of tests that you just inherited.

It's there. And nobody now at this point in remember why it's there or what it is testing, but I can nonetheless, it's just they're occupying the, the, the, the, the time and the maintenance overhead. So the how, you know, prioritize the, the actual approaches of prioritizing tests would be, could be as simple as designating a smoke test.

You know, for example, like, or common tactics is the annotate this, some of the test methods as a small test, which creates a smaller subset that can be run more quickly. Um, Google has a paper published that in which they allocate that rather simple, small, medium, large distinction, mainly along the line of the time it takes to run. And they love that like the, the brevity or the simplicity of the approach.

Like forget, like forget that common taxonomy, like you need test versus integration test, just focus on the time it takes the run. Um, and then there's a little bit more than that. It's little oversimplification to that extent, but that's, that's an example of prioritizations.

Um, that's, you know, data driven in this case the duration. Um, and then, so once you start prioritizing this test, you could start doing things like, you know, the running different tests at different stage in the software development. So like a running a small, uh, set of tests during the pre-mark, um, so that, like you get some, you get to keep the next stage cleaner, but the, um, keep the execution time in check.

Uh, you might still want to choose to run the full set test cycle at every, you know, every night. Um, that kind of like a stack guard approach so that you can balance the, the, the workload you're creating, the speed of the feedback you're providing to the developer and the overall pace of integrations. Um, and then these things often is constantly adjusted based on the phase of the software development.

You know, the product in the maintenance mode usually do not need the same kind of aggressive feedback, well, the set, same set of configuration as the young product in infancy and that sort of thing. So maintaining this flexibility, I call, I think of it as like a gear shifting in your car, um, depending on the speed that you're driving is pretty important. Um, so those are like a relatively primitive database approach, but the more sophisticated ones are what I think of as a testing part analysis.

And then, uh, this was, um, you know, this is the idea of assessing what needs to be tested based on what has changed. Um, and historically it's been done manually. I've seen, and I was in the one of these teams myself long time ago, and I've still talked to some teams who are like that, where the, you know, leadership of the development team and the QA team kind of come together and they, they, they look at, okay, in this disease we touch these things.

You know, they have long test libraries, they can't run all the tests. So they, they come to agreement that in order to sufficiently, um, the, uh, ascertain the, the curt, we are gonna run these subset of the test. And like a day later they discovered, oh, like we have this one last like hot fix that needs to come in.

So then let's done these even small set of tests to verify that the hot fix is okay. That used to be a very manual process, but, uh, the test impact analysis used to be a very manual process, but I'm now starting to see that this is where the, um, the, the ai, at least the machine learning was introduced. This was a paper out to the Facebook that came out of the five years ago or so, in which they deploy the, um, the machine learning.

They essentially, uh, make an estimate of which tests are more likely to fail. So like say if you have eight tests and then you're looking at, these are the changes that gets, gets tested in this round, um, then it makes a, the machine AI makes a reasonable educated guess about, well, it's probably T four and T eight are most likely to fail if five is not unlikely to fail because it has never in pill in the past, et cetera. And if you have some guess on that, then what we can do is to execute the test one in the order of the higher likelihood of the failure so that you get to signal, you get to discover the actionable intelligent bit first, or even like a stop running those frequency tests in certain phase of the development, right?

Like if it's a pr, uh, preor tests. So in that way, again, you, you, you get even additional tools to balance that speed of the feedback and the workload it's creating. Um, and then the overall, like the speed of the integrations.

Um, so this is something, you know, I learned, I think we've been really heavily working on for the past few years. Alright, so let's move on to the next phase. Now, you have successfully and efficiently run the test by using the ED data.

Uh, so now what's next? Like, you, you look at those test failures in the future way, the noise. So KY test is really that kind of problem, rely on share the problem here, but that's not the only ones, right?

Like what often happens is that the, if a test that was failed yesterday, that was priors drug as a P two issue for, I don't know, like a payment team, they're working on the fix. So that has not yet come, which means if you're gonna run the same test again, that test is gonna fail, but nobody else needs the, the quinte, um, because you know, it's already analyzed, right? So these are known failures, which creates the nodes once it was properly prioritized.

Um, the third kind is like if there is a lot of failures, like often the single problem with manifest, that's tons of failures and it sort of creates this effect in it's like older, like, you know, if you're in the nuclear power plants and you have lots of warning lights and everything turned red at the same time, then it's really difficult for you to spot, okay, these are like a series separate problems that needs to be tackled separately. So getting this like collapsing these large amount of data into, well, like these five failures are all same issue, one issue, here's like a 50 of those, that's another issue. Like those are another way of peeling away the noise.

Um, so, so what are people doing there? Um, so the one is this stuff is a flakiness. I think this is really, um, there's a lot of efforts taken on by different engineering teams.

Uh, it's in my mind generally, um, like I've classified in like a three pillars. One is kind of knocking back the fakeness, meaning like you wanna keep fakeness in check. You can't, most engineering teams have reported that in typical software engineer domain, you cannot eliminate flakiness, but that doesn't mean you don't need to do anything about it.

So there's a certain attempt to contain the flakiness, uh, by, by updating the test, by changing the code, et cetera. Um, and then the second pillar is by accepting the fact that there will be some flakiness, uh, you wanna like prevent it, you wanna make it keep build green. Um, so you wanna exclude flaky tests on certain place of the, um, the test executions even fakeness so that it doesn't want failure, it's not gonna block the integrations, stuff like that.

Um, and then the last bit is, um, you know, the escalate like measuring the flakiness so that people know what to work on. So we, so that leads fakeness can be pushed back on. Um, so the typical, the approach to flakiness is by basically looking at the past execution history and then see which failures will be attributed as flaky.

Um, so the, I believe this is done by Spotify and it's, uh, well actually the Spotify guys are taking one step farther by saying the fakeness tend to happen randomly. So when you see this like a one of failure or that's a, that's more likely that these a flaky test. But when things, a lot of things fail and mass like this, then it's usually less likely to take, maybe it's more of an external factor that's more like, so those are less likely to be random failure.

So that's one of the ways in which they're computing it. Um, there's more like a clear or a statistical analysis to proper mass based approach to try to assess the probability. So here, um, they, they're, let's see.

Um, so what they're trying to model is, um, the software, like what is, so when they have this observe observed result of trespassing in creating, what is the chance that the, um, the, the software was actually bad and therefore causing this program versus what is likelihood that the, you know, the software is actually good, but it just have a random failures and then you can make a, like a probability estimation based on those and then they go from here. Um, so these are more sophisticated way of a key test measurement, but the, the point is like you can use data to actually quantitatively measure what flakiness, how, you know, how severe a flakiness is in which part we have tested, as opposed to relying on like a human instinct. Um, and the, uh, that is probably the pre predominant way.

Um, and once you have that kinda information, you could use that to surface this information, ways of drive actions. So GitHub engineering team has reportedly built an internal software in which people, you know, they try to highlight the most severe flakiness problems and they try to like, you know, look at, analyze the history of the source code to see who is most likely to blame. Um, it's in, in the idea of creating a directory, responsible individuals.

So you can imagine there's like additional data crunching going on behind the scene like this. Uh, but it'll, you know, make, make it easier to both justify working of fakeness program, which is typically a frustration of engineers that they don't get time to work on fakeness issues. Uh, but it also makes it easier for the identify specific individuals.

So, um, so another example of, uh, dealing with noise counter test failures is, I, I love this example because this kind of clearly, uh, stows the idea that the test results have became a fire hose. That the, the amount of data comes out of it is so much that the humans cannot directly deal with it. Um, so this is a system that's deploy from the Dropbox engineering team where like a failure, if a, if a test failure is observed, it doesn't automatically like a flag, uh, notify the engineer, instead it gets put into this yellow sort noisy test state.

And then, so there are some still they're weighing the possibility that this is just a flaky test that needs to be like yanked out or it's a genuine code program. And that's a, you know, we, so you say is that recognition that we don't know yet, and then there are more need, more work needs to be done to figure it out. So what happens is, is that if the test is put into this state, the automation kicks in and start running.

Let's say that that test like multiple times, um, they start testing the previous versions of this, the same test to see if the same fakeness behavior can be observed back then, et cetera. Um, and then if that determines, for example, that the Yep, the failure is consistent, it's not the test program, you know, it's been behaving, it's been regular, consistently passing yesterday, it's been consistently failing today. Well, it's a real problem then that the code gets yanked out and then the leveler gets notified.

So they're given the opportunity to leave work on the fix if the test is identified to be faulted. Like, okay, this seems to be changing behavior actually about changing behavior yesterday to that we looks exactly, we got lucky. Then it's a test that gets quarantine and they're put into different queue, um, and those tests will be moved from the future rounds because it's just creating noise.

And then the engineers, the queue get to work on these quarantine issues. But we said, you know, these are usually of different priorities. So, you know, that's, um, you could, yeah, so system like sophisticated system actually having developed in different parts.

Um, I know some other companies who are building who have built a system of this nature, and this is a great example, using data to sort of, you know, the, they automate the lower level screening activity that the humans used to do, right? Like any developers, often the first instinct is seen a test year is they wanna deeper like a reproduce the failure locally to see if it's properly fails. Um, and so this, this system is, is, uh, is doing exactly that.

And then that's an example in my mind, it's almost like, you know, qualify as artificial. This is a form of intelligence in my mind, although you might not call this as ai. Um, and then last aspect is, um, this is, it's much more low tech, but it's more approachable for many engineering team, which I call the snoozing of a failure.

But it is like, once the test failure is ed genuine, and if somebody is working on either amyloid in the failure or working on the fix, then the failure, the test should be excluded to prevent it from blocking the future integration work. Um, so, you know, they should be marked. Um, so often, you know, the, the practice that's put in place is to define annotation that, you know, or the test, but by there is information to link to the ticket so that the, you know, the excluded test accidentally end up getting excluded forever.

So that's a lot of approach and it can, it can, you know, then like going up from there, like having a system that keeps track of that information from the, from the, um, chrome back tracker dynamic originator is statistics good and things like that. And the, the point being like there's no use keeping the same test failing over and over again. It only distract eyes away from the new, more important failures.

So that's, um, that's another easier accessible one data-driven approach here. Okay, moving on further. Um, the triers, uh, the, so now that you have, um, you have, uh, filter the way the failures, the next step up is to analyze those real failures that's, uh, left.

I'm sorry, now that you fit it, the noise, it's not, the real problem is to analyze the real failure that comes out of it. So this could, you know, this symbol, this is probably the most time consuming part for the humans today. And that ranges from things like, okay, where is the source of problem?

What type of problem? Like, is this a code problem, test problem or the environment program? You know, like what changed recently that might be potentially have advanced to this behavior changing the test we are seeing, you know, like where can I find more details about what actually happened inside this system that was tested?

Um, you know, was, did we, have you seen this failure mode before? If so, what was the previous call was? Like, what was the, you know, di diagnostic?

Um, and if this is the, um, or like just giving those, if I'm eventually gonna pass this information to another day so that they can fix it, how can I pass all this investigation they calls so that they can grasp the right context and get to the right away? Um, you know, how important is this failure, right? Is this a P three issue that's, that results in test getting excluded for this purpose?

It is. Or is this P one issues that requires like a me, like a DMing, uh, direct messaging, the person to get the confirmation, uh, et cetera, et cetera? Um, yeah, so it's really, I think from what I can see, okay, this is a real drug currently.

Like I almost seen as the, um, like experts. This is the one that needs a lot of expertise in the domain knowledge. Um, but it's also kind of fun because this feels like a detective work.

Um, and it takes a lot of contextual knowledge, therefore, it's, um, it's a painful process. And often this has a, this feeling of, um, opening up the black box that is the software under test, you wanna, you know, experience and the developers know where to find the logs, logs for different parts of the system and how to interpret those to, to, um, to connect the dots in the critical picture technology like distributed core tracing, you know, JP, king Jager and so on, uh, supposedly to help this effort, um, not just production during the production, but during the, the Test Q airport. Um, so those are all like, you know, shifting through large amount of data.

Um, and naturally I think some of the more recent excitement in ai, which is the large language models, created a great opportunity for this. So I've seen some attempts to, you know, like trying to analyze this program at least preliminary by using ai like a summarizing spotting, like, like oddities. Um, it could, I I, I've seen some attempt to do that, like a interactive conversation bot to help them analyze the, like, you know, the in-depth, help them assist in depth analysis by providing suggestions.

I think of it as like a glorified version of the a, you know, decade technique that is called the rubber active vacuuming. If you haven't done it, the idea is you try to explain, you know, the problem is to this in animate object, and then the process of articulation can help you realize the problem. So it's like a way of trying to emulate the dialogue in this case is, uh, with a og well now we actually have another AI that you can, you can have actual dialogue and this guy be helpful in actually suggesting things to, for you to look at, et cetera.

So that I think is, is exciting, especially for showing up, like, you know, like a mid to lower level of this pyramid. Like the, the people who are truly the expert at this doesn't might, might find, you know, this AI system less valuable, but the, like a larger number of people who are not so well sophisticated in this effort could greatly benefit from something like this. Um, so that's exciting development in this space.

Um, another also like a datadriven excitement in this space is more like eager the monitor and instrument, the test environment, much like the production. So get and get data out from during the test executions, which give you more like a cruise for somebody who is ODing issues. Um, so I've seen services like, like, you know, that like head spin, it's, you know, puts a lot of it lets you run the mobile app test, but this mobile devices heavily instrumented, so you can see the network behavior like a latency, all sorts of like a CPU usage, all sorts of stuff like that.

This could be, this could be really useful, you know, to figure out what's going on. Um, so the, the, okay, so now the problem is analyzed, so it needs to be passed on to the right people. Um, how are you gonna go about that?

So I think the common techniques that people deploy to use data to like a figure we to write the program is this like a, what I think of as automatic classifications. Um, like this could be as simple as a regular expressions. Like, you know, I've seen successful deploy of people pattern matching at common error messages and then added dispatch.

The failure to the right team is this infra program or the app program, uh, more sophisticated one. It typically is things like a base filter, which is a technique that they use to train the spa filters. So that's another, um, modest ones.

Um, it's a less techy approach perhaps is a prioritization metrics in which you'd say, okay, um, you know, the, if, if the, um, um, so, uh, so if, if a program is happening in, in the, uh, the core user journey is, then those are automatically P one issues. If the UI g that's affecting less than 5% of these are, so P three, um, but information leak, anything to do with potential information leak is automatic P one. So like, if you go through going through enough of these failures instead of looking at individual issues and figure the right priority, you can almost like, uh, create that prebuilt decisionally, uh, and then added streamlined is therefore quite a bit.

Um, so that's been successfully flawed. And finally, you know, like attracting the failure until they, in other words, like figuring out like when the fix was supposed to arrive. Um, and then here it's um, the two kind of key bits.

The one is, you know, the baking this like a commit hash of the source code into the binary, or if it's a web services, like create the endpoint that can serve that information information. Um, and then the second bit is the, this hygiene of referencing a ticket ID in the commit log so that they, there can be mechanical identification between what commit was that's fixing what, uh, what ticket. So if you have those of those two things and they create, let you create this, um, machine readable audit trail of a sort, so you could say, well, the build 152 was created out of this commit, next build was created out of that commit.

So there's a little, a bit of scripting you can easily see that, okay, these three commits are new in this new build and that contains a ticket difference to those two two tickets. So they are the, they're the one that gets fixed. And if you're doing the test, if you can record what build it's testing, then you could see, oh, the build 15 between the test 15 and 16.

Now we are testing these three new changes. So any test behavior change between those two things, uh, like one of the things, the first thing you suspect is, well, maybe it's caused by these two fixes, or if the previously the tests are known to fail because of the CX 18 issues and you expect this test to now make the failure pass. So that also creates, give you a hint of information about like, what was supposed to happen.

If the test keeps failing, then this would probably know right away because they need to remark on the fix. So all of these things previously as in the heads of people, is a little bit of like a gluing like this, like you could, you could completely automate these things away, which is exciting. So, um, yeah, in conclusion, you know, when the test results come in, like a real work for humans being in, um, and there's a lot of opportunity to make this more efficient, you know, and, and it's good to know, it's good to think about where are your teams are spending the most of their time in these activities.

'cause that would be the prime example of what the, where to improve. And, uh, the data and AI creates a great opportunity. And that's something that we are working on at here at much about the moment we the reality in this game.

Uh, but, uh, if this see is the kind of things that you're thinking about, um, and then these are the challenges for your team, I'd love to talk to you more about it and get to know your v i d. Um, so yeah, so your personality, you know, how to contact me and us. So, um, but uh, with that, with that, I wanted to, um, yeah, wrap up my, uh, presentations.

Thank you.