Challenges of Accelerated DevOps on Application Quality – Jason Davis, Sauce Labs

This is Techstrong tv. Hey guys, thanks for the throw. We're here with Jason Davis, who is vice President of product Management for testing apps for Sauce Labs, and we're talking about DevOps and testing.

And somehow or other, these always feels like two ships passing in the proverbial night. Jason, welcome to show. Thanks, Mike.

We've been at this DevOps thing now for the better part of a decade or more, depending on how you wanna start counting. And the question always comes back to, it seems like we're building software faster than ever, which is great, but the quality something, I don't know if we're making as much progress on that side of the equation. So from your perspective, where are we in terms of embedding testing in the DevOps workflows, and why does it feel like maybe we're not making as much progress as we should?

Yeah, no, that's great. Great intro question and something I've been looking at for a number of years, not only, um, in this particular role, but generally like looking at GitHub study of the state of DevOps and how teams who are really, uh, skilled at adopting those practices still end up spending about the same amount of time doing testing as, as teams that aren't, um, as adept at, uh, uh, adopting, you know, quality DevOps practices. So it is something that I think we've, we've consistently seen over the years, um, in, in addition to kind of just working with customers, um, recently, sauce Labs worked with, uh, uh, tech targets enterprise strategy group to survey, uh, the market.

And we talked to 300, uh, practitioners. So app devs, QA engineers, uh, DevOps, uh, practitioners, SRE engineers, mostly at enterprise organizations. Uh, th those with over a thousand, um, employees.

And the goal of that was to really understand, you know, what's the current state of testing approaches, uh, what are the challenges that they see there and where are they looking to invest in the coming years? So to kind of summarize some of the main, uh, findings there, and then we could dig into the details. Um, one thing that we've seen is even, even though DevOps has now been in practice for a number of years, just over the last three years, we're seeing more than 50% of our respondents are managing twice as many apps as they were before.

So in general, there's just more apps to be managed to be tested. Um, second key finding, and you alluded to it, Mike, speed is prioritized over quality. Um, and that's not necessarily a bad thing.

Uh, we were seeing that dev teams were 33% more likely to prioritize speed over quality. And what that means is we need better tools to support that speedy delivery, whether that's, uh, being able to test better pre-release or getting data after we've been released so that we can identify, uh, and teams can identify if there's a problem and how to fix it. So the second thing that, that we've seen is, uh, like I said, speed is prioritized over quality and, and teams need better tools to, uh, to help, um, uh, to support this.

And then the third thing is, um, as I alluded to actually in the beginning, the amount of time, um, spent doing code and new functionality hasn't really shifted. Um, the balance of time spent doing code and new functionality hasn't really shifted from, um, uh, from where it's been over the last few years where 60% of the time is really spent on new coding and 40% is still spent on testing and remediating and fixing. Um, so that's the state that we've seen, um, the challenges, like we said, like we said, uh, number of apps being managed, the fact that speed is being prioritized and and tool and, and really if we get down to it, what hasn't changed is that people are still relying on in-house DIY or do-it-yourself testing systems or shift shifting and shipping devices to developers so that they can run their tests locally.

And that just really doesn't scale anymore. To that end, we hear a lot about shifting left, and I guess I scratch my head sometimes cuz most of the developers I know wanna write code and asking them to test their own code is kinda like, you know, the fox and the henhouse kind of conversation. So, um, what is the state of the art for shifting left?

What should be actually shifted left versus what needs to be done elsewhere in the process? And I guess, what does a testing workflow need to look like these days? Yeah, great.

So, um, I think in terms of, we're seeing both shifting left and shifting, right? Um, when we talk about shift left, it's not necessarily about like unit tests and, and, and the longer integration tests, but it's more about identifying functional tests, uh, that should be performed as early as possible. This doesn't necessarily mean that you have to have like test driven design, um, but the idea is that while the the t the the, the capabilities are being built, making sure that there's a set of tests that are defined, even if they're manual tests, to, to capture the expected, um, uh, function, the use of the functionality, and address any sort of, um, uh, uh, complexities that might arise.

You know, uh, in general with the usage beyond that, when we talk about shifting left, understanding the breadth of devices or browsers that we need to support. And if there are, um, kind of in-depth, um, uh, complex features maybe on mobile devices, things like biometrics, um, or, uh, um, uh, or, uh, understanding the breadth of different types of Android devices and versions that are out there, if that's understood early on, defining tests for that early will allow you to, uh, to identify those bugs earlier, fix them earlier, saving time. So when we talk about shifting left, it's about functionality testing as well as the, the unit testing that folks do, and then trying to get breadth of coverage, um, uh, earlier on as well so that you're not stuck kind of remediating that code later on.

So the type of testing matters when and where it's done during the DevOps workflow, and you kind of need to think that through. Um, are you seeing more people think about security as part of the testing process? And is security really just a function of qa?

We're seeing a a bit on, uh, a bit security overall. Everything from, um, teams being under, uh, certain procedural, uh, guidelines for things like SOC two, if they need to have data, um, policies in place there, um, to, you know, just penetration testing and, um, and making sure that if crashes are encountered in your app, that it's not a back door into, um, accessing the code or, or taking over, um, uh, the application or the u the, uh, the app space. Um, so we are seeing organizations rely on, uh, doing security testing as they're doing their end-to-end testing, their integration testing, embedding it, uh, within that standard testing where they can.

I think that's the change that we're seeing. Uh, historically we had seen security testing as its own phase kind of later on in the development life cycle. And more and more we're seeing elements of it introduced earlier.

Um, I think a lot of that has to do with security teams, working with development teams and, uh, helping them understand what requirements there are upfront and baking that into the development process. You cannot walk down the street these days without somebody leaping out to tell you about their great new AI thing. Um, what is the state of the art for AI and testing these days?

I mean, can janitor of AI be applied or is it different kinds of ai? What should people expect? Yeah, um, that's a, that's great.

We've, um, actually at Sauce Labs, uh, we've had employed a team of r and d, uh, scientists, PhD scientists who actually have focused their study on machine learning and ai. And yeah, you're right. Over the last few months, just the, uh, the public introduction of chat, G P T and the um, uh, the emergence of so many different, uh, large language models.

There's, there's various different applications that we're seeing. Um, the way that we're looking at it, we see three core phases of testing, um, and we think that ai, m l and large language models can, can help improve, um, things in indeed, in, in those different areas. So first we see test authoring, um, in fact, the SAUCE labs has capabilities.

We've been working on, uh, N L P technologies for a number of years to support people who are doing low-code testing rather than having to, to to know how to develop automated tests and write, um, write testing code to be able to define your test in a natural language way, and then have the system generate the code. We see that one, that one area, uh, where the LLMs are really advancing, um, almost exponentially from what we were doing previously, um, in their ability to, uh, generate code in different frameworks. So having somebody define their, their, uh, their test steps in natural language and say, I'd like this, um, written as Selenium code or Cypress code, or Test Cafe code, these different frameworks that could be used.

Um, the advances we've seen there so far, like that, uh, it appears that the, the, the newer systems, the G P T four base systems are even getting like 80% of the code there. So test authoring and doing some N L P work is one area we see. Um, uh, uh, potential improvements also in test authoring.

Um, you know, especially with new mobile apps, uh, and new apps, when do we start testing? And another area that we see is the ability to explore an app in, in an autonomous way, um, clicking on different, uh, elements in the screen, um, and identifying then if there are crashes or errors that occur. Um, and then capturing things like visual snapshots to identify if there are diffs between different builds.

So this notion of, rather than even having to write tests, but have the system start to explore and get full coverage and iden help you identify where there are errors, uh, in your app is another area where we see support. So that's when we look at test authoring two key things, and there there's a bunch of other, um, uh, interesting projects in those areas, but that's, um, uh, one, uh, focus. Um, a second focus area is on the results, right?

Once I author all these tests and execute them, uh, I get pass and fail. When I get a failure, there's always time spent digging in what actually failed? Um, was it a real test failure or did my test infrastructure fail?

Could I not access, you know, uh, a system that should have been up? So maybe we need to do a retry, um, or I'll get through a, a full, uh, set of tests and they all, you know, are checked off as passed because they pass, but the system generates a few errors or crashes along the way. Um, so, so when we look at, uh, uh, the test results, all of this data is consistently being generated and it still requires the end user to sift through it.

These large language models are doing a really good job at summarization of that data. So if I have a failure, um, or if I see a pattern of failures, these LLMs are, are really helping to identify, um, what changed or where, what line of code is it, where the error occurred. Um, so that's the second key area where we see, uh, ai, ml and large language models starting to, um, uh, improve the environment.

It's around results analysis, log summarization, and, and then, uh, taking that feeding into the third area, which is test optimization and execution. How do I know what tests we should run, um, based on potentially like what code changes just went into place, how my tests normally proceed. If I can, um, understand those test results, I could feed them back into the engine that's doing the execution and the optimization and make some decisions for you on the order of tests.

Um, for example, if it's a new piece of functionality, the new test that were written should probably be run first. Um, and you can further, you know, take that a little bit further, but making suggestions and, and optimizing execution is kind of that third area. So we're looking at it in those three different phases of, uh, uh, of, of tests and how it can improve authoring, um, results.

And then, uh, execution and optimization, an orchestration. So every year, as long as I can remember, somebody stood up and said, this will, over the year, we won't need dedicated testing teams, and every year we still need dedicated testing teams. So what is the future of those dedicated testing teams and how will their roles evolve?

Yeah, uh, we're always gonna need, I I don't know if I wanna say dedicated testing teams, but there's testing teams. Um, and in fact we have seen, I think, a big shift in the, the level of responsibility for tests falling onto the developers in addition to just qa. So absolutely, I think we are always gonna need teams and responsible individuals.

Um, but I do see a, a bit of a, um, a a of a, a shift in where the responsibilities lie. Uh, that said, uh, one of the things that, that, again, anecdotally I had seen this, and then in the, uh, uh, in the, uh, uh, the, the survey, we saw this as well. Um, manual testing is still pervasive.

Uh, almost, um, two thirds of testing teams have at least half of their workflows in manual tests. Um, so that's huge area for improvement. Um, part of the reason for that is automating test is actually a relatively skilled, um, job, uh, writing selenium tests and so on.

And as I just mentioned, like there's some new technologies and test authoring that might start to, um, improve that, but we always need somebody who's gonna define what is it that we're testing, what is the functionality? Um, and that requires domain knowledge, uh, knowledge of the application of the, the, the, the, the, the goals of the app and how users actually start to use it as well. So one of the main things we see for the future is a push towards more automation, but not necessarily needing automation engineers, rather taking the folks who are good manual testers today, and as they're doing their exploratory manual testing, use that as the basis for low-code testing for rerunning those tests and future versions, et cetera.

Um, so that that's, you know, when we look at the future de definitely teams that are responsible for tests, um, leveraging some of this new technology so they can go further, um, towards automation and that, and that move towards automation is huge because we see that teams that have high levels of automation, um, c QA and, and testing in general as a competitive differentiator for them, and that, um, they, they feel that not only do they feel that their, their, their applications are more tested, but it, it shows up in the results of, of, um, of changes in production where those teams that do have high levels of automation see lower amounts of production code changes causing functional issues that require mediation. So if I step back for a minute, um, overall the survey tells us that about a quarter of production code changes, um, require some level of remediation because there's a problem that's introduced with that production, uh, code change. Um, when we look at just teams that have a high level of automation testing, we see that number, uh, decrease, where 25% fewer, um, production code changes require, um, remediation steps.

So overall, we see a continual push towards automation, and we see that the numbers are telling us that that's good because teams have more confidence in their releases, they view it as a competitive differentiator, and in fact, when they roll out changes to production, they are seeing less errors in people who rely more predominantly on manual testing. So in our minds, we have always made this kinda bargain between quality versus speed. Do you think that we can get to a point where we just don't have to make that trade off anymore and we can actually have both?

Yeah, the, the last few years there's been more, um, I, I'd say, uh, more level of comfort of doing what's called like production and testing where we'll release very frequently and you could release new functionality to a canary test, to a small pest set of users and make sure that it meets the level of quality that you expect, that the, that it meets the level of usability that's expected before rolling it out further. So I do see more of a trend towards that. Um, sauce Labs as well has seen that and is focused also on bringing error and crash reporting into the fold so that as people release not only their web-based apps but their mobile apps, that production error data gets fed back into the system.

And then teams have tools to help them quickly decide if that error is important enough to need a fix. And then when they make that decision that they have enough information to know what to fix. Um, so I do see more of a trend towards releasing more frequently.

Mobile apps do make that more difficult because mobile apps require a, uh, a cycle through the app store before they get approved. So I'm also, another trend we are seeing is more people doing hybrid apps where parts of their apps could be delivered over the web and they can make those changes more quickly. Um, but overall, I, I do see that trend of testing and production and pushing things out, um, to, to be more frequent, not only in smaller startups, but it's encroaching in the enterprise space as well as there's that expectation from their users that they get the latest and greatest as quickly as possible.

All right, folks. I think that the end user wants developers to spend more time testing their code versus testing them and their tolerance for updates in the future to fix things. So let's get it right the first time, as they say.

Jason, thanks for being on the show. Thank you, Mike. Appreciate the time.

All right, back to you guys in the studio.