Integrating Security in Quality Assurance TestRail’s Chris Faraglia

This is Textron tv. Hey guys, thanks for the thrill. We're here with Chris Faria, who is a solutions architect for testra, and we're gonna be talking about well, quality assurance, testing, security, and how hopefully all this is finally coming together.

Hey, Chris, welcome to the show. Yeah, no, thank you Mike. Uh, happy to be here.

Very excited. One of the things that we have been talking about for a long time is that, well, security should be part of quality assurance at some point, and yet when we develop software, we still kind of treat these as somewhat separate motions. What is your sense of where are we, what's our maturity?

We try to shift more responsibility left towards developers, and then they tell us that, you know, their head hurts. So how are we getting the, where are we on the journey? I think every organization is different, Mike.

I mean, coming from the test perspective with like what we do at TestRail and my background, which is very test in DevOps heavy, I think we're making really good progress, right? This concept that we think of mainly in the testing world of shift left, I think can apply the same as we start talking about, um, security aspects when we're both, uh, you know, building software during the development phase and then later as we're deploying it and promoting it, right? Putting into our milestone and staging environment.

So it's each stage of the development life cycle. I think we're getting better, but more recently, tools that I've worked with, like check marks, um, have these abilities to start looking at static code patterns where developers can analyze code as they're developing it. And then on the kind of the SaaS side, right?

And then on the das side, when we start talking about actual dynamic penetration and, you know, proxying security scans or proxying traffic through tooling, we can start capturing that type of data and then centralizing it into test management solutions or other tools. So we're getting now visibility not only on the backside or maybe pre-release or from a specialist doing like a pen test, but we can have it, uh, as far as shift left as, uh, static code analysis with tooling like check marks and there's others out there as well. Um, and we also can get kind of, uh, deeply embedded into our CI process with various ways to do proxy penetration scanning, um, even open source tooling like Oasp provides with their ZAP tool.

And there's a lot of things out there now that are even free. Um, and then combining that into central point of visibility again, like test management, dashboards, even CI type reporting. Um, so I think we're coming a long way, a lot better than what we used to be.

Um, so that's kind of my quick slice of the state of the state. I feel like we got lost somewhere on the path towards shift left, and we just assume that everything was gonna go all the way to the developer, uh, if it's important to push some stuff left toward a developer to give them context when they're writing code, but it seems like there's also other parts of the security quality conversation that still need to happen in the DevOps workflow or at the CICD level or the build level. So do we just need to get smarter about what takes place when?

Yeah, absolutely. I mean, I think in general, you know, I, I talk a lot about this in some recent webinars is really before we start doing and executing in these various types of actions, operations, investing in tooling, looking at what I would call the team or the product's overall security posture, right? What do we want to cover and what does that really look like?

And then once we do that, we can start to define roles and responsibilities, right? It might be a developer's job to look at things at a static code level, knowing that that's not gonna cover everything. But maybe we have, for instance, test engineers or even development engineers.

The roles and the lines have really been blurred now, and we really look at what we want to achieve rather than exactly who's doing it. But we have to ensure kind of that full stack of, you know, testing for both security functionality is covered both on the development and like the backside for test engineering, even DevOps engineers and how we're building and designing these things. So I think it comes down to defining the security posture that we wanna maintain and then really decomposing that into roles and responsibilities and then tools and different things may kind of continue to, to fall out as we kinda look at that design.

Um, but I think the big thing is, is understanding at the heart of it what is the security posture and the requirements, and then we can decide who does what on the left and what tools and who does what on the right. And then we build kind of a full stack security posture for scanning awareness and really catching issues as, um, as this lifecycle, uh, happens rather not on the backside or hopefully not in production. So I think it's really thinking about the whole process rather than individual pieces of it.

Well, speaking of the whole process, so we've had quality assurance gates within DevOps workflows for a long time, and um, we tend to talk about security separately from that and we talk about it as DevSecOps. Um, are those two things, do they just need to converge at some point because security is part of the quality conversation? Yeah, I, I am definitely a big proponent of not only, um, from a security standpoint, but also uh, reliability engineering.

I think I've been in, uh, test related roles both doing it, managing and building teams for a long time. And I believe the lines nowadays for the modern test engineer are being not only blurred but combined of their responsibility for functional and non-functional testing. And then non-functional testing now is encompassing concepts like disaster recovery and failover, which traditionally is site reliability engineering type problem or responsibility domain.

Um, and then the other side is what you just mentioned is understanding what security posture is and how we're gonna test it. And one of the biggest things I cited again in a recent webinar I just did is, um, you know, the AU top 10, I think the latest 2021 revision A oh one, which is like the number one item is broken or invalid, um, broken invalidated or misconfigured, um, access controls and that stuff, it can be directly validated. So like maybe mechanisms like AU or whatever off N or au Z mechanisms you use, if they're not configured correctly, we can directly test these in functional testing.

So that's something that can fall directly in the, the test engineer quality engineering role. So I think these concepts are now coming together and the lines are blurred more now more than ever, uh, for these types of roles and responsibilities. Um, it's a great question.

Speaking of that list that you just referenced, one of the frustrating things that people always talk about is that list doesn't change very much in the last decade. We seem to be keep making the same errors over and over again. Um, do we not learn from where our remediation efforts or what is the fundamental kind problem that we keep tripping over the same stuff over and over again?

Well, I think the one big thing that we're seeing, and I, and I've said this again kind of coming back to my domain of testing, and I jokingly say, you know, organizations really should think about this concept of releasing at the speed of testing and we can adapt that of releasing at the speed of testing site reliability, engineering and security validation, security posture, right? So I think the big thing is that organizations want to deploy faster, quicker, more frequently. And a lot of these concepts, um, again, you mentioned, you know, dev SecOps, right?

Focusing on security aspect, disaster recovery testing, these things a lot of times I feel are left behind. Um, and secondary to this idea of being first or, uh, quickest to market or, uh, supporting customers or new features. So I think we keep making the same mistakes, not realizing that, um, realistically we should only be deploying at the speed of, and I'll say as a generic term, DevSecOps, which includes testing site reliability, engineering and security and or DevOps maintenance.

Uh, I think a lot of that is left behind and it's a commonality. I think that, you know, things I've seen while we're making these issues time and time again 'cause we're sacrificing uh, for speed, essentially. Why are we arguably deploying software that's unsafe at any speed?

And we seem to just say, you know, that is the justification in its own right. And maybe that's just 'cause the thing we're rewarding and the metrics are wrong. Do we need to reevaluate what it is we're judging successful software projects on?

Yeah, absolutely. I, um, you know, along with my role at test drill, I've been a long time follower of some principles of very large core, uh, engineering houses, Google being one of 'em. They have a great testing blog, the Google testing blog, and they talk about this concept of releasing on yellow.

We don't really release on green, we don't release on red, we don't wait to release for green, we don't release on red, but we release on yellow. And I think it's an interesting concept of then how do organizations really define what yellow looks like? And again, you can extrapolate that to criteria of software testing and quality benchmarks.

What are our thresholds and um, posture for security, you know, what CVEs, did we not address this time, but we should have before release? Um, we talk about disaster recovery and failover. Did we really touch on our restore and backup process in the case that we have a security breach and now maybe we're under a ransomware attack, right?

We can go down all these kind of rabbit holes and scenarios, but I think that organizations need to look at and really reevaluate that red, yellow, green status for how and when they're releasing, um, to really make better decisions, be more informed on what the risk is gonna be. Of course, everybody and his brother is talking about AI these days. Um, where will AI fit in this equation in your mind?

I mean, will AI help us unify all these processes that, and maybe, um, lead to more secure software eventually? I mean, maybe not in immediate short term, but at least eventually. Yeah, so the short answer there is Mike, uh, we're just gonna click a button and all of these problems are gonna go away.

Actually, um, no, that's not my official stance nor test rails. Uh, I'm just being, uh, just kidding actually. I think everybody hopes that, right?

And we talk about, you know, NLP and, you know, uh, more predictive and analytics based, uh, you know, views that are AI driven and obviously generative ai. I just was at a conference, uh, Eurostar, which is a large testing kind of conference and consortium in Stockholm back in June. And that was the big topic.

How does machine learning and specifically even areas of generative AI help us for testing? And that can be extrapolated for any types of testing, right? Validation for ac uh, you know, SaaS-based, uh, uh, or sta uh, static code analysis, but also dast, uh, dynamics security application testing, where we're running things against the running environment.

How does generative AI can help us in those situations or even functional testing. And the end kind of thing that I took away, my kind of conclusion is we don't really know yet. Um, there's a lot of things happening right now where I think it can be assistive technology, meaning maybe we'll generate some test permutations.

Maybe it will give us insights of security vulnerabil vulnerabilities given some known domain of libraries and dependencies we're using in a code base. But it's not a full solution. Like I was joking, just beginning your question.

So we're just gonna click a button, it'll all go away. Uh, I think we're very far from the reality of fully autonomous solutions using ai. I think the reality is maybe using some predictive analytics for the data we have and maybe using a percentage of generative AI to create test artifacts scenarios, give us insights that'll help us then manually or using a development model solve some of these problems.

So it's not gonna be the click button answer unfortunately. So Are you a little worried that things might get worse before they get better? 'cause it seems like when I look at the AI tools, they're optimized for generating more code, but they were trained on code pulled from everywhere in the internet, and all of that is a varying quality.

So we might wind up with more insecure code, then we can handle it. Yeah, so I, it's funny, I mentioned this again coming back to, uh, my recent visit, um, with the test real team in, uh, Stockholm at Eurostar 2024. And I, I gave this parallel and I've been around a little bit.

Um, so maybe not show my age in the testing community, and I think we're in the similar kind of stance on security tools, but I remember when this idea of record playback, whether it's uh, if you're proxying traffic through a server for, uh, realtime kind of DA style security scanning where it just tells you every problem, right? You don't have to think, or you know, the old record in playback automation where you click a button, you do a bunch of things and you just generate code. And that was a very new and shiny thing and a novel concept that would solve all of your, you know, problems for, you know, security analysis, running, uh, dynamic security scanning or generating automation code.

And then now you automatically have an automated test. These, you know, proxy or recorded playback style workflows. And a lot of this, if you've seen in the past several years, has gone very by the wayside, uh, for core in-house development automation, security teams we're, they're really either doing custom things or best in practice approaches.

We're not click a button, generate artifacts, and now we have to refactor 50% or, or a lot of these things. So I, I parallel that with a lot of these new AI tools and technologies. They're very, you know, it's the new shiny object.

And I think one of the things that we'll see is similar trends where we'll realize that the benefit not is really, is not as much as, uh, was advertised or we thought, and then we're doing a lot of refactoring or secondary, uh, hands-on type of work to get these things to, to really function. Whereas we might've just been easier doing them than ourselves, like actually creating, designed and implemented solutions. Uh, and then maybe leveraging that more strategically longer term given a level of maturity.

Um, so I'll draw that parallel from kind of the testing world and I see something very similar evolving from, uh, ai, especially the generative tools, uh, that are out right now. So what's your best advice right now to DevOps teams as they kinda look at all of this? I mean, um, it seems like there's a lot of moving parts and it may be a little difficult to get your arms around.

So where do you start? I think really the best thing would be, and I, I come back to what I mentioned earlier, and we talk about really two kind of core pillars. I'll talk more in the testing domain.

'cause obviously that's more focus in my background. I'll talk a little bit about security, but I used to do that work as a, as a lead level engineer. Um, I think the big thing is we have to start with design just like all good software, it itself, um, you know, we have to root ourself in good design and, and requirements understanding what we want to achieve.

And then really building on that. So going from, uh, you know, requirements and design and good design practices, having good collaboration with the team. And when I say the team, um, you know, those that are specific in some of these domains.

So maybe involve your test engineers, your DevOps engineers, uh, cloud engineering, if you have those departments, if you're cloud-based. Uh, and then obviously like I said, any compliance environments, um, you know, these type of, uh, compliance environments or compliance officers compliance and regulated industry is another area. And I won't go off too much into that topic, influences all of these factors as well.

And I, again, I've done some webinar topics on this, but once we have a good basis of design, we can go from requirements to design, getting input, iterating on that, and then starting to look at actually what tools best fit these requirements rather than jumping right into specific tools, you know, whether it be, you know, tools that provide us generative AI or specific insights. And if you start there, and I've always had this saying, I'll leave your question on, is that, you know, um, you know, tools, uh, can help achieve process, but really, uh, tools are not processed, right? Um, so we focus on what the process is gonna be and we use tools, but tools are not a replacement for good process.

So I, I'll, I'll kind of be a little generic in that response, but I think if you do that, you'll find best in class tools and best practices to really fit the organization's needs. All right folks, you heard it here. There is cause for optimism.

That's the good news. But, uh, despite the rise of ai, there's just no substitute for hard work. Hey Chris, thanks for being on the show.

Thanks again, Mike. It was a pleasure. All right, and back to you guys in the studio.