Pankaj Chawla on Enhancing App Quality and Security with GenAI

This is Textron tv. Hey guys. Thanks Viter.

We're here with Pancha Chalo, who's Chief Innovation Officer for Three Pillar Global, and we're talking about Gen AI and testing in particular, we have seen the rise of gen AI tools. Uh, we're not quite clear where they fit in, but we'll explain that in a minute. And of course, all these tools need to be integrated into our DevOps workflows.

Jun, welcome to show. Thank you. Thank you, Mike.

Thank you for having me. Excited to talk about the subject. So walk us through, I think everybody's kind of loosely aware that we can apply Gen AI to testing, but I think we're a little light on the specifics, so maybe walk us through a little bit as to how you think that this might change the way we approach testing.

Of course, we've been using AI models and testing another forms for a while now, and this is probably in addition to that, but kind of bring us up to speed. Yeah. Um, lemme start by saying that, uh, we at three Pillar, uh, global, uh, is a company that is in the business of building software and digital products for our clients, including enterprise custom solutions that are, um, needed to build like products in general.

So in our, in our business, we come across, um, end-to-end software development, life cycle and testing. Uh, and DevOps is clearly baked into the whole cycle from end-to-end. Nowadays, that's the right thing to do, the primary way to build software, especially if it's a large, um, initiative.

So we run into, um, quite a bit of demand when it comes to doing testing, doing DevOps, uh, and they, they go hand in hand. And I'll explain that in a second. You know, testing, testing can be a lot more localized and faster if you're able to, uh, when you're building new features, you are able to spin up, spin up environments very quickly, test out that new addition that you did to the product and then spin 'em back down.

Um, historically that's been done through scripting. Um, when it comes to DevOps, you write Terraform scripts, you write configuration scripts to set all those things up. Um, it's a tedious grunt work, taking a lot of effort, um, by the DevOps teams to, to manage these operational environments for developers and developer engineering.

On the other hand, on the testing side, similar amount of groundwork, uh, where you are and if you wanna do it right, you should be doing unit tests as you write code. As a developer, you should obviously be doing regression tests, end-to-end integration tests. And, and again, a lot of this work is automated over the years through automation tools, um, like Selenium, selenium or whatnot.

But it's still, I still call that a lot of drug work. Um, typically they slow it, slow us down. They create a lot of friction, um, when it comes to productivity and speed to market type, uh, features.

And Gen AI is really exciting. How, how it's coming into play, how these models are getting baked into the testing automation tools. The, the terraforms script creation, because they're able to study through natural language and, and taking prompts through na natural language, they're able to study what an application does and, and kick out, um, you know, initial set of test scripts, uh, whether it's unit tests or integration tests or regression tests.

And as the application changes itself, um, these tools have the ability to keep up with the changes, which is again, historically being quite painful. Um, uh, like you would, like, you would expect, let's say you write a script, uh, using user interface as, as one example I'll throw out. You have all these fields and you are writing a testing script that's automated and it's, it's used as a, as a exit criteria for every hot fix, every release, every service release, and then you make a small change.

And then now you gotta go back and change all the scripts with Gen AI's ability to understand the context, understand the nature of the application, it really speeds up that effort. It speeds up, keep creating and keeping up with the changes, as well as keeping the up with the environment. So a lot of work can be done in parallel, uh, and not serially.

So it's an exciting time know to, to have these, these, this AI based solutions come in into, into the mix. com and they had a, a serious criteria that you cannot put any new changes in the environment unless you have 70%, seven, 0% unit test coverage successfully passed. And I'll tell you what, it took more time to get that criteria passed versus actual actually doing the coding and delivering new features.

So it, it does, like I said earlier, it does create a lot of friction for, for us to have speed to market. And, and that's where this, this creates the best benefit. We have been talking about continuous testing for a while, but to your point, it was hard to do.

So are we getting to the point where we can do continuous testing because the AI will help us with that? I mean, to your earlier point, it seems like a lot of testing doesn't occur simply because we let it build up and then it takes too long. So we skip, We skip or we slow down releases.

Yeah. Um, that is, that is the nature of the game. Um, continuous testing is wonderful.

com was dropping 3000 changes per day in their production. Now it's, now it's up to 3000 changes per hour in their production. And it's all through DevOps and testing automation.

Uh, but it's still a lot of, through a lot of brute force, a lot of brute force, a lot of developers. Um, but with the tooling that are, that's coming into play, it does become more of a way of doing things naturally baked into your software development lifecycle. And continuous testing is one piece of it.

So I, I try to combine DevOps and testing ops into one category, if you will. Um, they, they were separate. In fact, they were very separate.

And one time DevOps didn't even exist as, you know, uh, given that, how, how much of work you have done with DevOps. Um, so, but, and they've been slowly coming together and now to a point they've come together so tightly that, um, I I I, I am saying that even the customer support work that used to be separate, separate and decoupled is now getting mixed into this three, three-legged stool, if you will. Uh, what I mean by that is, traditionally you release a release or feature, something happens and, uh, customer reports a problem with it as an incident, and then you use tools to manage that incident.

Well, why not proactively know that that incident is going to happen in advance and fix it before the incident actually happens? And that, that level of customer monitoring, customer usage of the systems and customer support was an afterthought. Now it's getting baked into your basic DevOps cycle.

Mm-Hmm. Um, How, uh, complete is all of this gonna be? And I asked the question from the perspective of, um, are the tests that we're kicking off gonna be integrated with every gate and every part of the lifecycle of the DevOps workflow?

Um, 'cause historically somebody kinda had to make that happen sometimes by hand. But is all this gonna be much more of a hand in glove kind of process all through every stage of the DevOps workflow? That's, that's the, that's been the vision all along.

I'm sure that, that, you know, that, right? Um, the, the end-to-end continuous integration, continuous deployment cycle in an ideal manner, DevSecOps is baked into every gate, but like you said, it's been, a lot of this has been manual or not carried through. And the exciting part now is that there are no excuses.

You are able to bring in these, these, uh, these processes in an automatic fashion during the engineering cycle. But also, like I said, in the production monitoring scene, that was, that was, that was looked upon as a separate instance to monitor companies like Dynatrace is bringing that kind of ability right into the DevOps cycle so that you can not only de deploy in a CI ICD pipeline, but also monitor immediately so that you're able to predict failures before they happen. In fact, we have a, we have a client, um, big data center client.

They, they push code in their data centers to do a lot of the things automatically, but they're baking in their AI capabilities through our help to, to monitor these, these logs like lots and lots of logs. The, the data exhaust that happens with all these logs and through gen ai, it's, it's able to see things whether, whether it's, whether it's something more sophisticated like, and the heat levels are going high or there are failures about to happen. It sees that and, and brings in that early warning systems in place.

There's a lot of folks who are concerned that gen AI will result in more imperfect code than ever being pushed through our workflows because, well, the models weren't necessarily trained on the best examples. And so we'll get something subpar on the other side of that equation though, might we not get better testing so that ultimately we might get to better applications just because we have better testing? Yeah, I mean that's a, that's a fair point.

Uh, there is still, these are still new. Um, so there is a fair level of concern with tools like GitHub, how good they are, because they were trained on a very much very large sets of open source code and data and, um, you know, people would just plow, you know, um, paste, um, um, stack overflow type examples, small snippets of code, and now it has been trained on it. So yes, it can bring in insecure code or not complete code.

How do you handle that? But the big model providers are, are recognizing that and they're creating more and more focused and dedicated, um, models that are designed for, for, um, accurate way of generating code. An example of that is we, we, we have been using a lot of product called tab nine.

You may have heard that, um, tab nine is far superior than GitHub. Um, for example, copilot because tab nine was originally trained on very much signed off code, uh, in production code that really made it as a legitimate code in production versus a broad set of open source libraries and, and code snippets across the internet. Um, and not only that, you can take that night and actually train it further on your own enterprises best practices and code in place.

And that delivers a far more secure way of testing and delivering on, um, solutions. Mm-Hmm. Quality has to still be there.

Um, your question is really valid. You still have, you cannot still, you still have to do code reviews. You still have to make sure that the, that the code created by automated tools is making in correctly, but it does speed up the process.

Do you think as part of this, we might converge DevSecOps and quality assurance? 'cause we've been kind of talking about that for a while, but there's still feel like very separate motions within our DevOps workflow. Yeah, that's what, that's what I'm excited.

That's what I was trying to say earlier. Not only just DevSecOps, but, but DevSecOps testing automation and ongoing monitoring and support needs, need will, will indeed get, we'll see it coming together as one set of processes versus, like you said, siloed processes that have their own tooling. So what do you think the future of a DevOps team looks like?

'cause in my mind, I can envision there's a bunch of AI agents trained for specific tasks that are working with a, alongside a bunch of humans who are managing the overall process. And some of them are optimized for different testing functions. I mean, is that kind of where we're headed?

Definitely where we're headed, and definitely not without humans. Uh, I don't see the world where humans are not engaged. Um, you know, it's, it's, it's, it's like any other tool.

You know, we've been building rocket ships and, and, and cars that are far superior to human capabilities, but you still need humans to be overseeing and creating those machines. This is no different. We're creating far superior and far faster engines, but human supervision, especially as the, especially the other thing that I'm seeing is the, the nature of the applications themselves are evolving.

So, you know, just like, just like when we used to have simple applications on desktops that gave way to web applications and mobile first applications and digital first always on, always connected, scalable apps that we see nowadays as a, as a given, the complexity of our applications will, will get larger and bigger, uh, because of tooling that we're putting in place because we will be able to do a lot more sophisticated stuff. So I do foresee humans driving and sharing these com complex applications, building new category app of applications altogether, um, with, with the assistance of these tools. An example of that would be, for example, in a nursing home, um, you know, today's application says that you have a nurse looking at a patient, you have a text base interchange, uh, of an application's all text related back and forth.

But in the future you will have categorically new applications that takes an audio, video, text as inputs and audio video, text output. That's a new category of applications that we're not building today. Mm-Hmm.

And so, so, so the, the advancements will increase with these tools. What's your best advice about how to go about getting there? And, and I ask this question 'cause a lot of organizations already have DevOps workflows that they've kind of been building out and customizing for all these years.

And do I need a new platform entirely that is AI driven, or can I add these capabilities over time and stitch 'em in there and alongside everything else I already have? Yeah, you know, you don't have to, uh, one doesn't have to. In fact, we're not seeing that there are clients.

We're not throwing away the work we've been doing over the years with DevOps. Um, the good news is that the tools that are there, and there are new category of tools coming up as well, but the, the established players are adding this AI features into their products. And even though we have been building DevOps and testing operations and automations for years, like I said in the beginning of the call, we struggle with keeping up with it.

There's a lot of grunt work that's there and it's, it's making it more efficient, making it more, um, reactive versus throwing it away and starting with a brand new AI platform, you know, so, um, there there are some new tools that are, that were not there in the past. Like I give an example of tab nine and GitHub copilot and Amazon Whisper Code Whisperer. But, um, they will be baked into the existing DevOps and, and all, all these tools have the capability of reading the existing code, providing the nature of what the code does and providing, um, recommendations on how to, um, steer this, this, you know, historical code into the new direction with the changing applications.

Mm-Hmm. One of the things that we're all kind of trying to figure out a little bit is that what rate will software be developed going forward? And I asked the question 'cause some people say, you know, more software will be written in the next two years than have been written in the last decade.

Um, AI not withstanding is that feasible because, well, you know, at the end of the day, humans has to manage all this, and I'm not quite clear that, uh, testing faster is gonna make me be able to manage more projects simultaneously. But what's your take? I think, I think, um, I think it will be possible to write more software.

I don't mean more lines of code. I mean, it may create more lines of code, but not the lines of code that we will have to worry about too much. The best analogy I can give you is, you know, I, I'm old enough to start, start when I used to write an assembly, um, and then came higher level order of application, um, sorry, development languages like C and Java.

net to build applications, it generated a lot more lines of code under the hood. But I was still dealing with higher order of logic using higher, higher order of programming languages. So we have a lot more software out there in the world than we ever did in the past 20 years, but that hasn't resulted in more necessarily more lines of code.

It's resulted in higher order of application development capability resulting in higher order of complex application functionality that we did not have in the past. So, um, you know, you just, you can page an Uber, it shows up and it shows your maps, it does your billing, it does, you know, it does the right completion and does security and compliance checks. All that stuff is happening behind the scenes with a very few lines of code, uh, compared to what would've been the case if in the past we had to write that in assembly.

All right, folks, you're hearing it here. We're not just gonna have a lot more software. We're gonna have a lot better software.

Hey, paja, thanks for being on the show. Thank you, Michael. All right.

And back to you guys in the.