Navigating AI Regulations: Balancing Innovation and Protection with Niranjan Vijayaragavan

Hello, and welcome to the latest edition of the Techstrong AI video series. I'm your host, Mike Bazar. Today we're with Raja VJ Ragga, who's CTO and Chief Product Officer for NM Tex.

And we're gonna be talking about, well, AI regulations. They seem to be in a state of flux. And how should people think about that?

'cause, well, on the one hand, it could be a good thing, on the other hand, could be a bad thing. Hey, Rajan, welcome to the show. Great to, uh, great to meet you.

And, uh, excited for the conversation. Thank you for having me. What you've seen in the US especially, there's been a move to kinda loosen the regulations, but even the EU has pulled back somewhat.

Um, a lot of folks on one hand are kind of celebrating that, and others though, I have to wonder if we should kind of maybe just make our own rules going on from here because a lot of this stuff might bite us later, but what's your take on what's going on here and what should enterprises be thinking? Yeah, uh, it's actually a pretty dynamic, uh, domain. 'cause I think in the Biden administration there was a lot of guardrails that were put in place, um, to protect consumers and sort of consumer focused, uh, guardrails.

And then in the new administration, there's been some loosening happening. I think there's always a, a dichotomy between what drives innovation versus where can we be careful. I think our stance right now is we need to be pro innovation, which is where the administration is going.

Um, my guidance would be that is good to drive a lot of innovation in this area that is dynamic and not slow down the momentum, but we have to be careful about having certain guardrails that doesn't actually create challenges later on. Because once technology is further along, it's always hard to pull back. Um, the areas that I would think about, uh, where we need these guardrails is a few different things.

So if you look at the application stack itself, you need, um, guardrails, security around the infrastructure itself, where you're deploying models, et cetera. You need security and integrity in the model domain, and then you need security and privacy on the data domain. And so I think even the application stack, I think those are three areas that we have to be careful about because these, for any enterprise, whatever they are building is global in nature.

And you have to cater to all different, uh, jurisdictions and each jurisdiction may actually move in a different direction. And so those are three areas. And then outside of that, your ability to, you track AI performance, model drift, quality bias, et cetera, is also equally important.

So the good thing is a lot of enterprises are very cautious about this. And so even as, as different vendors are signing contracts, they want a lot of language in their contracts. So there is some level of self-governing happening anyway, which is a good thing.

Uh, but I would, I would actually encourage enterprises to think about all these four areas. How they, how are they ensuring governance around data, uh, governance around the models they're deploying governance around the infrastructure in which AI is deployed. And then, uh, performance related characteristics.

To your point, are there a hierarchy of guardrails that we need to kind of think about implementing? 'cause I think part of the challenge sometimes is people are overwhelmed by all the things they're supposed to think through. So are there some things that are more important than others?

Uh, I think we would, yes, there is a hierarchy. Uh, I would say you want to start with the customer and what do customers care about the most? Uh, which both the, the privacy of their, uh, what they're sharing with the, with the AI capabilities, and then what responses are they getting in return.

So I would, I would be most careful about protecting the assets of the consumer, uh, within their tenancy. And so this would be data privacy, any kind of specific model or model training that is happening related to that, uh, needs to be very explainable. I think those are similar to the, the journey we went with data, right?

So when data became went to the cloud, there were a lot of regulations that came about on what is doable with data versus not. I think similar GREs need to exist in the AI domain as well. And the second piece is the explainability of the model.

So how do we, how are you tracking the quality of the model? How are you tracking any drift in the model? Because what you don't want AI model to do is, uh, output something that could be construed as inappropriate or may actually, uh, reflect poor quality on behalf of the, the, the enterprise that is providing the, uh, providing the capabilities.

So I think that's where I would start. Um, and then work your way down into your own IP protection. And so start with the consumer.

What does the consumer care about or the customer care about? And then go down into your own protection, which is you are building proprietary models. How are you, how are you ensuring the security of that?

Um, and then infrastructure security layers as well, because, uh, there's a lot of ability to actually, um, infringe the model and make, make it do things that, that you may not have tried or you may not have designed the model for. And so that becomes your own security layer. And so that's the sort of hierarchy I would think about.

Protect the cons customer first and then protect yourself next. I feel like sometimes the models are designed to be too helpful and then people either for their own amusement or for malicious purposes kind of try to tease data outta that model that it wasn't meant to surface. And uh, so can we make the models themselves, uh, a little more guarded in their responses?

Or how do we kind of think about securing the models in a way that doesn't allow people then maybe easily exploit them to the degree they do today? Yes. I think, I think guard rails need to be put in on a layer, uh, some kind of authentication layer or some kind of, uh, role-based access control on the model outputs themselves.

I think the model creators or the users of the model, how to put that in place. And so I almost think of the model as a fairly atomic asset that is programmed to respond in a service certain way, very akin to a database, right? Like a database has a job to do, which is to structure and make data access performant the responsibility of who has access to what lives in a separate layer.

And so I would think about models the same way. It ultimately is the responsibility of the people building capabilities to actually surface them in a responsible way where you're not surfacing someone else's information. And that could go down the application stack.

So if, if there's a software vendor who's offering an AI capability, ultimately it becomes their responsibility to protect, uh, unwanted data access. But you could actually argue all the way to the LLM and should the LLMs put controls over this or not. Um, I think the, the access to information that the LLMs have is powerful, and they could put guardrails around specific types of data access, which, hey, give me the, give me the roster of all the people who live in a certain neighborhood as an example, should not be a query that the, the, the, the LLM should be responsible for or should respond with.

I think some of those guardrails are coming around anyway in the LLMs, like certain kinds of queries don't yield a response. And so we just see a evolution of that as we go along. I think at every layer of the service, there needs to be a responsibility to the consumer.

So everyone who's providing the service needs to be thoughtful about what is okay or the customer to get and what is not okay for the customer to get all the way from the models themselves, uh, to the application stack and the software that that vendors are building. Who's in charge of that? Because I feel like I'm kind of witnessing the move to the cloud again, it seems like, uh, security people somehow or other think that the data science teams are doing something about this, and the data science teams think that security people will do something about this and it kind of just falls in the middle.

Um, so do we need to rethink how we're structuring our teams to go address this though? Look, I'm of the belief that security is everybody's responsibility. It's not like a, a particular group that is solely responsible for all security concerns.

And, um, I think that's a good sort of philosophy to run your teams by. And so the infrastructure teams are responsible for infrastructure security as well, right? The application teams are responsible for application security as well.

Um, the model teams, the data science teams have to be responsible for what they are putting out, uh, in the, through the models, into the, the upper layers of the application stack. So I think if you ask me, my, my response would be, yes, it is good to have a central body that is governing our security policy, but I think ultimately is the responsibility of the individual feature teams, service teams, however you define them, to be responsible and put the guardrails on what is okay and what is not okay to do. Uh, and these, if I think about the traditional SDLC process, these need to become requirements and how we build software.

So, uh, just like we have non-functional requirements for many of the capabilities we build, I think this just becomes another non-functional requirement on how we sort of, um, how we guard against unwanted use of the models that we are offering. We started out talking about regulations, and yet if I think about what goes on in security, there may be a, a compliance mandate, but usually it's the bare minimum and most organizations have to do a lot better than the bare minimum to ensure security. Yeah.

So is that same mindset playing out here with ai, whereas no matter what the government says, we need a higher level of standard anyway? Yes, I, I think that is very true, yes. And I think, I think somewhat the market will gravitate towards that as well, because everybody has concerns about how their data and what else is getting exposed naturally because of business performance.

And that'll automatically lead to some level of compliance and regulation. It may not be regulatory mandated, but you'll start to see that language in many of the contracts and those contracts will become so standardized that customers and vendors would have to implement guardrails regardless. So if you look at Azure AI as an example, like the tenancy level privacy they have is because all the consumers of Azure AI want that, there's no regulation that is actually asking them to do it.

Um, and so, um, I feel that that's by design. We are gonna go in that domain where, where government regulations will mandate certain minimums, but the market standard is gonna be over and above that As you kind of partial way through that too. Won't, um, customers themselves, once they see, you know, some sort of incident, they'll start voting with their feet to go somewhere else, right?

Yes. Yeah. Yes.

But the market, I think the market force governs what is required in software by design, right? So if there's a security incident, something untoward happens, this is gonna create a lot of buzz and people are gonna put constraints around whatever they build. Uh, and that'll just evolve over time.

Um, and yes, there'll be legal frameworks that'll come about because what you'll see as a result of that is, um, some of that bleeding into sort of lawsuits and, and regulatory domains. And there might be certain laws that might come about, or maybe there might be secondary or knock on effects of that because certain lawsuits resulted in a certain way. And so now there's precedents being created that'll drive more guardrails into the organizations.

And so one way or the other, the market forces will actually steer us towards a broader set of constraints that balance both innovation and then the right level of guardrails for customers as well as the, the vendors themselves. I'm not sure how many people are checking this box, but you can check a box that says, don't use this data to train the next iteration of this AI model. Um, are those agreements, you know, reliable or people not just, you know, finding that in graph 27 of their user license agreements and therefore they're not checking that box and data winds up in these LLMs.

And unfortunately LLMs don't. I think we, We are very diligent about checking that box. Uh, my experience has been our customers are very diligent about having those discussions.

And even in RFIs and RFPs, you will see a lot of language around this. And so I think if I go back to the data, the data evolution that we have seen, a lot of people have learned what things they had to be cognizant of, and you're seeing a lot of that language and scrutiny, uh, in many of the contractual languages right now. The flip side of that is, even though some of that is a lot of focus data in many ways has been used to train models even before generative AI was a thing.

Like if I go back 10 years, like there's been fraud models and typical machine learning models that usually use a level of obs obfuscation and pseudo pseudonym sort of, uh, hashed functions, uh, to anonymize data and use it for training. And so we just have to be very prescriptive about how if data is getting used, how is it getting used and what's the traceability back to the customer? So, um, I think that level of scrutiny exists.

I I don't think it's getting lost in like the page 27, line number 13. Um, and I think most, most mature organizations, including ourselves, are very cognizant of having a very robust discussion around it and putting infrastructure around it as well so that we can be compliant to what we are telling our customers. Sometimes you talk to business people and they'll say, the only thing that's worse than regulations is no regulations because then they don't know what they should be doing and they suffer a little bit of paralysis of analysis.

So are we seeing that here where folks are kinda a little hesitant because they're not quite clear what they're allowed or not going to be allowed to do in the future? I don't sense that, uh, Mike, so at, at least in my, in the way we are building software and in the way we are actually having discussions with other vendors who we use who provide AI capabilities, it's not been a primary focus to look at what, what government regulation mandates us to do. It's very much catered by what are the things that we care about as value and protection for us, and what are the things they care about as value and protection for themselves?

And that's, that's generally defining what guardrails we need to put. I think going back to our earlier conversation, I think that dynamic is ultimately gonna define the standard, uh, more than more than government regulation right now because there's so much activity that the peer feedback and the peer standard is setting the bar versus what government is gonna come up with. And I think ultimately government regulation will be somewhat influenced and informed by what that standard becomes.

And because majority of the companies will be adopting a certain level of, of guardrails and security and compliance within their AI infrastructure as well as their models and, and, and the software they offer, uh, that, that may become the standard for, for regulation, um, which is not a bad model actually. Um, it's, uh, it's actually a good model to land towards what the market deems as the right equilibrium, um, for the consumers as well as the providers. So among the companies that you see managing this process well, what are they doing that kind of sets them apart from everybody else?

Uh, there's a heavy focus on customer level, uh, tenancy and privacy, uh, which gives you a lot of comfort that your model, your whatever your, whatever data you're sending is, is not getting trained elsewhere. Um, I think a lot of companies are focusing on explainability and auditability. I think that's becoming a pretty hot topic.

Uh, I think it's still an evolving field. I don't know if people have perfected it, but, um, your ability to explain why the model did something, um, will become more and more important. And as we use it across sort of business workflows, which is our primary domain, um, Nintex helps automate a lot of business processes, business process by themselves, need a level of traceability and auditability, like without that, like, there's a lot of things that will get, uh, scrutinized and, and customers are not gonna like that.

So the ability to explain every step of the way why certain decisions were made will become more and more important. Um, ability to trace that, log that, uh, explain that, um, is the capability that people will start to value. And so the leaders in the space will, will take those compliance elements that have existed for a while, seriously.

So both from a data privacy and, and, and, uh, protection perspective, but also explainability of models and making sure they're repeatable and they're doing what they are doing. There's no model drift, there's no quality deterioration. Um, and to do those things, you not only have to monitor those things, but also secure your model and infrastructure because you have to avoid somebody hacking it to the model and changing it.

All right, folks, I heard in here there's a lot more focus than ever on how, when and where data is being used. And that's the good news, and it may also be the bad news. Hey, Rajan, thanks for being on the show.

Thank you so much. Good conversation and great to meet you as well. ai video series.

You can find this episode and others on our website. We invite you to check them all out. Until then, we'll see you next time.