Securing GenAI in the Enterprise with Opaque Systems’ Aaron Fulkerson

This is Techstrong tv. Hi everyone. Welcome back here to Techstrong tv.

You know, this next gentleman here. I know, my God, it's gotta be 15 years at least, right? Maybe more.

Yeah. Um, this is, let me introduce you to Aaron Fulkerson. Aaron is the CEO of Opaque Systems.

We're gonna hear all about it, but Aaron, I, I, we gotta start from where, how, where I know you're from, which is a company called MindTouch, and wasn't that San Diego was MindTouch San Diego? Yep. Why don't we start there and, and tell people kind of the Aaron story up till today from there to now.

Okay, sure. Um, yeah, so MindTouch was, uh, one of the first companies I did, uh, in, in, uh, uh, software. It was, it started off as an open source project with a buddy of mine, Steve, and it became very popular as an open source project.

Uh, it was kind of general purpose collaboration. People were using it for a variety of different use cases. Um, we, we then, uh, narrowed the focus of the business to be on, uh, customer self-service for support and, uh, staff knowledge management.

So we built like a really sophisticated knowledge management solution that would help customer self-serve and agents deliver better responses as part of the, uh, customer support channel. And, um, had a real successful run with that business and then sold it to, uh, nice systems, uh, several years back. Excellent.

And then, uh, from MindTouch, you were at, uh, ServiceNow for a while. Correct? I joined ServiceNow in 2019 and, uh, what a terrific company that is.

I had the privilege of, uh, helping them take their customer workflows business unit from, uh, its early days to just a, a huge success. We, we, uh, grew the business very rapidly in the two years I was there, um, from what was a fledgling business that, uh, you know, folks like Salesforce probably didn't care that much about to within two years. Uh, we were, we were hearing about, uh, what a threat we were to, uh, Salesforce Service Cloud just within two years.

Um, and then I launched a new business unit at ServiceNow called ServiceNow Impact. That was, um, a, a huge success for the company. We, uh, broke every record for customer adoption and revenue within the first year of general availability.

Crazy, crazy stuff. And, and ServiceNow is a juggernaut in every, in every sense of the word. com, right?

Yeah, that's correct. com, but it's dot co. Yep.

That'ss, why I mentioned here The opposite of transparent, uh, opaque. Exactly. Um, which, which is, uh, fitting for, uh, the technology that was developed.

So let, let's, let's, you know, let, let's get the opaque story out of the way. Yeah, Well, I'll, I'll tell you how I, I found the company, um, when I was building impact at ServiceNow. Uh, we were building AI into the product, of course, and, uh, we were challenged because the data that we wanted to use to drive personalization was customers usage and telemetry data.

So we, we wanted to take data from how the customers use ServiceNow products to train AI to drive highly personalized recommendations. But a lot of our customers opted out. And what I was, uh, grappling with was, um, I saw this supercycle around ai, you know, generative ai, but I'm gonna use the term ai broadly speaking.

So analytics, machine learning, generative ai. I'm just using AI as a general term. Got it.

Clearly AI is the most important supercycle in human history. It's the largest technology supercycle, but it's colliding at the same time with a cultural trend that is the erosion of trust in society. People don't trust each other.

And a lot of where this came from is the high tech companies over the last two decades, mining personal data, refining it, I mean, let's be candid, weaponizing it to make their products addictive or buying things on their platform addictive. So what I saw was the most important technology supercycle in human history, colliding with this cultural phenomenon of the erosion of trust. And when opaque approached me, I just felt that it was such an unique opportunity that I had to leave what was a very lucrative position that I loved at a great company.

ServiceNow. So opaque, what is it? Uh, there's this famous computer science lab at uc, Berkeley that's run by a fellow named Jan Stoica.

Jan and his team of researchers are the creators of Apache Spark. Mm-Hmm. Which became Databricks.

Jan was the original CEO and still remains the chairman of the board of Databricks. Also, the team created Ray, which is a foundational technology for AI machine learning, uh, that became any scale. Uh, Jan also serves as the chairman of board for any scale, and the team also created a foundational technology called Mc two, uh, which became Opaque Systems.

Wow. So each one of these technologies have a particular focus on ai, uh, opaque, as you might imagine, uh, is focused on building confidential data pipelines for AI so that you can deliver trusted ai. Love it.

What a great story is, is he also the chairman of Opaque? Uh, no, but he serves on our board and he's one of our founders. Got it.

Very cool. Um, and now look, you know, Jen AI burst on the scene. And where are you guys in terms of a product?

Is it a generally available product? Is it Yes, that's correct. So Opaque built a platform that allows you to, and, and this is, this is the, the magnificence of the platform is you can run AI jobs on encrypted data sets.

So let me restate that 'cause it's kind of a, a mind bender. You can run analytics, machine learning and other AI jobs at massive scale like spark scale on data while still encrypted. So there's some, some magic that that works, uh, that allows you to have highly performant, highly scalable because it takes advantage of, of some CPU capabilities that others have not yet unlocked.

Um, but the opaque software stack originated as a way for like financial services institutions or insurance institutions or government agencies to take multiple data sets from multiple parties, share the data in opaque, encrypted, and without showing the data mine insights. So things like money laundering, human trafficking. Um, uh, more recently we've, we've got a customer in the European Union that takes EU member country data.

It's while encrypted and identifies ransomware attack vectors, does alerts and signals off of the data sets, and then is designing countermeasures by processing the encrypted data. So you're sharing, not showing the encrypted data so that they have countermeasures. That's fantastic.

So that, that's where it was born is, is like more general like Spark or Python analytics and ml. And then, um, I joined in September and the team had built a gateway product for generative ai. So I, I'll I'll say what we've seen is, let's say 2023, the, um, estimated market size of AI was $200 billion in 2023, and yet about 3% of AI projects went into production in 2023.

Now, that's a huge problem. So everybody in 2024, this is the year of productionizing your AI projects. But the factors preventing these projects from going into production principally is, uh, data concerns.

The problem is data, not ai. And the problem specifically is while the regulatory and security landscape is constantly changing, people are concerned about data security and privacy. Furthermore, I hear from our customers and companies I'm talking to data sovereignty is their principle concern.

This is our data. If we use it in some generative AI implementation, how do we retain sovereignty of the data? So that's why the team built a gateway for generative AI on the opaque platform.

What it allows you to do is to have a user write a prompt, and obviously people are going to be doing some kind of augmentation of that prompt through RAG or some other technique. So when, let's say Fidelity has a wealth management advisor who writes a prompt to support Tracy Garcia, who's asking for tax advice, uh, once they write a very simple prompt, multiple, commonly it's multiple, sometimes it's one, but what I'm seeing is there's multiple data sources that are doing rag augmentation on the prompt. And that data that gets put into the prompt invisible to the user who's writing the prompt consists of proprietary and sensitive information.

Tracy's home address, Tracy's accounts, the proprietary investment playbooks that the company's running with Tracy, the gains and losses of her accounts gets added to the prompt and sent to the LLM. The LLM processes the prompt and comes back with a quality response because they were able to do rag That response might be something like, tell Tracy to sell her tech stock stock losses in 2023 and buy these ETFs that align with the investment playbook and our propensity to buy. Now with the opaque gateway, what we do is we put a middle layer so that as soon as the user writes the prompt, it goes through rag augmentation or some other augment, whatever the augmentation is, and it's encrypted.

Nobody can see the data. The user who wrote the prompt owns the encryption key, and that prompt gets sent to an opaque gateway within the opaque gateway. We're processing the data and we're doing prompt compression.

We're able to do NLP standardization of PII think of it like a firewall, but not for your network, for your data. That allows you to retain security, privacy, and sovereignty. And then after we're done processing it, it gets decrypted, but the data's been tokenized, so there's no PII in it or sensitive or confidential information.

It gets sent to the LLM. The LLM provides a response, it comes back through the gateway gets re-encrypt. And some of our customers wanna do post-processing, for example, we've seen where people have done fine tuning of their LLM models and they wanna make certain that that fine tuning data doesn't come back out of the LLM, no problem.

You can do post-processing inside the gateway to ensure that none of the trading data is being sent back to the user. We do it all in this encrypted environment. It goes back to the user de anonymized decrypted, and they're able to get a quality response.

Wow. So Aaron, I know you guys recently came out with a white paper securing gen AI in the enterprise, is what you just described, basically the, the model that's, you know, described in the white paper or is that something different altogether? No, it's the same.

So that's it. Um, the opaque founders are some of, there, there's several PhDs who are founders of opaque, including Stoica. Uh, they have, uh, specially specialties in security and ai.

Um, so, uh, re Luca, ADA Papa or Sha Putar, uh, they co-wrote a white paper. Um, re Luca did her PhD at MIT and teaches at Berkeley and Rashab, uh, did his PhD at Berkeley. Uh, their specialization is in securing ai, uh, and they put together a white paper that goes through all of the different techniques that, uh, people can use and, um, covers the techniques that we're using, uh, in our software at Opaque that we make commercially available for others too.

I love it. Where can people maybe grab this white paper? co Mm-Hmm.

And there's a resources section, which is accessible right off the top menu bar. Click on resources and you'll find the white paper featured on the resource section above the fold. Love it.

Aaron, you know, we only have 15 minutes or so. We're probably over that, but I want to thank you for coming on here, telling us about Opaque. ai.

And, and this is the kind of information that our audience there really, really enjoys. So I'd love to have you back on and keep, keep us up to date with the opaque story as we talk more and more about ai. We do a lot of events and virtual events as well.

So love to have you on in discussing with it. Best of luck with Opaque, it's great to have, you know, be connected here again. Yeah.

It's so great to re connect with you and I hope to see you in person soon. Uh, I hope so. Hopefully you're at the SSW two conference in, in, uh, Colorado this year.

Um, I will try. Which is Where we met, like way back in Two. Yeah.

Way, way back. Yeah. Brad, Brad Fel and Eric, nor lead the introduction.

Yeah. Yep. I, um, you know what, I'll look at it.

I'm, I'm building out my, I've got a lot of conferences I'm going to, but I, let me see if I'm on that one. I'd love to. It's not E Eric Newland's not doing that one.

Is he still, or no? Yeah, That's Eric's conference. It's his new one.

Yeah. Correct. All right.

I'll reach out to Eric. Maybe we'll do a, a video thing there. Alright, Eric, great seeing you.

And best see ya. Best of luck with Opaque Systems. Great to see you again.

Bye. Alright, bye-Bye. We're gonna take a break here on Techstrong.

Stay tuned.