Impact of AI Data on IT Infrastructure with SymphonyAI’s Raj Shukla

Hello, and welcome to the latest edition of the Techstrong AI Video Series Army host Mike Bazar. Today we're with ralo CTO for Symphony ai, and we're talking about, well, the impact that all this data that we're collecting for AI is gonna have on our enterprise IT environments. And for that matter, what to expect from AI in the enterprise.

Hey, Raj, welcome to share. Hey, Mike, thanks for having me. Glad to be here.

What is your current sense of where we are on this journey? 'cause I think we had a massive amount of enthusiasm for all things AI last year, and then we got into it and discovered, well, maybe we don't have our data organized enough, and we certainly maybe don't have the right infrastructure in place. So where are we on this journey?

Yeah, I think, uh, you know, for people who, uh, just got enlightened, uh, from chat GPD onwards of what AI is, uh, I think, uh, there was a bit of a process to, uh, understand how to apply AI on your enterprises, because here, there was a model or how large language models bond is, they are trained on a lot of publicly available data, and then they don't need to access any of that data. So you ask it a question and it just replies and, uh, you know, uh, uh, a lot of CIOs and CXOs expected the same to just apply to their side. But in reality, uh, when applied to enterprise data, the models are trying to act on that data so that that data has to be prepared, has to be ready.

Uh, you know, if you imagine how Chad GBD was trained, there is years of data processing and prep that went into before trading that model. So in some sense, the equivalent of that had to be done on the enterprises. The AI companies knew how, how hard it was.

The enterprise AI companies knew how it, how it was. I think the, the, the, the companies applying AI themselves didn't understand how big massive of a task it is. But, you know, one way or another, uh, that realization has, uh, has come in, uh, I would say, uh, you know, in, in a lot of cases in enterprise, uh, structured data, you know, transactions, tables and all that, that has had a long legacy, a lot of infrastructure around it, data warehouses, et cetera.

Uh, but unstructured data and how to, you know, uh, files, presentations, PDFs, images, uh, which tends to be, uh, you know, even more than what enterprises have a structured data, uh, it was, it's, it's staying in a, a state of transition of how to make it ready for AI to act on it. Mm-Hmm. One of the little secrets out there is that, you know, companies like, uh, OpenAI and whatever, paid actual people to sit there and classify data for a very long time, and, uh, they spend in some cases billions of dollars doing that.

Yeah. Average enterprise, a lot of the data isn't classified at all, and we have to go through that whole process. Is there some way to automate that thing, or do we gotta like really sit down and do this manually?

Uh, that's a great question, and I think it's important to understand that you can get a lot of value out of just unsupervised AI techniques. Meaning you don't need to always go and label all the data with humans to get a value out of the data. Uh, the second thing to know is you can use, uh, these large language models, which are out there, they are pretty good annotators themselves.

So you can probably extract 80% of the value in case you need to annotate or get your data judged, uh, without the need for humans to do that as well. Um, and, and finally, you know, if, if you want, uh, a highly supervised model for your use case, then yes, there is a need for, for data annotators. Uh, but I think the, these scenarios are, are, uh, anywhere between unsupervised, semi-supervised, which is, uh, uh, you know, helped by these large language models and then fully supervised where you need that data annotation.

And where you are in that journey is, it's, it differs a lot based on the use case, based on the model you want, based on the outcome you want. It also seems like we're all trying to extend these LLMs. It's not clear to me that everybody necessarily wants to build their own LLM, but certainly we are trying to use techniques such as rag to extend them or semantic search or whatever it may be.

Um, but I'm not a hundred percent clear that we understand what data to expose when and how. Sometimes I feel like it's a batch process. Other times we want it to be in real time because the response has to be immediate and sub-Second kind of thing, do we really understand, uh, the data workflows we need to create?

Uh, I think that's a, the most important question that enterprises should focus on, or they're, uh, as they're trying to adopt ai, uh, you know, what their scenario is, what the SLAs are, what data pipelines, training pipelines, and then inference or serving pipelines they need to have to serve those sles. Uh, in some cases, you know, uh, you know, uh, symphony AI serves like, uh, uh, fraud detection and, and financial crime scenarios there, you know, the sles are, are sub police second or like subsecond and in few milliseconds, et cetera. And so you know that a lot of the LMS out there cannot directly work.

Uh, but you also know that you could build a layered system where, uh, there is a layer one or level one of a model, uh, that is just getting the candidates out there. And then there could be a level two model that is trying to, uh, rank them, or there's a level two model that's trying to refine them. And so, uh, that's very similar to how a rag architecture works.

Example, for example, like the level one is candidate generation, and the level two is, is trying to get the right answer out of it. Uh, you know, and in terms of getting your data prepared for it, uh, it's a journey. Uh, for anything, as I said, unstructured data, you have to think of it as always this two step process.

You know, you have to create a search or a retrieval access to it, uh, and then you have to probably build some kind of knowledge layer on top of that. So having, uh, you know, having, having a search retrieval index, whatever you call it, and having some kind of a knowledge graph on top of that, if most enterprises keep those two things in mind, and we definitely keep in mind for our use cases and customers, uh, then you start to put a structure around all of this. Mm-Hmm.

Is part of the challenge, therefore, orchestrating all of this. 'cause to your point, there are multiple LLMs at work, and sometimes I'm using an LLM to check the output of another LLM, and in other instances, I'm trying to create a process that spans multiple L lms. And now we talk about, um, agentic ai.

So now I've got multiple agents at play, and these things require a certain amount of finess and skill. So how do I orchestrate that in a way that creates something meaningful, Right? Uh, absolutely.

I think that's the, uh, if you think about what is the hidden, uh, magic or, or, uh, kind of the shovel behind all of this, uh, uh, besides the large language model themselves, the orchestration layer, uh, above it has emerged as, uh, probably where, uh, you know, most of the magic goes in some sense. And there's a lot of players out there. There's the Lang chains and LAMA index and all that, which are typically known for, uh, conversational or chat bots.

Uh, but then enterprises have long running workflows and, you know, things where in multi-agent system one, agent two agents interact and they have a recommendation, they have to send it to a user, and they have to sit and wait for that response over email. Uh, so this is still an emerging field. Uh, I think the orchestrator companies out there are trying to figure out their role in the lms.

The companies that be on LLM abstractions are trying to figure out their role on the orchestration. Uh, you know, uh, my recommendation to the companies out there is again, uh, uh, some of these orchestrations are pretty well understood, conversational or, uh, copilot orchestrations are pretty well understood, uh, out there. Uh, on the other side, if your orchestration involves multiple hoops, multiple humans in the middle and actions, uh, then you've gotta, you know, uh, plan a little bit better for it.

Uh, we certainly do that for our customers, uh, in our use cases, and we package it, uh, in our, you know, retail financial crime or industrial use cases. Uh, and I, you know, but it's an emerging peak. I think it'll evolve over time.

The other thing that seems to be at work here too is, um, I'm gonna be using multiple LLMs that come in different sizes. Some are small, medium, and large, and, um, I need to figure out when and where to use those things. 'cause some of them I might build myself if they're small and domain specific, and it's in my knowledge wheelhouse per se.

Um, but I wonder also the infrastructure underneath that. Sometimes I think we're a little maybe too GPU happy. So are there instances when I can use less expensive processors that I might actually be able to get versus GPUs that are hard to find?

Um, do I need to kind of really think through what processors are to be used for the right job at the right time? Uh, no. I think there are enough abstractions out there now in terms of, uh, you know, fine tuning pipelines or rag pipelines that you should not have to worry about, uh, what architectures and processes, uh, what you should worry about or any enterprise should worry about is do I need all that?

Uh, and, you know, if it was a, uh, hyperscaler or, or, or, uh, some other companies in this space, they will obviously tell you, yes, you need, uh, your own, uh, fine tune model, or you need as granular a model as you, uh, as you need, as you can. Uh, but in reality, I think there is, uh, there is levels at which a company should think about it, right? So again, I would apply a bit of this framework.

Like if it's a search problem, you probably don't need, uh, uh, your own, uh, fine tuned LLM in, in many cases. You want it to be a search problem. You want it to be for the answer to be grounded on, uh, what your data is and what it is exactly saying versus trying to train a model that will learn a bunch of that and then combine, uh, some of that knowledge across those data sources to come up with some mixed answer.

Uh, so, uh, you know, thinking of it like a search problem can probably be solved with rag, uh, versus it's a, with something that is very proprietary to you. I think many enterprises think their data is very proprietary, and there's a lot of hidden knowledge in it, only to realize that there's a lot of that knowledge out there already. And so that brings us to the topic of evals, right?

And, and I'm a big fan of LLM evals or evaluation, uh, frameworks. Uh, uh, I think if, if companies build evaluation frameworks or test sets before they go into deciding whether they need to fine tune or not, uh, they would realize very quickly they can realize how different models compare. Uh, you know, A GPD four oh versus a cloud or something else.

They can compare how simpler models, uh, small language models or even rag architectures compare on it. And then based on how far they are, uh, you know, they can decide how far they have to go to reach that. And then it becomes a sort of, uh, accuracy and cost trade off.

Uh, that's, uh, you know, we certainly help our customers with it. We sit a level above, uh, the foundation models. So we focus on use cases and problems.

We have clear metrics and evaluation on it, and we help make those trade-offs that, uh, if we are getting a, you know, 76 versus 74% reduction in false positives, but if we are reducing cost by, uh, 10 x, then it's a fair trade off to make. Uh, but having those evaluation sets or having those LLMA valves in place is perhaps the, uh, most understated, most, uh, under, uh, appreciated work, uh, in the industry right now. And I am, I am of the belief, uh, that belief that ideally our hyperscalers and, and other companies should have led with that.

They should lead with saying, do you have an evaluation set? And then say, then do you need a fine-tune model or something else? But, you know, uh, uh, that, that's, that's my view on this whole thing.

So ultimately, and to me at least, there's always this kind of gut check about bill versus buy, and you guys have already done a number of use cases using Gen ai and how many organizations are just really gonna be better off kinda buying something as a service to handle a use case versus, you know, doing all the cartwheels that we just described. Yeah. Uh, I think we deal with that question all the time.

I think the, uh, uh, there is on one extreme, if you think about it, uh, completely horizontal, you take in a model and it's, uh, trying to solve a very broad use case like a Excel copilot, right? Anything you have in Excel, you can ask a question or a word copilot. Uh, on the other hand, there is very specific use cases no one in the world has seen or heard, which is very specific to your company.

What we realize is that at least in, uh, many of the industries or verticals we work in, uh, the use cases are common, right? In a, uh, in, in financial crime, the money laundering use cases is common across max or fraud use cases common. Uh, in retail, uh, demand forecasting or planogram compliance use cases are common.

They don't differ across industries. So your, your common logic would say, you know, my model will probably be very similar to my competitor's model or, or something else. And that's where a company like Symphony AI comes in, we say, yes, it's true.

Uh, the model template will be very similar. The model, how it behaves when trained on your data will probably be different because your customers are different. You are in different region and all that.

And so, uh, I think the lens, and we are very upfront with our customers, I think the lens you have to put is, is my use case very horizontal that every company in the world will have it? Is my use case specific to my vertical that me, my competitors and that market has it, or is it very specific to my company? If you can, you know, categorize your use case into that, you will know, uh, where you could do build versus buy decisions.

And honestly, depending on obviously your IT budget or your r and d budget, you can make different decisions. But, uh, we like to believe that in these vertical use cases you would buy from a company like us, you know, horizontal use case you would buy from a hyperscaler and in a very, uh, company specific use case, you'd have to go to a vendor or a consultant or, or you invest yourself. You've been at this a while, obviously.

So I guess my last question to you is, what do you know now that you kind of wish you knew when you first started down this whole journey? Uh, you know, I think the, what I love about the area we work in, which is why I have been in, uh, AI machine learning space, is there's just constant evolution, right? So you never, you never know enough, right?

So in some sense, your, your question of, you know, you having been in this, yes, it gives you that experience, but you kind of always have to be on your toes and you have to always, uh, be, you know, provide the best to your customers for what out there. One of the things that excites me in this areas and verticals we work in is we have an opportunity to know the best in the latest models out there, latest techniques out there, and bring it very quickly to our traditional industries and verticals. That usually would've taken five, 10 years to do that.

What I find very fascinating now is that time to market from the latest in technology to that on the ground can be very small. It can be as small as maybe a quarter or two quarters. It used to be many years.

Uh, and that has been brought up by, you know, all the advancements in, uh, in different fields of ai. Obviously generative ai, but also predictive AI models, also the automation tools we have around them and so on. Uh, so, uh, if I were to restart or redo it again, I mean, we do it every day, every quarter, uh, we look at our data stack or data platform and we say, huh, why do we do it this way?

We should have a generative AI model writing that code to transform that data from one format to another, right? And we used to have these, uh, you know, thousands of lines of code and millions of pipelines before that. So we are in the constant journey of evolving our own platforms, which were traditional into modern.

And I think for me, it's a constant ideation. There is no, you know, kind of one magic bullet that came in, but we have to be on our toes, and that's how we operate. All right, folks, you heard you here.

Hey, this whole AI thing, and maybe it's not for the faint of heart, but there's a lot of joy in the adventure. Hey, Raj, thanks for being on the show. Thank you.

And thank you all for watching the latest episode of the Textron AI video series. You can find this episode and others on our website. We invite you to check them all out.

Until then, we'll see you next time.