Data Challenges in Operationalizing GenAI with VAST Data’s Neeloy Bhattacharyya

This is Textron tv. Hey guys, thanks for the thrill. We're here with Nelo Charya, who is Director of Solutions Engineering for Vast Data.

And we're talking about while just what is going on with AI and data and the challenges of processing and managing it all, Nelo, welcome to the show. Thank you. And I guess the answer is what isn't going on, right?

I think that the good news, I think is, is that the, the market as a whole has, has started to recognize that data plays an important role in, in ai, right? I think if we were having this conversation six months ago, that might not have been as obvious, certainly wouldn't have been as obvious a year ago, but I, I definitely feel like I'm having more and more conversations where we're having to do less to explain that data is important and more getting into, okay, how and why and what do you do with that, et cetera. I feel like I'm having deja vu, and I'm old enough to say that because back in the day before the cloud, it used to be all about bringing the compute to the data.

And now here it is all these years later, and we're talking about that again because, well, there's too much data to move and it's problematic. So we need to figure out how to bring the right processing power to where the data is, and is that kind of changing the way we think about data management? And it, And it's such a double-edged sword, right?

Because let's face it, business and commerce would not be executing at the rate that it is today, were it not for the SaaS providers and the clouds, you know, the hyperscalers and, and all of those pieces, right? All of those players had a definite impact on accelerating the rate of business. But unfortunately, the, the, the drawback, if you would a little bit, is that it means that your data is now scattered everywhere.

And so, you know, you've, you've got all these lines of business teams that have made these decisions to leverage certain SaaS companies or leverage hyperscalers in different ways. And so now when you say, I wanna look at all of the data across my business, it's actually a lot more challenging than it would've been 10, 15, 20 years ago, where you would've been able to go into a data center and see much of your data in that location. Yeah.

In some ways I feel like the cow is half over the fence, half my data is in the cloud and the other half's in an on-premise environment, and I need to, uh, stitch these things together in a way that's federated. Is that where we're headed? Yeah.

I think streaming plays a, a big role in it, right? So I think you're, you're, you've definitely gotten to a point where you're not going to reverse the decisions that were made in the past, right? Whether it's a SaaS decision or a hyperscaler decision.

Decisions were made in the past for specific business purposes. Those business purposes have not gone away, right? They still exist and they still have the characteristics associated with their data needs that, you know, made the original solution fit for purpose.

So it is, it's, it's a, it's a degree of, of streaming data between locations. It is a degree of making sure that your, your governance and your, your legal framework is, is in a manner that it can be extended across different locations. And that's especially important to ai when you're using third party models, uh, depending on the countries and the areas of the world that you operate in, you are going to, as the consumer of AI, gonna have regulatory requirements put upon you, right?

So it's very important that you reflect those regulatory requirements and the vendors that you work with and you choose to work and choose to leverage their, their AI capabilities around. But yeah, it's a, it's a lot of streaming, it's a lot of, uh, you know, selectively transforming data and making sure you're moving the right data, and it's not always your hot data, right? Many times, especially for generative AI applications, the data you have need to access may have been put into a lower tier of storage because you didn't think you needed to access it, right?

Think about like raw documents that you maybe grabbed some key pieces of metadata for what that in your data lake, but the raw document itself has been put away in some glacier tier, right? Think about in, in many of the, you know, contact center type use cases and applications where you're, you know, you may have transcribed voice, but you didn't keep the original recording in the same hot tier, right? You thought all you needed was the transcript from it, except now with ai, you want to essentially create voice-based, you know, generative applications.

You wanna understand sentiment, which we all know from text messaging, you know, the transcript is the worst way to know sentiment, right? You need to listen to, to people speak and their intonation and things like that. So yeah, we're actually seeing people, you know, that that necess maybe parking, lotted some data that they didn't think they were gonna need, and they're having to now rehydrate those to be able to, to, you know, meet their generative AI needs.

And there's a lot of different nuance between the data that I'm using to train a language model, and then the data that I need to process on the inference side when I deploy the model. Um, and in some ways, I think we're starting to see a shift, right? Because the data science team seems to be managing the training part, but more and more it looks like the IT ops folks and the DevOps folks are running more of the inference side of that equation.

This is, this is the, the thing that renin talks about most frequently, right? Is it's not just ai and in a traditional, you know, ML sense or in a BI sense where you're looking at historical data or you're making, you know, future predictions and using it to plan your business, we're seeing AI in more and more real time use cases, right? So this is where ai, you know, whether it's agentic workflows or whether it's RAG or any type, any of these things, these are real time use cases where the, the, the actual applications that people interact with every day are now AI enabled.

So yeah, not only does that require a, a different data set, a more current data set, but the non-deterministic nature of AI means that you need to log every interaction that goes on. So now you need to, to keep all of that content in place, you know, track it, and, and then it, it's very valuable because it actually provides you fine tuning data, right? That you can use to improve your models or evaluate new models.

So it's, it's very valuable data. Um, but that's a, that's a whole other dimension as well, right? So you have the, the source side of it, right?

The, the rag and the, the other data sources, you know, structured data that a, that an agent may use AI enabled agent, but then you also have the, the logging and the, the keeping track of all of those interactions, so you can use those to improve your models. Uh, and so yeah, all of those pieces are very different. Um, as far as the players, um, they span the gambit, right?

So you have, you do have your data scientists involved, you've got your application teams involved, uh, you've got your, um, vendors in many cases that, that help make up your, your customer facing and employee facing applications. They've got their own capabilities, so definitely requires a bunch of different actors working together. Speaking of that, I feel like it's taking a village to build anything in the AI space.

I've got developers, DevOps teams, people running MLOps platforms, data scientists, data engineers, and let's not forget those security folks. How do you see all this coming together in a way that, um, is manageable? Because I wonder if all these silos that we have today maybe not lend themselves as well to this whole motion.

Well, it, it broadens, you know, towards the, the end of the, you know, the big era and focus in DevOps. You saw this uptick in platform engineering and, and platform engineering actually plays a very important role when it comes to organizing this stuff in ai, right? Because you don't want a bunch of bespoke patterns being created by each line of business, right?

You want all of those, uh, those actors that you mentioned, you want all of them contributing their templates and their best practices into a structured environment that all the lineup businesses can consume from, right? So it's actually a really cool, you know, co uh, convergence of, of all of the work on the platform engineering side that was getting a lot of attention just a couple of years back with all the AI pieces that are getting attention now. And, and we see them, uh, we see them converging.

Two, we have the infrastructure needed to drive all this. It seems to me a lot of the, uh, infrastructure we had previously, the earlier points was more around batching applications and, uh, it's a different architecture. So what do we need for infrastructure?

It's a, uh, the, the infrastructure equation is a, is a large heterogeneous pool of capacity, right? Um, yes, we certainly have some infrastructure that is deployed that, that can handle, you know, sort of the, the very tip of the spear of these workloads. Um, we've got some infrastructure that just absolutely cannot do anything at this speed and, and just needs to be, you know, refreshed or people need to move into different environments.

And then you've got a, a pretty good middle ground, right? Not every piece of AI needs to run on the latest, uh, in, in fastest GPUs as much as my, my Silicon partners by disagree with that statement. Um, so there's a, a continuum of, of capabilities, and specifically from a vast standpoint that has been a big focus for us over the last year and will continue on for the coming years, is sort of addressing the data gravity problem, right?

Making it so that wherever your compute resources lie, because let's face it, the concentration of networking and power and cooling needed is not gonna be in one single space. That it's just impossible. So wherever those appropriately sized compute resources may lie, we wanna make it sure so that you can get the actual data that you need into those environments, right?

Not have to replicate everything all around the world, not have to stream everything all around the world, but actually be able to access the data that that particular set of compute capability needs to access. That's been one of the big focus areas for us across all different access modes, right? This isn't just about object and file either.

This is about structured tables, this is about vector databases, this is about all of the different modalities of interaction that, that AI needs with, with data. So what's your best advice to folks as they kind of contemplate all this? 'cause on a certain level it can be overwhelming.

Think big, think big AI does best when you tackle the biggest problems, right? I think the number one challenge we see with some of these AI projects is people try to develop a very incremental minor, relatively minor MVP, and so therefore they, they end up, uh, you know, not getting the ROI the effort to do that incremental step with AI doesn't warrant the effort, the the cost associated with it. Whereas if you think about the biggest problems in your business, and then you start backing into, okay, if I've got, you know, 15, 20, 30 really big problems that I wanna solve, how would I solve them using ai?

How would I have all of these different solutions take a similar approach? Then you start getting into, okay, I need a platform engineering function. I need a COE, I need these people to create internal service offerings that could be used by multiple of these really big problems that I'm gonna solve.

So if you start with the biggest problems instead of the smaller problems, you end up cre, you're, yes, you're gonna make big investments, but you end up getting the ROI off of those investments, and you're creating services that those biggest problems are gonna need, not your small incremental gains. You guys have a conference coming up and a lot of your partners are gonna be there discussing these very issues. Kind of plug us into what's going on there.

Yeah, so, so I think you guys have seen, uh, us us talk about the cosmos and, and the, the notion of, of pulling together a bunch of companies who can help model trainers, model builders, you know, enterprises doing inference. We, it, it takes a village, right? It takes a lot more than vast.

And, and that's what, that's what we're looking to do with Cosmos. We've also got a significant number of innovations that we are gonna be announcing. Uh, so yes, please do join us, uh, hear from our partners, hear from our own teams about some of the things that we're doing, and, uh, we're also gonna be doing some, some activities in, in the field afterwards.

So we encourage everybody to participate. All right. And the date for that is again, So we're going live October 1st for the Americas, and then obviously we're gonna follow that around for the rest of the world.

So it might be a second for some of you out there, but it's gonna be exciting. All right, folks, you heard it here. There's a world of difference between a proof of concept and something that's actually gonna run in a production environment.

Do not learn this the hard way. Hey, Nely, thanks for the, thank You. All right, I'm back to you guys in the studio.