LatentView Analytics CEO Rajan Sethuraman on How AI Is Reshaping IT Infrastructure Requirements

Hello, I'm Mike Vizard, and welcome to another edition of the Techstrong AI Leadership Series. Today we're with Raja and Rahman, who is the CEO for Latent View Analytics. And we're talking about the infrastructure costs that go with AI because, well, it turns out that it might be substantial and maybe highly underestimated.

Rajan, and welcome to the show, Mike. Thank you, uh, uh, thank you for having me on the show. It's a pleasure joining you today.

I Think we're all talking now about how AI is gonna be pervasive, and it probably will show up in every application we use, but it is compute intensive, right? And maybe more so than even traditional analytics applications, and that's gonna cost some money because what are we not thinking through properly when it comes to the cost of ai? And, and is this going to maybe be prohibitive in some way?

Yeah, this is a concern, uh, that's, uh, definitely on the top of minds of, uh, a lot of people. Uh, if you look at the evolution of, uh, analytics initiatives for the last few years, you'll see that a lot of the low hanging fruit has been taken, uh, initiatives tended to be run by, uh, owners of, uh, functional areas or business units, whereas now we are talking about much larger initiatives spanning the entire enterprise, which means that you need to pull in data from across the enterprise from all the different silos where the data might be reciting. And oftentimes, you're also talking about real time streaming data, right?

And making instantaneous decisions and deriving insights right from the top of it. That dramatically increases the volume of data that you need to handle. And with that come the cost of storage as well as computations.

So absolutely, at this point in time, uh, people are looking at how do we manage all of this stuff, uh, at a reasonable cost while making sure that, uh, they're getting the benefits from all the AI tooling and all the developments that is happening in this space. It seems there's also a fierce debate as to whether or not it is less expensive to do AI in the cloud or in an on-premise environment. You see a lot of people talking about the cost of tokens in the cloud is adds up too quickly, and then I wind up finding out that it's less expensive to run in an on-premises environment, but then again, I gotta go buy new infrastructure.

So what's your take on what's going on there? Yeah, and, uh, I see, uh, a little bit of back and forth happening on that. There was a point in time when everybody was, uh, looking at moving, uh, data completely from, uh, uh, on-prem to cloud, if not public cloud, at least a private cloud kind of a model, uh, because, uh, you could provision both the compute and storage right, uh, on an ask needed basis.

Uh, there's been a bit of a swing back on that in recent times because of the, the cost of tokens, right, that you referred to. I think, uh, it's interesting to see that the hyperscalers as well as companies like Snowflake and Databricks are coming with better models in terms of how the data can be stored and handled, uh, drastically reducing, right? Uh, the, the extent to which you might, uh, need to incur costs on storage as well as compute Databricks, for example, in their, uh, recent data and AI summit that happened a couple of, uh, weeks ago, uh, they talked about how they've integrated the capabilities through the recent acquisition nim, uh, into their platform that separates storage from compute, right?

And, and, uh, dramatically cuts down, uh, the time as well as the access time latency, as well as cost of doing all this stuff. So I see that, uh, there's a bit of evolution that is happening, and that'll now, uh, determine what directions companies take on this matter. Is tokens the wrong thing to be using to maybe set the pricing models for AI usage in the cloud?

Because it seems rather granular and it's, I have to get a token for input and output, and is there just another way of thinking about it? Yeah, I think, uh, tokens, uh, has, uh, come up as an initial model, uh, given the kind of, uh, computing requirements, uh, that, uh, lums, uh, have, uh, in terms of, uh, addressing, uh, analytics requirements as with, uh, any other, uh, technology, I'm expecting this space to evolve as well. Over a period of time, we have seen that, uh, other technologies, now I'm, I'm talking about, uh, even, uh, uh, uh, application, uh, programs like, uh, ERPs in the past, uh, when they would've started out, there would've been a certain kind of a pricing model.

And then over a period of time, uh, it evolves to adjust to what is the most appropriate, uh, kind of mechanism to use. Uh, I'm expecting that there'll be a lot more linkage with the use cases and the business impact and the metrics, uh, that one intends to drive as opposed to input parameters, right? Like tokens at this point in time.

Uh, but this is of course gonna be an evolution story, right? And we allowed to see how it plays out. There will also be a lot of nuances here in the sense that I need to bring the AI model to where the data is and where it's being created and consumed.

So in a lot of instances, I may train the AI model in the cloud, but I need the inference engine to go run in a local data center or some, maybe even out of the network edge. So is that gonna factor into our cost equations? Yeah, I, I, I believe so.

And, uh, partly this is not just, uh, driven by the cost aspect of it. Uh, there is a great deal of concern around, uh, uh, enterprise level security and governance mechanisms and access controls, uh, maintaining audit trail from a transparency standpoint. I think, uh, that will definitely dictate how, uh, data is stored and used, right?

How much of it was being done in the cloud, how much of it is done locally at the enterprise. Also, both enterprises, as well as service providers like us, are looking at how we can build better semantic layers between the data and then the, uh, and the LLMs, uh, uh, that might be, uh, eventually used for answering things. And these semantic layers can bring in a lot more intelligence right into the particular use case, and also cut down the extent of tokens that are, that needed to be passed, uh, and the compute required right at the LLM end itself.

So I'm, I'm guessing that all of this, uh, will evolve over a period of time. Uh, I'm, I'm seeing, uh, the emergence of high quality semantic layers, uh, on different domains and use cases, and these are being built by enterprises as well as service providers. And this is expected, uh, to continue in the years as well.

Of course, the list of potential projects involving AI in every company out there is probably longer than your arm. But will all these cost issues require CIOs and the executive leadership to kind of narrow down the number of projects that they're gonna actually fully fund because these things are so enterprise wide? Yeah, I'm, uh, expecting that some of the advantages of, uh, uh, gen AI and, uh, agent TK solutions to start, uh, kicking in a bit, uh, on this front in the last three years in particular, uh, because of the macroeconomic scenario and the uncertainty as well, we have seen that a lot of, uh, larger enterprises, fortune 500 companies have held back on big initiatives that will require a significant, significant amount of spend.

Uh, I talked earlier about how it is important to harness enterprise wide data, and typically these mean centrally driven projects, uh, where you go ahead and set up the platform and the infrastructure a little ahead of the co expecting the business cases to come later on. Uh, what we have seen in the last three years is a bit more of a tentative approach that, uh, let's first, uh, uh, get a good hold of the business case and then let's get sponsors for each business case. And that led leads to a bit of an incremental approach.

I'm expecting that, uh, the j and agent TKA evolution that is happening will help cut down the cost of other initiatives that organizations are running. We are at least, uh, already seeing, for example, uh, the impact of, uh, uh, these tooling and technologies in cutting down, uh, the effort and time required to do diagnostic descriptive analytics work, for example, and even in building, uh, data pipelines and predictive prescriptive models. So all of that should hopefully feed in, into the, uh, into the budgetary process with savings and productivity gains emerging from the other work.

Uh, the expectation is that, uh, we'll be able to accelerate a little bit more of the, the platform spend on the infrastructure spends Who's taking the lean on AI infrastructure these days. Because I think initially when I saw these projects, they were led by a data science team that had somebody in there who knew something about infrastructure, and that's how they built one application, and that was kind of as far as they got. But you could argue that if everything's gonna have some AI component to it, I need some centralized approach to managing AI infrastructure at scale.

So is the responsibility for the infrastructure and the inference engines, especially moving back to an IT department and the CIO is kind of stepping in there, or who's taking the lead? Yeah, I, I, I'm starting to see the trend in, uh, multiple organizations. Of course, it's also dependent on the, on the culture and the background and the context, uh, of the companies in terms of how they have approached not just data analytics and ai, but even IT infrastructure and applications in the past.

Uh, and, and some of the organizations are going through this cultural shift as well. But clearly there is a secure trend in terms of the emergence of the chief AI officer or the chief data analytics officer, and they are starting to bring a little bit more, uh, gravitas into the, into the decision making process, right? Uh, like I said earlier, uh, that certain things need to be done just as cost of doing business.

You can, you cannot just wait to do it on an incremental case by case approach, but we need to take a call that this AI infrastructure is absolutely necessary to compete in the emerging scenario, and therefore there is that appetite, uh, and the budget, right, to go out and spend on creating and building that infrastructure. To your point earlier, there are some advances being made, especially in the cloud, and Databricks being an example thereof. Um, it seems very difficult to make an assessment about where the cost vectors are really gonna be in six months.

'cause the pace of innovation seems to be pretty rapid. So what's your best advice to folks who are trying to figure out, you know, an ROI around an AI project? Yeah, I mean, my advice at this time would be that, uh, don't worry too much about, uh, uh, trying to cost this out over a three year or a five year horizon, because there are definitely several aspects of, uh, uh, of the, of the platform, of the infrastructure and the solution that are going to continuously get less expensive as, as we go quarter on quarter.

Even. Uh, in fact, at the Databricks, uh, summit that I attended, uh, Databricks talked about how several things will now be available native within their, uh, platform like Gemini, for example, right? To do AI a work, uh, or integration with Azure at the backend, right?

Uh, Databricks is also pursuing, uh, a fairly defined strategy where they're saying that you don't need to bring all the data into the Databricks platform. You can leave it where it is in the legacy system, and you can just pull it in right as and when request to do the compute and the modeling. So these models are evolving quite fast in some sense.

I will feel, uh, I feel that we are currently at a point where, uh, all of the innovation that is being done, not only by the, the large companies, but also the startups, uh, they're all starting to converge. And that is gonna help, uh, take care of the cost and the, uh, and the cost benefit aspects of it. So therefore, the advice would be that focus more on the use cases that are going to give the biggest bang for the buck, evaluate it more from, uh, the end output, right?

What are we targeting, right? Uh, is it a top line metric? Is it a bottom line metric?

Are there other business process metrics? And what kind of lift can we see or needle movement can we see in that? And then prioritize on the basis of that cost?

I would expect that given the competitive scenario, uh, they will all trend in a direction where no, it's gonna be beneficial. Are we kind of bouncing between two extremes here? And let me describe those two extremes.

But, you know, when I first started 2, 3, 4 decades ago now, the wisdom was bring the compute to the data, and then we shifted it to the cloud, and we wound up moving a lot of data to the compute. Are we kind of trying to find some middle ground between those two extremes? Now, I would say that, uh, it depends on the, on the use case.

I mean, uh, if you're really talking about, uh, uh, realtime, uh, analytics on, uh, streaming data and, and many companies are starting to explore what they can do on that front, uh, there, the technology, the current technology and the tooling that's available might mean that no, you need to bring, uh, uh, the compute to where the data is. Uh, if, if dealing with very large volumes of data, uh, the tooling and technology is evolving there as well. Uh, but overall, I feel that, uh, uh, there is a lot more happening today in terms of bringing the data to the compute, uh, because it's also now possible for, uh, identifying just what the changes are to the data, right?

In fact, uh, uh, the separation between storage and compute, uh, a lot of the, the technology that goes into that is really about, uh, how do we identify not just that data element, but also the incremental change that is happening and just capturing that essence so that we don't need to move the entire copy right, of the dataset in order to do the compute, but they are just able to understand the changes that are being made and then just reflect that, uh, into the compute environment. So I'm expecting that, uh, more and more in the future, data will move to the compute rather than the other way around. But today, depending on the current stage of evolution of the technology and the use case, uh, there will be a balance that plays out.

What do you see among your customers who are getting this right? What are they doing that others are not, that you kinda wish everybody else would kind of be more cognizant of? Yeah, I mean, the one thing that, uh, we keep reinforcing, uh, with, with all the clients, uh, the, and prospects that we are having conversations with is, uh, the need to do a lot more experimentation at this time.

Uh, there is a great deal of optionality, right? That's, uh, evolving. Uh, our vision as a company is to help, uh, company, you know, businesses harness a power of data and analytics to thrive and succeed in a, in a digital world, right?

And that can be possible only if there is a really good understanding of how the ecosystem is evolving and how to cut through the optionality. Just like how there was a systems integrator role in the past over the last 20, 30 years, uh, I believe that there is a need for a AI integrator role, given that, uh, there is optionality of all layers of, uh, data analytics and decision making now, right? From the data layer, uh, to reasoning models and beyond.

Uh, so at, at this point in time, uh, the, the, the friends that I see, uh, in companies that are, uh, really leapfrogging and taking the lead on this is the willingness to do a lot more of that, uh, experimentation. And that is what is helping them come up and do pilots, POCs at speed, take them from pilot grade to production scale, right? And then start implementing within their environment.

That, and then the other thing I would say is that, uh, the environment, uh, that good that companies create, uh, is very important. Uh, some of the organizations that we work with, uh, they have very clearly defined data analytics at the core of their decision making strategy. One of the, one of the large accounts that we work with, for example, uh, they have this, uh, very, very clear, uh, data driven operating model that they use that, uh, every meeting and every decision that they make has to be supported by data analytics, right?

That is coming from their data, from their enterprise data and, and their data platforms. So driving that culture of change in terms of saying that, let's look at every decision that we make from a data perspective, uh, plus that, uh, uh, culture of experimenting, right, uh, at large scale. I think those are the differentiators that I see.

Last question. Are you seeing organizations kinda look beyond GPUs or are they looking at others classes of processors to run some of these AI workloads? Or is it pretty much, you know, GPUs or the answer?

What was the question? Yeah, I would say that, uh, uh, at this time, uh, there is the, uh, the, the great amount of, uh, excitement, uh, about, uh, GPUs. Uh, I talked about semantic layers emerging, right?

Uh, in response to the earlier, earlier question, I think, I think those semantic layers and some of the intelligence that get built into the tooling will reduce the need for a, uh, for a battery of GPUs. I mean, there will be use cases, uh, which do not require that kind of, uh, computing power. Uh, it also depends on, uh, what kind of decision is being made, right?

And, and whether the data is being processed, realtime, batch, and so on. Uh, that combination with the semantic layer could determine the architecture, right, that, uh, that companies use and, and preference for the compute environment, uh, overall, I think even, uh, large provider, you know, large players like Databricks or Snowflake, and the, and the hyperscalers will start providing a spectrum of compute environments, uh, so that again, they can manage the cost depending on the use case that is being executed. Instead of saying that, now everything needs to go to A GPU.

All right, folks, Sharon in here, I would argue in the last two decades or so, we kind of took infrastructure for granted. I think in the age of ai, if you don't get the infrastructure question right first, everything else is gonna go bad. Secondly, Hey, Rajan, thanks for being on the show.

Thank you, Mike. Thanks for having me on the program. Nice chatting with you.

And thank you all for watching the latest episode of the Techstrong AI video series. You can find this episode and others on our website. We invite you to check them all out.

Until then, we'll see you next time.