AI in IT: How Roles Are Evolving with DataBank’s Vlad Friedman and Joe Minarik

Hello, welcome to the latest edition of a techron AI series. I'm your host, Mike bn. Today we're with Joe Minick, who is CEO of DataBank and Vlad Freeman, who's CTO of DataBank.

And we're talking about the rise of AI infrastructure. There's lots of GPUs now, we're running inference engines and, and the way we manage it is fundamentally changing along the way. Gentlemen, welcome the show.

Thank you. Glad to be here. Yeah, thank you.

Joe, I don't know if a lot of people know who Data Bank is. Exactly. So maybe you might wanna start with the, you know, why this conversation matters to you guys, and, um, how much are we seeing these inference engines these days in production and Mars?

So, Uh, data Bank is a data center provider. So when you think about all of your infrastructure, interconnected infrastructure that is happening, anything from the applications on your phones to your interconnected devices, uh, to your computers, laptops, the web, all of that stuff runs through a data center. All the backend infrastructure that is supporting that, that sits on those servers, the data centers provide that infrastructure and connectivity to support that infrastructure.

And so Data Bank is a US-based, uh, data center company. We have the largest geographic footprint in the US of any of our, uh, any of other pride providers. We're in about, uh, we have about 70 locations spread across about 27 geographies, um, in the us.

And so we see a lot of, you know, the adoption and change and, uh, technology that goes through into the data centers and what's kind of coming about. So we see a lot of ai, a lot of movement in AI from anywhere from kind of the, the AI companies themselves, uh, through hyperscalers as well as down into, uh, the enterprise level and where their adoptions and pieces are sitting. And so we do see an influx of, of that happening where, you know, they are launching their AI aspects are, which is higher level of compute, which requires more cooling and power in our data centers.

But it, uh, we can see that kind of deployment happening and the adoption kind of happening through there, which is exciting for us to be a part of and continue to help that grow. Well, initially, um, it seemed like data science teams were doing everything here and they were training the models and then deploying these things. But I feel like that shifting their responsibility for the inference engine and the processors that it runs on, is that running more over towards a, uh, traditional IT team now that has to kind of manage this stuff at scale and or the roles and responsibilities kinda shifting?

You know, I absolutely, but I think we're seeing old patterns play out kind of today where, you know, as the cloud came about, folks would leverage the cloud or initially to run a lot of their experiments, right? Figure out what's the right technology, what's the right engine, what's the right software to use. Um, as those started to mature, that's where we've actually started to see enterprise take these AI workloads back into the corporate data center and, you know, have their teams really manage that because it's, it's an affordable alternative to actually operating in the cloud.

Joe, a lot of people have a kind of a natural default to a hyperscaler for this kinds of things. Why, you know, does, why run the data bank for this kind of thing? What is different about what you guys provide than I might see from, you know, one of the big three that everybody seems to know.

So we Don't really provide, uh, competition to the hyperscalers. The hyperscalers will actually utilize, uh, our data centers in the backend and enterprises will use our data centers in the backend. What I think you're seeing is that, as Vlad had kind of mentioned, you're seeing adoption of the AI platforms being internal to the enterprises.

Um, as with the cloud, there's use cases that make it feasible to outsource. And then there's use cases where you wanna have it insource to manage. And predominantly from what we see that comes from the information that is used to train elms as being proprietary information for those companies.

So they'll bring it in-house and then need that data center capacity, uh, typically through us or, or others. But, uh, we like that they come to us and manage through, uh, building that out and that infrastructure out so they can support their own internal information and code and what they're trying to util utilize that AI to solve for. And then it's not exposed or they don't feel it's exposed, uh, to outside sources.

Well, and we hear a lot of people are trying to use these GPUs, but there's also alternative processors out there. And now I think there's more flavors of GPUs out there. Am I able to use different classes of processors for different types of AI workloads?

And what the, what are the drivers for that decision? What's reasonable and what made me a little too much to expect? Uh, sure.

I, I think it's, it's early, right? You know, with Amazon, for example, coming out with their own chips. Uh, and really that's about driving efficiency.

I think really at a software level, the, you know, the place where enterprises really care about, I think really it's just about controlling costs. I don't really see too much of a functional difference, whether it's a GPU platform and our, or a different type of ARM platform that they're leveraging to run the, the ai. 'cause AI is just software, right?

And you're providing a compute behind the scenes. So I think it's a matter of folks are looking for ways to practically apply ai. As you start to do that at scale, the costs start to grow and the processing options are really about driving efficiency into the process versus driving a different result.

Joe, we hear a lot about GPU scarcity these days. I mean, can you get an access to enough of these GPUs? Are they still hard to come by?

Or is that situation getting better? Yeah, my understanding is that the, the market for GPUs is still still constrained. Um, you know, they're, they're releasing the H one hundreds and people are getting those, the, the next rev and version of H two hundreds, they've got a backlog that NVIDIA has.

And so there's uh, definitely an an, you know, demand and adoption that people are wanting these. So I think there's still constraints in the market for that today. But are the models themselves getting better?

And I'm asking this question from the context of are they more efficient? Are they consuming the infrastructure better than they were initially? 'cause early on my sense was, you know, data science folks, they didn't know whether this thing was gonna make it into production or not.

So it might not have been their highest priority. But now as you IT ops teams look at this thing, they probably have a whole other set of metrics. And are the developers getting more efficient?

I can tell you absolutely, yes. You know, we, we started early experiments, you know, with large language models some time ago when it was new. And you know, a lot of our development teams, uh, attempted to leverage them.

And they, I can't really tell you that they were truly satisfied with the results. You know, with more, with modern, uh, LLMs, I can tell you my teams are using them on a daily basis to help us write more efficient code to help us write, you know, better test cases. But you still have to be mindful, right?

The AI doesn't understand intent. And the best analogy I've heard is it's like having a million interns that perfectly understand syntax but not what you want. Mm-hmm.

So it still takes, you know, good developers to truly understand the result, evaluate the result prior to applying it in production. But I have seen cases where folks can get 300%, 500% increases in productivity just by using AI to compliment their skillsets. Now with that consumption, um, I would tell you, I, I believe the AI chip are being more efficiently used because I think in the beginning, folks were trying to figure out how to use them effectively now that they're being practically applied more often.

You know, I'm certain that consumption's going up. Joe can speak to, you know, our consumption within our data centers around AI workloads are certainly growing. But I, you know, I don't believe the technology is sitting idle.

It is being consumed. It is driving meaningful value into the equation to accelerate businesses and drive productivity. Well thanks bud.

'cause that was where we were going next, Joe. Um, what is the energy footprint look like? 'cause there's a lot of folks talking about how we simply will not have enough electricity to fund all these workloads in the current form.

They are. And they might get bigger if we do things like, um, you know, agent AI or the next generation of AI where AI is smarter than of, of us all. But, um, how do we keep all this, uh, energy consumption reasonable?

Yeah, that is, uh, one of the largest challenges we have today. You know, when you, when you look at the kind of in the US and the energy footprint, you have, uh, about 4% of the total US energy being consumed by data centers. And that's expected to double.

And some even are estimating triple I'm hearing now in, in, in the marketplace, you know, but that goes to eight to 12%. Consumers take over 30% and are projected to go well over 50% of the consumption. And when you look at our technology today and just general consumers, you know, everybody has an electric car or they're getting pushed to electric cars.

Everybody has a phone, everybody's got electric devices, the consumption is large. And then data centers coming in with these AI loads are putting spot loads on areas that are already constrained. If you recall going to California, everybody come home, turn on the air conditioners and you'd have brownouts, you know, they're experiencing these problems in Atlanta and Phoenix, you know, other places all around the country have had these problems.

And now you add these large power consumption aspects for the data centers, which is still a small percentage of overall, but it's a point load in a populous zone that is already experiencing issues. And to build out that distribution of that power, it takes years, 10 years, 20 years in some cases. Um, and you wanna drive energy efficiency through all of this as well.

So, you know, we're shutting down coal plants and trying to be environmentally conscious. Um, but we don't have replacements of that power. Green energy is great, but the wind doesn't blow all the time and the sun doesn't shine all the time.

So getting some of that renewable stuff is also problematic. So we're seeing a lot of advent into the nuclear space in what they call SMRs, um, which are small modular reactors that can kind of fit on a site and give that dedicated power to alleviate those aspects from the grids. But those are still years out.

And so it is definitely a challenge that we have to manage in order to continue to have this technology adoption in front of us. Bon are people coming to you and asking for a little more help to secure these AI models? I think it's becoming a bigger conversation out there, and they're worried about also the safety of these things.

But, so there's a lot that goes into these inference engines. 'cause there's these things called cyber criminal syndicates that would just love to steal an AI model here and there wherever they can. So how do we kinda protect these things that we put a lot of time and energy into building?

Uh, you know, I think it's a great question. You know, you can't really speak to the, you know, backend proprietary technology. But one of the things we talk about a lot, especially, you know, my group and our information security group under our CISO is, you know, how do we keep our data safe?

How do we understand the policies of the company that's providing the AI service? How do we make sure that there isn't a co-mingling of data so we don't put something into our ai and all of a sudden someone asks a question and out pops our confidential information. I'm using our in a broad more, you know, broad sense for, for organizations.

So we actually go through and invest a considerable amount of time, every time that we're letting up a new a AI product to understand what will they be doing with our data, how do we protect our data, how do we make sure that we validate our data isn't being co-mingled with others and used to actually train the model in a way they can get out there and potentially reveal confidential information. And I think those are the questions everybody should be asking. It's really another reason why you see, um, a number of these enterprises driving to a co-location footprint.

Because at that point they can really control what happens. They can control the flow of data, they can control how they train it, and they have assurances that that, uh, trained model with proprietary information isn't getting out there into the marketplace. Joe, who's in charge, and I'm asking this question because there are CIOs, CTOs now there's chief AI officers, there's CEOs involved and throw in a few data engineers and it takes a frigging village to do anything with ai.

So who's making the call and the decision? Oh, that's a, that's a little bit of a loaded question, right? So everybody's got their expertise and, and their area, and I think, you know, those are weighed in on, uh, how we approach it.

Um, typically you're gonna usually go up to the CEO of a company, right? And they're gonna make a call on what is what's being used. That's, that's ultimately their responsibility.

And everyone underneath is gonna have pieces that are going to, they're gonna manage and they're gonna talk about what the benefits are, what the ROI is, what those, uh, what the costs are gonna be and what's something they can con, you know, conform to. So it has to kind of go through what all of those applications are. And so that's why it does take that team to understand what the varying vectors are for the business, to manage it the right way, and then they put that at a consolidated view to make the right business decision.

Juan, is there something you wish that customers knew before they showed up at your door? And yeah, do you feel like there's conversations you're having over and over again that maybe should just be table stakes at this point? I mean, um, I'm sure every customer is different, but, um, what is that kinda one thing that you wish everybody kind of already had under their belt Y You know, it's, I think that AI isn't a magic bullet.

It's a tool. And just because you show up and you know, you, you with a, a large number of GPUs doesn't mean that tool is effective. You still need to figure out how to practically apply that tool to solve a real world business problem and the, and take data, turn it into information that's really actionable.

So I think folks are looking for, you know, here's a platform, it's ai, it's magic. I'm just gonna throw at some kind of problem and it's gonna spit out a solution. I think the talented engineers along the way from all of these enterprises are a key component of actually make it to make the AI do something useful for the enterprise.

Joe, if I look back, 2023 was kind of the year of irrational AI exuberance, and 2024 was the year of let's do a million experiments. Um, have we gotten to a point now where we kind of need to just pick two or three things that we can afford to do and kind of put all the wood behind a couple of projects? I don't think we're quite there yet.

It takes a while For these models to really be fine tuned, really be able to understand the data and to really be able to give you results, right? So we continually are running the inference aspects where it's collecting all of this data and those large language models are learning and how to manage through that. And as we give it more and more data, it's gonna fine tune what we can do.

Now there's, you know, there's aspects, what they call hallucination where it gives you the wrong answer because it's got too much data in what it's trying to compile. So I think it's gonna take us a, a couple more years to really fine tune some of this stuff and really start seeing where some of these advancements can impact us the most. So it's a little too early to tell, uh, in the environment of like what big one or two are gonna be the most successful ones.

And then, but I'm gonna give you the last question, but we hear a lot about, uh, the models themselves are getting bigger, but we also hear that there are small language models being built for different use cases. As you kinda look out through the coming year, how much are we gonna see, you know, what we're calling LLMs and how much are we gonna see something that we might call, uh, a small language model? And I'll make something up here, maybe there's something in the middle here, and they're all t-shirt sizes, small, medium and large.

You know, I think what you're gonna see is a hybrid. It's a little bit different than what you asked. It's not gonna be around, do I wanna, you know, a small parameter model, a medium parameter model, or a large parameter model.

I think what's really changing in the structure of how these LLMs are being built is the fact that I think they will all be large language models, but they'll be compartmentalized. Because as we talk about the power usage from ai, what winds up happening is the power usage is really from a chunking through vast amounts of data and a massive database and you know, with billions of parameters. And I think the fundamental change you're seeing that's coming around the corner now is these AI companies is have figured out how to create a directional model.

So if I'm asking a question about puppies, I have one database, and if I'm asking a question about SQL Server or Snowflake, I have another database. So what it really does is it narrows the number of parameters quickly to drastically minimize the size of the data set that it's working with to make itself more efficient. But all of those models are combined into one large LLM.

So you don't have to say, here's my LLM for this, or here's my LLM for that. It's about dynamically reducing the base of data, which makes it more efficient. And that's what I mean by hybrid.

It's almost the aggregation of different training data sets together with a directional engine that points you in the right place before going to answer your question. Okay. Folks shared in here, not all AI models are created the same, but they have a lot more in common than you might appreciate.

Gentlemen, thanks for being on the show. Thank you. Thanks.

Thanks so much. All right. Thank you for all watching the latest episode of the Techstrong AI video series.

You can find this episode and others on our website. Please check them out. Until then, we'll see you next time.