AI Events and Current News – Techstrong AI Podcast EP8

Hello and welcome to this week's Techstrong AI podcast. I'm Amanda Ani and I'm here today with Mike Ard. How are you doing today?

I'm great. I'm in Paris. What's to complain about?

This is, you know, been an awesome trip, but we'll get into that in a second. I know. Sounds wonderful.

I can't wait. So we do have quite a lot to share with you today about a few conferences going on this week, Mike being at one of them. And then we'll dive into some other interesting news.

So Mike, can you start us off with the NVIDIA conference? All right, well we're gonna start with halfway around the world from where I am today. But, um, there was a big AI conference hosted by NVIDIA in San Jose, California, and they rolled out their latest and greatest Blackwell processor for training AI models.

This puppy can train models the size of a trillion parameters or more. That's really, really super big. So AI is probably gonna get a lot smarter.

It's supposed to be available starting at the end of this year. I don't think the average organization is gonna be building trillion size parameters for these models. I think that's in the realm of a chat GPT kind of thing.

But Wall Street seems to really like that. But it was interesting because um, the CEO of NVIDIA was, was addressing questions about, well is uh, prompt engineering a skill or a job or what is it? And well turns out from his point of view, it's just a skill we should all pretty much have.

I don't think it's gonna be a job either. Seems to be a lot of HR folks that didn't get quite get that message yet. 'cause we see all these job roles for prompt engineers out there.

But, um, I guess some people may be more skillful at a than others, but I'm not entirely clear that that would classify or qualify as a, shall we say, engineering job. But I don't. Amanda, what do you think?

Yeah, I think, um, even from my own experience, I feel like, um, everybody is taking on this job of prompt engineering from personal and business standpoints. And really you kind of have to know how to prompt on your own across the board when you're dealing with the ai. I, I don't really see what one individual person is gonna stand alone and do because each person needs a set of tasks or you know, certain things to be spit out by the ai.

I don't see how one person is is gonna handle all that. Yeah. And then the unspoken elephant in the room that wasn't really addressed was GP news continue to be scarce and it's hard to come by and the average organization, uh, frankly is looking to figure out how to run inference engines on anything but A GPU because, well they don't wanna wait 12 months for the GPU to get there.

So, um, there's a lot of interest in using various, um, intel architectures for running these inference engines. A lot of work is being done on that side. Some folks are trying to figure out maybe how to do training, but that may be a lot harder to do without a GPU per se.

But, um, the size of those models that are being deployed on the inference engines are getting smaller measured in terabytes 'cause they're being trained with more specific sets of data. Um, so there's a lot more interest in that whole, uh, deployment aspect of things. And of course everybody's keeping an eye on a MD who's out there with A GPU and people are wondering, well, is that gonna be the thing that breaks this scarcity?

But I don't know how many GPUs A MD can make available either, but at least there'll be another option in this play. But it's amazing to me that, you know, one of the greatest advances in our time is being limited by the availability of hardware. Yeah.

And I don't know what the solution is other than to have more companies, uh, open up and, and maybe this is just an area that's wide open for companies able to come in and and start producing them. Yeah, I wish it was that easy, but it seems to me by the time you start making one of these things and design it and get it through a fab, we are measuring this in years. So, um, I don't think this problem is going away anytime soon, but we will see it has been interesting.

There's been, um, you know, folks in the NVIDIA world talking about virtualization and these drivers that they make available and an outfit that we talked about on the site with clear ML has, uh, made available some tools to help you virtualize A GPU, so at least you can run more workloads and increase the utilization rates, which of course is also a problem and has implications for sustainability and all that good stuff. But, um, I'm not, you know, if you ask me we'll be talking about this issue a year from now. Yeah, it's a fine balance.

I think, uh, you know, as this technology is advancing so rapidly and being implemented, uh, with every solution comes a a back step and a and a problem to address. There you go. That problem was also evident here at CubeCon plus Cloud Native Con Europe.

Where I'm at for this in Paris, it's um, a number one topic was, uh, how are we running or optimizing AI workloads on Kubernetes clusters? 'cause the primary mechanism for running most of those, um, workloads is on this emerging cloud native platform known as Kubernetes that is coming up on its 10th birthday in June. And the problem is, is it's not easy.

The folks who are running these AI workloads in Kubernetes environments or the likes of, you know, big companies like Open ai, they have lots of engineers to throw at this issue. And the conversation here was all about how do we make this simpler and how do we create APIs and templates that make it easy for data science teams to run this stuff? Or are they always gonna need somebody who sits between them and that infrastructure to manage that.

Folks here we're talking about the rise of, uh, new titles called the AI engineer, which is very similar to a DevOps engineer in terms of what their job function is. And the need for greater empathy was thrown out a lot another term that you hear in the DevOps world. So it feels like we are slowly but surely moving our way towards, um, creating a set of DevOps like principles for deploying AI models.

And I think they're just gonna rip and replace the same terminology and hopefully we will get this right. But it still felt like a lot of work needed to be done. It's still early days.

So again, I would say we're gonna be talking about this issue a year from now. Yeah, it's interesting because it feels to me like it's all part of sort of the same process. So I'm surprised, um, that they're just now realizing that.

Yeah, I, I think, um, it didn't occur to them that this stuff was gonna go so mainstream that, you know, there would be average enterprises trying to figure out how to make these things work better and optimize that. I think initially they had it in their head, there might be a, a small number of models that will be run by very large companies and this will be a problem for a small group of people. But, um, you know, success sometimes bites you.

And here we are having the same conversation again, so I'm we'll say one more time, Well, We'll be keeping this at going for another year. Yeah. But there is a possible solution that you recently, uh, shared about a, an AI engineer.

Well, there's some, um, progress being made on that front. There's a, I guess an early stage project called Devon. com because rather than just kinda probabilistically guessing what the next, um, piece of code is that I should write, um, which you see in a lot of these copilots, you can actually assign this digital agent a task.

And the task could be everything from go create a website to, um, go optimize or rewrite this entire, uh, code base to, for a particular language in a particular use case. And so it kind of showed just how far we can go with this technology. And I suspect that Google is working on something similar and Microsoft and Amazon will all do something similar in the future.

But, um, the interesting thing about this is, you know, writing code is only the beginning the way that this thing can be used to, um, orchestrate a set of tasks because it wasn't just telling it, you know, to go do one task, but you could tell it to do some series of tasks and it would execute that. I'm sure a lot of DevOps engineers are going, you know, well who's about to move my cheese and what does this mean for my job? But you still need to know how, what task to tell it to do.

So I don't think Devon will figure that out on its own, but a lot of the kinda low level scut work will increasingly be automated and maybe we'll have a lot more DevOps engineers who can orchestrate things and we can develop software a lot faster than ever. But I'm not quite clear, I'm ready to say that the DevOps engineers will be entirely replaced by ai, but it sure is a lot smarter than we thought. I think it's interesting how it can be told to go, um, to go repair bugs and different issues.

And if we could take that side of it moving forward and it can just constantly be scanning and repairing and fixing all the issues, that would be great. Yeah. 'cause maybe everybody can be a software engineer then, but you know that.

And then we'll be building more software than ever and we'll have all kinds of cool new things that previously we would not have attempted because it would've been just too hard. But, um, I cannot tell you where Devin is right now in terms of it's a private beta. You have to sign up and subscribe for it and, but, um, it seems like to me it's, it's a taste of things to come or as they say the future's here, but it's, uh, unevenly distributed, shall we said.

Yes. I think once we get a lot of these tasks automated, we're gonna have an explosion of new technologies perhaps on, on the future. We sure hope so.

Now, there are other stories that are going on, um, on the site there. A lot of it has to do with the AI economy. So why don't you walk us through that?

Yes. Okay. So, um, Sean Mullaney recently contributed a byline and, um, he says that AI is going to be the answer to establishing a Goldilocks economy, which is when the economy is pretty much growing at a perfect rate of no, neither hot nor cold, just perfect.

Uh, he thinks AI is the solution because of its ability to automate and, um, speeding up processes, improving processes. And his belief is that, um, with this, um, the companies won't be firing individuals, they'll be hiring more individuals because as everything speeds up and everything improves, they're gonna need more humans. All right, let me get my, uh, rose colored glasses over here somewhere.

They're lying around on my, on my desk here. But, um, I think that, you know, maybe that's, uh, possible outcome in a best of all possible world's philosophy. But, um, I think, let's be honest, there's gonna be a significant amount of disruption before we get to his glorified view of the world, if ever.

Um, we are going out of our way to make sure that everybody feels like, you know, AI isn't gonna, you know, kill their job, but uh, your job's definitely gonna be different. There's just no question about it. And, uh, who's doing your job may change because the level of skills required to do it will drop and there's just gonna be, um, a lot of folks who are gonna be shifting careers.

There's, it's just a fact. Now you might wind up doing something else that's different that uses some of the same concepts, but I think we should just stop trying to go out of our way to tell everybody that, you know, don't worry you're gonna have a job. Yeah, you are gonna have a job, but it is not gonna be the same job you have today.

There's just no ways about it. And it may not even be for the same company that you're working for and it may not even be in the same field that you're in today. So, uh, it would be better if we spend more time coaching people and preparing them for that level of disruption rather than kinda, you know, trying to get everybody to feel good about AI following.

I do, I do agree with that, um, in the fact that yes, there may be more jobs opening up, but they will be different and the current jobs will shift and change and I think, um, companies do need to provide more skill up training and um, get ready for the future as far as staffing in these different areas that are gonna open up. Yeah, and, and I was having a conversation with some, uh, some folks I grew up with. I've known him all my life and he's a developer and you know, his perspective was, you know, I'm happy I'm two years away from retiring and I think I'm, I'm just hitting this just right because, um, he was like, I am, he was unsure what the kids were gonna be doing and you know, he says a lot of them have skills and stuff that they've learned in college that maybe redundant real soon.

So he was basically thankful that he was not at the front end of his career 'cause he was just like, I don't see how this is all gonna play out, but who knows? And I think there's also a simple fact that um, maybe the older you are the less inclined you are to learn new skills. So that's a bit of a challenge.

So there may be a generational issue in work here. So, um, those of us who were um, a little longer in the tooth maybe need to keep an open mind because, um, clearly everybody's relearning everything at all times. And if you're not in the mood to relearn, well then we're probably gonna have more trouble than others.

But we'll see how that goes. There's a separate related story from, uh, Nutanix talking about on the upside, the amount of infrastructure that AI is gonna consume will be going up through the roof. So if you're in the IT hardware business, as we were pointing out early in the show, uh, AI is good for you, right?

Because ultimately the amount of CPUs and storage and data that's required will increase and a lot of that will be in the cloud. A lot of that will be on premise and a lot will be at the edge. So I can almost say with certainty that AI will be good for the hardware business, but other than that I'm not entirely sure.

Yeah, I think you're exactly right on the fact that, um, depending on where you are in your career, it may be a, a good time where, you know, like your friend is happy to be stepping out soon. But I think most people in technology spaces will need to have certain characteristics such as adaptability, um, able to pivot quickly, um, and always wanting to skill up and be constantly learning and training to stay ahead of the next, the next change in technology. I think those are important characteristics to have for other people coming into the career.

Yeah, I was kind of half laughing with, um, some of the folks in the Text Strong Gang podcast that you and I occasionally show up on. But um, I was talking about, you know, I'm not sure how I feel about this, but I, I think I'm essentially creating content for machines now that are using that to create more content that, um, may or may not be competing with the content that I use to help train the machine. So it's kinda getting a little weird out there and we will see how it all kind of works out.

But that moves us to our next topic 'cause it's, it's not clear how people who produce content for training machines are actually gonna get paid. And um, the head of the Linux Foundation was over here in Paris for this cute kind event. 'cause CNCF is a arm of the Linux Foundation and he was talking during a press conference about how he thinks there might be a need for some sort of consortium that would create open data sets for people that data scientists could then use to train, um, AI models for free.

And his point was that there's a lot of data suddenly being moved behind licensing terms because the people who created that data are a little wigged out about the fact that their data is being scraped for training these AI models and they're not getting compensated for that. And his response though, on the open side was to figure out a way to encourage more entities to make, uh, free data available to help train those models. And there's a natural amount of tension between those two things because, um, I'm sure you're aware, but, uh, there's large numbers of industries that are all about creating data and getting compensated for that.

Um, Jim said he is not against those people creating that data and licensing and making money, but he was still making the point there was an apparent need for data that didn't need to be licensed. And so there's, there's tension in the system as they say, and it's not clear to me how that's gonna play out as we go along here. But something, uh, for everybody who produces data to keep an eye on because, um, I am certain that the folks who build AI models are gonna go find ways to fund free data and, and then how that kind of circles back.

And they will be interesting 'cause um, they might wind up using a lot of the money that we give them. They use their services to go create that free data, which, uh, then, uh, from a com competition standpoint will be challenging for some folks. And, um, yeah, it's, again, it's coming back full circle.

It's right, it's like, I don't know, are we working to create data for machines to create data then in turn, or is gonna, not just me, but put us all in some sort of new job and role and we'll see how that plays out. But, uh, interesting times as they say. Well, and the biggest question is the, the funding necessary and also, um, how people are gonna continue keeping it updated, who, you know, if it's all open, who's gonna keep updating and and managing it, Right?

Yeah, and you're absolutely right about that because part of these things with these AI models is, you know, you train 'em for a specific date of what the data that you have, but then, you know, six months later or nine months or a year, they start to, um, lose their cohesion because the data that they were trained on is outta date. So, you know, somebody needs to keep kind of continually updating those machines. Now sometimes machines will create additional data that will train the machines, but other times it will be humans.

But, um, you know, the dirty little secret about open AI is a lot of that stuff was trained using, um, people who were paid, you know, pennies and nickels to sit there and say, yes, by golly, that's a cat. So, um, you know, that doesn't seem like, uh, shall we say, rewarding work, Right? And, and then we've seen what, what's happening with that too.

It was open and now they're going toward a, a paid model and now there's lawsuits flying around and everything else. And so, uh, so it just comes into question how you can keep these open, um, platforms, how you can keep them open without funding and without people constantly managing them. Yeah, I mean, to be honest, we're all going forward here without much of a real plan.

We're kind of, it, basically, you know, some data scientists discover that we could do this thing and it'll be cool, and now it's um, basically, you know, buckle up buttercup time because we don't know where it's all gonna lead. But, and anybody who says that they do know for certain is, you know, probably telling you something that is the human equivalent of a hallucination. Yep.

Well, it'll be interesting to see how this all plays out and I think nobody, definitely, nobody has a clue. And from a year ago to now, it's surprising how many things have happened. All right.

Well I'm looking forward to being back in the States next week and we'll pick this conversation up again. But in the meantime, I think I'll go and enjoy some, uh, non-AI food at the lovely Paris. Sounds good.

Enjoy your trip. I can't wait to hear more. And thanks for tuning in.

We will be here next week at the same time. Thanks, Mike.