Project Lightwell, Taiwan’s AI Bet and the Open Model Safety Problem

Hey everybody, welcome to Techstrong gang, and as usual we're talking about billions of dollars being tossed around, so it's just your typical Thursday, I guess. But let me introduce our gang members today. We've got Gina Rosenthal.

Gina, how are you? I'm good. How are you?

Good to see you. Fred Wilmot, good to see you as well, my friend. Likewise, sir.

And John Schwartz, I think, judging by the background, are you home? You look like you're home. I'm home.

It's raining. There are a couple of car accidents out on the streets, but as you know, Mike, having lived here, if it sprinkles, there'll be car accidents everywhere. This is true.

And I've noticed that is true of everywhere outside of the Northeast and Washington. Nobody seems to be able to drive in the rain, but it's another reason why we need those cars that drive themselves, right? Just they get all those people off the road.

They are everywhere here. Waymos are everywhere where I live. Yeah.

Right. But I digress. Let's jump into the news of the day, which is IBM and Red Hat are collectively pledging something like $5 billion to go find all these open source vulnerability issues that will need to be fixed, hopefully before AI models discover them, because, well, we may be having a serious problem coming up here if suddenly there are more vulnerabilities being discovered than we can possibly patch or remediate, unless maybe we start using AI tools.

But I'm confused about this, so Fred, I'm going to toss this to you. But if I listen to some folks on one end of the extreme, they are saying that the vulnerability butcher's bill has finally come to the point where it needs to be paid, and this is now the time, and whether we like it or not, the cost of application development is going up. Then I have other folks who are saying to the effect that, well, all the serious bugs have already been discovered, and that basically these are all low-level bugs that will be discovered and they should be patched, but it is not this impending apocalypse.

Fred, I don't know where you fit in that whole spectrum in between those two extremes, but what's your take here? I think about it the same way we think about the number of pictures you have from when you were a kid. " But at that moment in time, I took a picture of something, and today my recollection of that is, yeah, it was a great time.

It was amazing. This is the same parable of what we think about vulnerability research. At some point in time it was bad, now it's going to get better.

The answer is, sounds great. Let's wait and see. But objectively, what IBM has decided, and I think this is great, is that they're going to get together and invest a bunch of money with Red Hat to devise a way to clearing house these vulnerabilities.

And oh, by the way, there's a great business model here behind it. And oh, by the way, it has some impact maybe on the investment in Quantum that we'll talk about from the federal government later. But here's how it works.

Here's what they're thinking. Instead of thinking about AI engineering as a tax, it's a benefit, and you're welcome. IBM will take care of both the vulnerabilities for you, as well as the AI computing requirements using the 20,000 AI engineers that they want to hire.

Brilliant. Question becomes, what are they solving for? And is it really about the entirety of all of the stacks of open source technologies?

It's an easy punch list to suggest that 6,000 vulnerabilities from Mythos or... Look, the model isn't really the thing that gets that problem across the threshold. It is the remedy and the cure.

And so if IBM saw this as an opportunistic way to capture a market where the rate of change, we're no longer taking pictures with a camera or a Kodak photo box. It's now a live stream. Vulnerabilities are live stream.

Remedy's got to be live stream. And so in this case, perfect business opportunity to do this. Does $5 billion really make a dent in this problem?

Let's see how they prioritize which projects they're going to work on first based on what's good for the industry or good for customers. That will be the end of Skype. Now, this comes on the heels of a paltry $12 million that was pledged to help the open source community go fix this stuff.

And at the end of the day, you got to wonder what IBM's thinking here, because 5 billion is a lot more than 12 point million and change. And I'm assuming that if this is a for-profit venture, that they think that they're going to make more than $5 billion as a result of this thing. So to me though, I don't know, John, are they just trying to make a buck off of our existing illnesses and this is suddenly them trying to be the medical industry for open source software?

I don't know. Yeah. When I come across these partnerships or these funding where they just throw these numbers out and we have these big names associated with them, we're so early in the game, we don't know what the eventual outcome's going to be.

One of the things that I think about when I looked at this, and this happened this morning, is how are IBM and Red Hat going to balance the commercial nature of these Lightwell subscriptions with just kind of the free sharing ethos of the open source community? That kind of jumps out at me. I know they have a lot of ambitions on their plates, but I just wonder how they're going to be able to execute any of this.

And again, it's unfair for me to pass judgment because it's so early in the process, but I've almost... This might even bleed into our next subject, but I'm just starting to become so kind of-numb to all these numbers being thrown in, partnerships being thrown out, and ultimately what they accomplish because they're so early in the phases that they all are in. Yeah.

I think what we're really calling for here is universal healthcare for open source software. That would be kind of cool, but we'll see how that goes at the end of the day. Gina, I want to get your opinion on something here.

So theoretically, we're going to use AI to create all these patches to go solve all these vulnerabilities that are about to be unleashed. Are IT teams prepared to actually apply those patches? Because normally it takes them months to test, validate, and install, and usually they get around to it on their own sweet time.

And the bad guys seem to be able to now create exploits in a matter of hours, days. So is the whole system going to collapse on its own weight? That's pretty dramatic.

So I'm not sure. I just listened to a podcast from utilizing AI from the Tech Field Day crew this morning, about just how much is changing and how fast it's changing, and how supply chain impacts that and what you could do, and how do you make a decision about what to do because of that. So I think two things.

I think IT people are probably stuck in the middle between the old world and the new world. You're not going to be able to just toss off what you're doing now because that's what supports business now. But of course, you're trying to figure out how do we implement agents and all the other things we want to do, even open source, how do we implement that into our new stack?

What's that going to look like, and how do we do all the things to keep our business secure? So I think they're kind of stuck in the middle, but you can bet that the IT folks are trying to figure out how to do both at the same time, because who wants to be stuck supporting a dinosaur? Nobody.

Mm-hmm. Everybody's trying to figure out the new way to do this, so that they can be part of this new world with AI. What I got from this article is this idea of working out loud.

This is what I feel like IBM is doing with Red Hat, and they have the resources to do that between their server, their hybrid offerings that Red Hat has, and then all the money and resources apparently that IBM's going to contribute. They have this chance to look at, okay, we all want to continue using open source, but we need to triage so we can get it from the dinosaur ages into the new ages of AI. So we'll throw a bunch of money.

We'll figure out how to do this. We'll then have this whole consulting service built from the experience of our people solving this problem for the industry. And then Red Hat will be able to say, "Look, we've got all this experience in how we solved it on...

" So that's what I feel like is going on. Right now, everybody's in this whole, let's try it and see what works. Let's throw what we know could be a problem against it, and see if we can solve for that problem.

And back in the golden days of blogging, that's what we saw. Maybe that's what's going to happen with podcasting now. With people who are trying it and try to figure it out, they work out loud, everybody else gets to see it, and it's not contained in a box of a corporation that is trying to save those learnings for a business model in the future.

Mm-hmm. So I would say everybody needs to work out loud and do this and try to figure out what the best ways to go are, what are the workarounds when the supply chain becomes constricted, and let's see what we have to do to make this next stage of computing a reality. Can I ask a quick question, either of Mike or Fred?

One of the things that cropped up in my mind was is things like Mythos preview become increasingly adept at finding zero-day vulnerabilities. Would Lightrail's AI defenses be fast enough to patch code before bad actors weaponize automated discoveries? Is that- This is the core question, and I'll add to that and let Fred answer, but it would seem to me like I can pay IBM, Red Hat, or somebody else an inordinate amount of money to go do this, or I could invest in new systems to automate patch management and create the patches hopefully faster using AI coding tools, which I suspect might be a whole lot cheaper than asking IBM and Red Hat to go fix this stuff.

But I don't know. Fred, is that reasonable? I think it really depends on where you are in the journey and how big your business is.

This really is a pretty smart move by IBM. They use more open source than most companies in the world. They can patch the software that they're providing.

They have an infrastructure to deploy it on, and they're going to identify a workforce that will allow people to utilize them to do that patching. Mm-hmm. I just don't think we're going to invent a better mousetrap here.

It's decades of legacy technology. Doesn't matter how fast you move. The slower you move, the bigger you are, and vice versa.

And I think that it's sort of a counter-narrative on how AI works. So it's also pretty smart for an organization like IBM to look at the requirement for I don't want my banking institutions and my banking infrastructure to be held accountable to Mode AI or anybody for that matter, on whether or not they feel like this model is acceptable for public use. And so here is another little bit of a, let's call it independence, a third horse in the race here that allows something like a very stable financial sector to useA methodology and also to support an infrastructure that allows a clearing house, paid or otherwise, right?

That rising tide may raise all boats considering the amount of open source they do utilize, so that there's actually going to be some industry impact. I think just the same as we say, "Hey, it's really cool for you to put $10 a month in an IRA," putting $5 billion in that IRA has decidedly better effects. So I'm hoping that this has a pretty positive impact, and I would say there's also a lot of other signaling that would illustrate that this is well beyond the premise of can we fix patching at this moment?

So, it's good to see. Here's my issue with it. Sorry.

Here's my issue with it. So let me get this straight. To your point, IBM is using a boatload of open source software all through their organization, which they already do need to patch and will patch.

And then most of their customers who have a relationship with them have some sort of consulting engagement with them. So aren't they already kind of getting some service from IBM? " So I don't know.

Gina, does that sound crazy or what? You or I vet the plan. I think that people that use IBM services rely on them and trust them for them to understand their very complex systems and understand how to provide solutions to them, right?

And I kind of agree with what Fred was saying. This isn't just we're going to fix a patch, we're going to throw a patch out there really quickly. And we've talked about this before, and Fred's talked about this before.

It all depends on how your environment is and how you've got it set up. If you even patch at all, maybe it doesn't make sense for you to patch. Maybe it makes sense to solidify how that infrastructure is being approached and how fortified it is.

So this is literally one of those it depends statements. I agree with what Fred said. They've got the infrastructure, they've got the experience.

They're willing to train more people to have the experience. They've got the money to put in this to fix it. I think as they learn from one client, and it's not going to be a tiny shop, most likely it's going to be a banking system, a medical system, something very involved that not only has very legacy equipment, maybe even including mainframes, all the way up to agents and AI and all sorts of equipment that nobody even knows about and how it gets on a network, but that's core to their operations.

So they'll have those case studies and be able to give people examples of, okay, here's what you want to do. You might want to think about all these things before you rush to put on a patch. So I think it's a more complex question, and I think the more complex environments that they are able to get in and really work with and address these issues with, then they'll have even more experience and lots of case studies to show us how awesome they are, and people will depend on them more.

And I bet that's what they're betting, too. John, here's my bet, my prediction, because as you well know, every action has an opposite and equal reaction. And so therefore, what we'll see is, since Red Hat accounts for a large segment of the open source population, but hardly all of it, most of the vendors that are in that space will circle their wagons, go find a bunch of consulting firms to align with and launch a very similar service.

Maybe they'll- Oh ... even say that collectively they're putting in, I don't know, since we tend to make up a lot of numbers, what the hell, $15 billion in investment. Yeah.

Right. And then they'll say that they're going to deliver that service more efficiently and at a lower cost. What are my odds that that's going to happen?

The odds are very good that's going to happen. It's a great commercial opportunity, and for every action, there's a reaction, it seems daily in this field. And so, I think you're right, Mike.

I say that a lot. You're right, Mike. I don't know about that.

We'll see. It's sauce for the goose, Mike. Yeah.

And the downside is, or the question is simply how many of these organizations are going to commit to the open source projects, their patches, and how fast? Right? Right.

Proof is in the pudding. Sounds great. Make a lot of money.

Sounds great. Get competitive about it so it improves faster. Sounds great.

How many of these projects are you going to fix? True. And then there's another unspoken part of this thing.

" I don't know, Fred, you're laughing, but what do you say? Man, if it were that easy, yeah, that would've been done a long time ago already. Yeah.

That's just not a reality. I wish. I wish.

In a lot of cases. Not going to happen anytime soon? All right.

I'll take your word for that. So I think that means that ultimately we're kind of stuck, and I think it leaves a bad taste in some folks' mouth because they're like, "We've been using this software, we've been paying for support for it, and now we're going to have to pay more to get it fixed and maintained and-At some point, where does that function belong? Is that added value, or is that just something that should be cost of the service?

Gina, close us out here, but what do you think? I'm not sure that they'll have to pay more. If it's a contribution that IBM is making with that billions and billions of dollars, I don't believe they're doing it out of the goodness of their heart.

I think they have a business plan of how supporting this and helping solve this problem will elevate them in open source communities and their opinions. And then there's also another money-making profitability on that track that they can take from it. But I'm not sure that cost is going to be passed along, or they wouldn't be committing so much money.

That's what I think remains to be seen. Will the cost be passed along, or this is open source that IBM is just contributing, and they're going to recoup that expense with reputation increase in the open source community, as well as a paid way to do this with closed source private applications? " Right?

I don't know that I agree, but maybe. Maybe. org, so somebody's got to make some money out of this thing.

All right. I'm going to shift a gear here and talk a little bit about, well, more billions of dollars. And our friends over here at NVIDIA have pledged to spend $150 billion a year- A year ...

a year now to support the development of an AI hub in Taiwan. This strikes me as kind of interesting because our esteemed president just went to China, and they had a visit, and he brought all those execs with him, and I guess maybe, John, I don't know, did they not hear what they wanted to hear- ... so now money back into Taiwan in a big way?

I thought all this stuff was coming to the US anyway, but maybe- Yes, exactly. It's the counterintuitive move. So rather than diversifying away from a geopolitical hotspot, NVIDIA's leaning heavily into its reliance on Taiwan, which it refers to as the epicenter of the global AI revolution.

That's Jensen Huang's comment about it. He thinks in the foreseeable future, cutting edge AI infrastructure can't be built anywhere else at the scale it can be built in Taiwan. So you mentioned $150 billion a year, which is their now their new commitment.

Four years ago, it was $10 to $15 billion a year. So this is going to be every year, and the centerpiece is this AI hub that they're building. They're going to employ a huge number of people.

This localized expansion is this multi-billion dollar Taiwan headquarters and office complex called Constellation, which is in northern Taipei, scheduled to break around this year and be operationally in office and use by 2030. So in a sense, this whole idea is we're moving NVIDIA closer to TSMC, and there's a projection among analysts that NVIDIA's going to surpass Apple this year to become TSMC's largest customer. There's going to be more reliance or more partnerships with folks like Foxconn, Wistron, Quantic Computer there.

It's going to be interesting because in a sense, as you mentioned, mainland China's this cloud hanging over all of the Taiwan investments, and it's a big unknown. This is something that I was talking to Jack Gold about, especially the Trump administration seems to be softening its support of Taiwan. So maybe something Jensen knows or doesn't trust from the Trump administration, but he's heading headfirst into this, despite the geopolitical risks.

It's interesting because today, I think you passed this on to me, Mike, there was a World Economic Forum report on geopolitical influences in AI, and NVIDIA is making the big bet because they think if they want to grow at scale, this is the place they have to be. Mm-hmm. Now, I'm assuming that at some point there's a tax applied to this, and I think that the tax cut on that $150 billion might just buy a lot of drones that Taiwan apparently is going to need if they're going to remain independent.

So how much of this is also part of the Taiwan defense complex? Because ultimately, that's what it's coming down to is we need to protect that semiconductor industry because, well, whatever we're doing elsewhere in the world isn't going to be relevant this decade. Right.

That's a good point. It's interesting. This is a shrewd political move, I think, also, by Taiwan and NVIDIA, in a sense, it protects the island in a sense, or kind of emboldens or pushes the Trump administration to be more protective because the hottest company is NVIDIA at this point.

He doesn't want to compromise it in any way. So in a sense, they force his hand into protecting them. And Taiwan actually, in a sense, creates a kind of protective cocoon.

I think of that as the political element of this, in addition to the innovation argument. Right. And it remains to be seen how much of this actually happens because pledges are pledges.

But GinaAre you enjoying the theater that's going around all this stuff these days with processor companies and AI and countries and... It's high drama. It really is.

Maybe somebody should make a movie about it or something because- ... I really feel for all of us because it's very hard to get your mind around what's actually going on. Right?

So is this really going to happen? Was it to protect Taiwan? Is it to force something to happen over here that's not happening fast enough?

Who the heck knows what's going on? You just have to sit back and wait for the hardware to roll out, honestly, in my opinion and go from there. But it's very hard to keep up with all of this.

It's true, that. The other thing that occurs to me though, and I don't know, John, your thoughts maybe, but it is 2026. Midterms are coming up, and by the time- Yeah ...

all these deals play out, it'll be past November of 2028 or January of 2029, depending on when you're counting. " Yeah, we were talking a little bit about this yesterday in terms of data centers and the impact on local areas. We were talking about impacts on places like Louisiana and Texas.

And we talked about the short-term benefits, but then we also look at the long view and where things are actually going to end up being. And like Gina was saying, this is all so unpredictable. I think all bets are off too, if some things start developing, and I'm talking about market conditions or economic conditions that cause these companies to, oh, I don't know, scale back projects or drop them altogether.

I think we're going to see a fair number of those stories in the next several months. And that will be several years, where we're just going to forget about these things that had been brought up, these projects, or they're going to be a minuscule version of what they had in mind. And with NVIDIA, I would tend to believe them more so than others.

But in terms of a lot of these projects, the hundreds of billions of dollars that are being allegedly going to be spent and are going to be poured into these economies, I'm not so sure it's going to happen at the pace we've been promised. I'm certain it's not, actually. Yeah.

I'm curious what the conversations were. Right? This announcement comes out after a trip to China, not before it.

Right? Right. So there's got to be...

Jensen was born in Taiwan- Yeah ... so he has a close affinity. I am sure none of that is lost on him, so I doubt he would hedge.

Right. So and so, I think it's going to be interesting to see what does happen, but what I like about this is that maybe it means that we're not going to have the type of shenanigans we think we're going to have future forward. Or, he is hedging a bet, and maybe a conversation was had in that case that irrespective of what happens, right?

But maximize your revenue. 50% of your increase of revenue came through Taiwan, and you lost 50% in Hong Kong and China. It makes a ton of sense.

Yeah. Given his reputation and his heritage and his knowledge of that market, I would believe him significantly more than I would believe other companies, I'm thinking of Oracle or even to a lesser extent, OpenAI. So I give him a lot of credit.

Now evidently, I think there is going to be some sort of Asia Pacific meeting later in the fall, and then I believe the G7 is going to be in Florida after that. So there are going to be more meetings between the US government and China, but regardless, like Fred said, I'm pretty sure that this is going to be legitimate. You think they actually broached this with the Chinese officials when they went over there and said, "We're going to do this thing," and the Chinese look at this and just go, "Yeah, well, that's fine.

" It was interesting because Jensen wasn't initially on that trip to China when Trump went with a couple of other CEOs. I think Tim Cook was there, Elon Musk. They picked him up in Alaska at the last minute.

So perhaps, I'm sure this was broached. Mm-hmm. No surprises there.

The other thing about NVIDIA too is, and I think one of the things that people are overlooking is they are investing more R&D in things that go beyond GPUs. They have recognized that GPUs are the answer of the moment for AI but may not be the permanent answer. So they are building out CPUs and probably also looking at AI accelerators and all kinds of other things, all of which were probably, shall we say, better suited to being manufactured in Taiwan than just about anywhere else because as I look at the other projects that are out there, it seems like they're all aimed at more general purpose processors.

So are we splitting the market where maybe the folks at TMSC are saying, "Yeah, you can have the general-purpose commodity stuff. " What do you think, John? Yeah.

That's an interesting point. Yeah, I do think that is the case. NVIDIA is so fascinating to me in so many ways.

His legacy, I think when all is said and done and when the smoke clears from this era that we're in, the oligarchy tech bro era, he was the guy who stuck by his convictions, had a plan, didn't kneel down, bow down to the certain political... And he wasn't overt about it. He played this really well and adroitly and smart.

He's been playing three-dimensional chess compared to the other tech CEOs, and I think in the end It's going to be vindicated, it is every year, by their market valuation, by their revenue, and by just the scope of what they do. So, I think this guy is just the master strategist, and this is what makes him, I think, the best tech CEO in the world right now. So I don't know about that.

So a counter to that. Mm. Nvidia has a complete control on the narrative around what is required in a data center to run AI, and it's not always the correct narrative.

There's been a lot of things that if they saw it as a challenge, they either bought the company or they dragged their feet wanting to work and cooperate with other big organizations, or they find ways to silence them in a lot of ways, which I won't go into here. So they have a complete control over what you should be using, and that's why they've created this whole fever around their GPUs. Which not to knock the GPUs, they absolutely work, but they're not the best thing for every part of an AI pipeline, and they're not the best thing for every single scenario.

It is a tool, and it's getting put into being used for every single thing no matter what. And because of that, that's why the market rating is so high. So I think there's also probably another dimension to this- Yeah ...

maybe we don't see it because he is such a good three-dimensional chess player, that you now have the other chip makers getting up to speed with their stories, getting ready to launch things that can work, especially on the inference side of the house, maybe perhaps better than anything else works, that are claimed to. Well, yeah. So- Yeah, you're right, Gina.

He has a great public face, and he sells the narrative of him as this kind of benevolent leader that's liked by everyone. But you're right, behind the scenes it's probably an entirely different story, but that's part of the effectiveness of the way that companies run. They play hardball, but they're not publicly overt about it.

But- Exactly ... they can do both. They can throw hard, they can throw with panache.

Well, they throw hard, and coming from having to work with them- Yeah ... from a product marketing perspective, it's not just hard, it's dirty. So that's the reason they can control the narrative the way that they do.

I would- Go ahead. Ultimately, I completely stand with John on whether or not Jensen is able to weather political ambition and build one of the biggest companies in the world. There's no question about whether or not he's got the competency to do such a thing, the messaging, the spin, whatever.

The fact is, is he's created that market capitalization. So, the numbers tell that story rather than the intent behind it or whether he's a good person, or... That's great.

Anyway, last time I checked, we were all locked into the software libraries anyway. So I think we're all in for the ride whether we want to go or not. And the other thing I would also say is, hey, like a wise man once said, follow the money, because that's where the story really is.

All right. Let's shift to our next topic, which is we're returning to a conversation about open source and AI. And we talked on a previous show about how the Linux Foundation was sending signals that the open source folks have yet to begin their fight and their response, and their models will get better and the cost will get down, and then they will create these open data communities.

They have data sources that are less costly, and the world will be a better place. However, on the way to the forum, we discovered now there's a report saying that somebody was using a tool readily available on GitHub that strip out all the guardrails that folks had wrapped around various open source models, and basically, they could be used to create anything, including weapons of various types, and attacks, and all kinds of vulnerabilities and just general mayhem. So Gina, have we reached something of a crossroads here where is AI just too dangerous to make widely available under an open source license?

Or what do you think here? I think there needs to be a lot of discussions had on this topic for sure, because that article was a little scary, and what stood out to me was not just that the AI safety guardrails that are trained into a model can be bypassed, but how easy that is to do. So the report talks about researchers who found that free tools like Heretic, which is built by a guy named Philip Emanuel Waidman, AKA Pugh, can strip safety layers from models like Llama and Gemma in just minutes.

And they use this technique called, I'm going to say this wrong, abili- Yeah, I had a hard time with that too ... obliteration. Obliteration.

Obliteration. Obliteration, basically. Which that's a really scary sounding technique, too, obliteration.

So basically, it deletes or overrides all of the layers that enforce the policy that you can't tell someone how to build a bomb. You can't tell someone how much poison they should use per body weight to kill somebody. Those are bad things, and we're going to have these safety layers against them.

Obliteration wipes those all out, and the models also behave really differently, and they'll just respond to whatever you ask it to do. So that means that if they're that easy to wipe out, there's not a safety control plane in the models at all. Because as soon as you get rid of the weights, the control belongs to whoever has downloaded the open source model and wiped all the weights out.

So, can we have a safe model? And II don't know. If safety can be taken away that easily, it's just not persistent.

It can't survive distribution in an open source world. So, I don't know where we go from this. It means that there has to be a way to make it difficult to wipe away the layers, but I don't know enough about large language models and the machine learning that goes into that to know if those layers are there, can you ship an open source model and secure those layers so that they are indeed safe?

So that's a question back to you all, I guess. Friend, does this just also expose the fact that maybe the guardrails themselves weren't worth a damn in the first place, and we were just handing out a bunch of AI pacifiers that made people feel better about things, but the reality is the reality. What do you say?

A lot of layers to this onion, literally and metaphorically. The idea behind obliteration is not new. It's not novel.

But the usefulness of the model afterwards is kind of why this is interesting. The removal of guardrails isn't necessarily really the problem here, it's just the next instantiation of something. But Tina's right, there's a harness that needs to be had here.

And Wednesday, the Agent Control Standard project was released. So the philosophy isn't that models are going to save you, or agents aren't going to save you, or the harness that you use in order to drive an agent is going to save you. It's all of the above and the strictures therein.

So whether or not somebody is able to take the guardrails off of a model, relatively speaking, I would rather know what does and does not affect the guardrails for an open source model than not. The delta for this is how do you re-secure something once it's been torn up, opened up, or obliteration basically makes a model available to the things that you want the model to do, and done well, it reduces the harmful effects of obliterating in the process. So the real question isn't whether or not downloadable large language models that are unsupervised and use open weights, and this affects the transformers that are associated with those, is what level of control can we surface that removes the danger of what happens when you utilize information therein or agents that manifest it, right?

It's really less about humans and more about agents from the standpoint of how do you make sure agents don't make bad decisions, right? Because agents don't have conscience. They might have intention.

They might have a number of other things, including some level of intuition. But they're not going to have the conscience part of this equation. So the reality of it is that instrumentation behind how all these pieces go together is really, this is great sauce for that goose, as Shakespeare would say.

All right. John, can you imagine a world where we have politicians on both sides of the aisle who are asking questions about AI? Will there not, at some point in the near future, maybe be a hearing where some people are called to explain this stuff in maybe a language that the average person can understand?

There's even a fellow running for Congress here in New York who has a computer science degree. He may be the first one in Congress who actually understands any of this stuff. But are we going to get to the point where a larger conversation is going to be had?

God, we've been down this path so many times before. I'm just so jaded by the whole process because you're right. Those members of Congress, you can probably count on one hand, members of the Senate and the House who have any type of computer background who understand what's going on.

And the executives know this, so they're going to give them a line. And it's been perfected by people like Zuckerberg, who are able to mollify folks or just talk around a subject or kind of almost symbolically pat them on the head while saying, "We'll get back to you," and they never have any intent of doing so. So- Is that before the contribution to the campaign?

Which one is it? Yeah. I think if this applies to things like social media, which are, I think easily more explainable than this, I can't see any productive hearing.

There will be some sort of... I'm sure a Congress member who has a good staffer will mention that software tools can strip safety boundaries in under 10 minutes. And there's going to be this argument of where the burden of legal and ethical responsibility falls.

Is it with Meta or Google, one of the developers? Is it with GitHub as a hosting platform? Is it with the creators of the bypassing tools?

I think that would be constructive if they were to talk about something like that, but I don't have a lot of hope when it comes to congressional hearings on any level. That's not the right forum for this. Well, and think about it, this isn't about building a better model because the damage is already done.

And the damage was done originally when they gathered the data to build the models. " Right? So you're going to get a visit.

" And then it would give you the step-by-step description. So they trained all of that out, so that really harmful information wouldn't be readily available. So there's two questions, right?

Do we have toBut how do we fix the models that have that information in it? We know that no one should ever access it. Is it possible to put a cement layer over that to make sure that nobody ever can access it again?

And we can't forget that so many people's work was stolen to build the models in the first place. It's there. That's going to come back to bite everybody over the next couple of years, too.

And when you talk about Congress people, I think there's so much danger to this. There were articles in the morning news about, again, after all these years, service members in war zones, that's not really a war, but whatever, that are having their personal data tracked by their social media apps, their Google, whatever, and they're very easy to find now in war theater. So I'm very sure that that's very easy for the common, not common, but maybe a not espionage level bad actor to find that because of the tool, by coding this tool or being able to build a tool to do it.

All sorts of layers and layers of bad actors now that don't have consciences, let alone the agents that don't even have agency to figure out how to do something like this. Bad actors now have the easiest tools they've ever had in their lifetimes to build entire businesses out of being a bad guy with really, really strong tools, and this is just another piece that's handed to them to do so. I don't know.

We need that concrete layer to put on the models, and I don't think that exists, but... Trent, what's the right forum to have this conversation? Yeah.

So the first thing is there needs to be a regulatory inflection point here, and I think this will help prompt that. But the regulatory inflection point isn't going to come from Congress. It's going to come from insurers.

It's going to come from bankers. It's going to come from places where money transpires or transacts, and that is the right sort of dialogue to have. I think the implications are where does risk and where does the liability lie, are always going to be the indices by which capitalism moves.

So from this perspective, we're going to get into a debate about whether or not open-weight versus closed-weight models are better. No. " I think it's more philosophy than safety debates.

And the interesting part about what really matters here is whether or not models are pushing forward the boundaries of making things better, not whether or not anything can be used for adversarial use. Everything can be used for adversarial use. The question is, is somebody in industry enough to be able to take advantage of that?

And the more people that do, right? The same way we think about whistleblowers, the same way we think about responsible disclosure, all of those elements are designed so that people do not get the hammer thrown at them simply because they broke a model, broke a software license, broke into a particular implementation that was weak to begin with. And that is really the bedrock of how cybersecurity is made for progress in spite of itself over time.

And the folks that do that work should not be admonished for doing it. They should be put in a place where they can safely do it, and we can monitor the outcomes and benefit as a result. So I think there is a community problem here.

There is definitely a regulatory inflection point that needs to hit, and this will help with that. And then I think the final part of it is the opportunization of what happens with these models is going to have commercial effects, and that will all drive an insurance conversation. Everything we see on insurance retainers now, in two years become policy.

So we'll let the insurers go advocate for everything in Congress. Two years from now, that'll be a regulation. There you go.

All right, folks. Well, you heard it here. Insurance industry is going to save us from ourselves.

I'm having a hard time with that one, but hey. And we're paying them to do it. But I would say, the theme of the day is with AI, hey, there's still many more things unknown than known.

So if you're getting that sense that we don't know what we're doing exactly, you got a lot of company because that's about the size of it. I want to thank everybody for sharing their insights and their thoughts. As always, want to thank you all for spending some time with us, and please watch the rerun of the Techstrong TV lineup today coming up right after this.

And until then, we'll see you all tomorrow.