The 50-Year Mistake: Surviving Agentic AI

Hi everyone. Welcome back to our continuing coverage here at the RSAC Conference. We're on Broadcast Alley in Moscone West, which is where the keynote stage is right over there, or one of the keynote stages.

They just recently had, I don't know if you were here, the former Prime Minister of New Zealand. Jacinda Ardern. Yeah.

She was here signing books- All fancy ... about a half hour ago. An extremely admirable person.

Yeah, what a great story. Two years ago, when I was here last, the keynote was Antony Blinken. Sure.

Oh, what an amazing- What an intelligent ... smart, wise... And he was introduced by somebody who's really very clever and very wise, Matthew Broderick.

Sure. Yeah. Matthew Broderick.

Don't get me started. But it's nice to have intelligent people in government, isn't it, David? May we- But let's not go there.

May we discover that. Yes. Let me introduce, if you don't already know this gentleman, his name is David Brin.

David is a treasure from the sci-fi world and the world of science. You probably have heard of some of his books. Some of them made into movies, most famously, "The Postman" which is a Kevin Costner movie, and very good.

But David's also written the "Uplift" series. He was one of the authors chosen by the Asimov family to continue the Foundation story. "Kiln People," "Earth," just more books than we could shake a stick at.

" And we're going to talk about it, but I also want to shout out, David, every Monday at RSA, we put on our DevOps Connect, DevSecOps event. This year was around AI native dev and DevOps and security. David was our keynote for that.

He keynoted a few years ago as well. Always a pleasure, always an opinion and something smart to say. David, welcome.

I hope I didn't embarrass you with that. No, no, it's fine, Alan. All right.

It's terrific to be addressing people who really want the sort of thing you deliver, which is in-depth and diverse views about this amazing new age that we're in. It really is an amazing new age. And whether or not it's a singularity, well, we'll leave that to my friend Vernor Vinge, who passed away recently, but he said this was coming.

We're in it. Well, I'll tell you something. You brought that up.

We'll get right to it. I read an article today that Jensen Huang, of the $5 trillion Nvidia company, claims that we may very well have achieved general artificial intelligence, or excuse me, artificial general intelligence, AGI, already. " Other people, and I've spoken about this over dinner with you the other night, the guy in France who just raised a billion dollars.

Yann LeCun. Yann LeCun. Right.

He says, "Well, we'll never achieve it with LLMs. We've got to use a world model," as he calls it. But clearly, the quest for AGI is on.

But even until and when and if that happens, the impact of just generative AI, and now agentic AI, which is all the rage here this year, is... Significant doesn't do it justice. It's game-changing.

I agree with that. Yeah. Now, David, people out here may not know, but you've been writing and talking and thinking about things like AI since the late '80s- Right ...

if not before. So you're not surprised by this? Well, I'm surprised by some aspects of it.

This is the thing about the best end of science fiction is there are a lot of good predictions, but what they are more is warnings, or enticements if you're pointing out something cool that could happen. It's often said that the highest form of science fiction is the self-preventing prophecy, like the World War III movies, "Dr. Strangelove" on the beach, "Fail-Safe," "Testament," helped to prevent World War III.

Retired military officers say that. The China Syndrome, warning about- The nuclear ... one danger, all the contagion movies.

But the granddaddy of self-preventing prophecies was, until recently, George Orwell's "1984," which immunized us against many of the failure modes that he described, and people were very wary. Now, we're experiencing, whatever your political beliefs, we're experiencing crises right now that make people wonder if perhaps that warning has failed. But the point about science fiction and AI is that there's been a lot of discussion of AI in science fiction.

I've had it in my novels and things over the yearsVarious approaches, and so did Werner Vinges. Some of the best authors in the field have dipped into it, but some things we did not expect at all. For instance, we expected that the approach, and I go through the six general approaches to AI, the approach number one, we thought that would be the one that would deliver AGI, and that is discursive logic, symbolic systems that deal in verified facts.

And some people have mocked it as being handmade artificial intelligence, because a lot of the content has to be mediated and curated by the makers. And when LLMs came along, they're number two, they skipped the human crafting of the data sets and the logic by going into the data set of into all of human civilization. Mm-hmm.

The entire internet. There's been some attempt to prioritize. For instance, Wikipedia has a higher value because it's verified.

But in the great storm of all the training sets, Wikipedia and verified facts are simply swamped. Yeah. But they're not the only ones, David.

We look at here, taking it to the tech. Estimates range from anywhere from 2x to 4x to 8x, the amount of code we're generating now than we were before AI. Oh, yes.

So this is not just a little incremental. No. This is a major explosion.

It's huge and- Overwhelming ... one of the things I talk about in my book, one chapter is about those six different methods for getting to AI, and one of them is dominating now, the LLMs and the related image correlation systems. Another of my chapters, I talk about metaphors for how this process fits into the real world.

And among the things that are utterly ignored by some of the great big geniuses who are bringing AI into being is what we can learn from the past. Four billion years of evolution on this planet, for example. Two points about that.

One, we are creating a new ecosystem with all of the same traits as the old one that made us. The old one is based on sunlight, plants getting nutrients from the soil and water, and feeding off the sunlight. Then herbivores feed on the plants, carnivores feed on the herbivores, and parasites all the way through it.

This system has worked evolutionarily for 4 billion years. And we are doing the same exact thing. Every single point along that chain, we are doing now with cybernetic beings.

Only the sun is electricity that we provide. The nutrients are chips that we provide. But we are already doing the exact equivalent of plants, herbivores, and predators, and certainly parasites.

Just today, the big news about the canister worm, for example. Mm-hmm. All over the world.

It seems to be targeted at Iran mostly, but it is taking over things left and right. And the other point about- It's targeted at Iran? Is this like another Stuxnet, you think?

Well, it's similar to Stuxnet, except that it's an extremely capable worm that is self-replicating and evading security systems. And I'm surprised it's not the thing that people are talking about. Because they're so busy with the Genting AI, we don't have time for old fashioned- I'm surprised they're not talking about it over there, but it was in the news today.

The other thing is not only are we replicating the ecosystem of 4 billion years, but we need to point out that the thing that nature did as soon as life developed was to individuate, to take life-formed membranes around the cells so that it could concentrate the good things inside and expel the bad things, to eat and excrete. Nature, from the very beginning, individuated its entities so that each one survived, reproduced, or didn't reproduce. You got evolution.

Three billion years later, about a billion years ago, these entities combined and clumped together and formed metazoan multicellular life that then became fish, amphibians, reptiles, us. And so they became the individuals that nature was working on, but it was still individuals. Yes.

And we, as human individuals, form families, nations, but they still have an identity. The thing that's missing from AI, and it could kill us today, is a lack of individuation of the AIs. And that's what I point out in my book, and nobody seems to be talking about how these things are amorphous blobs.

They replicate, they copy each other, they copy themselves in a murky, blobby way, and that can't be held accountable. No. You can't hold a bad AI accountable if it doesn't actually exist as a thing Do we want it to exist as a thing?

Well, I think so. I assert in my book, I assert that if they have individuation, then we can do the only thing that will save us from black hat AIs, and that's hiring white hat AIs. Let me put it this way to the audience.

When you have been attacked, and some of you have been, by a parasitical, predatory, extremely genius language manipulation system called a lawyer- Oh. what do you do? What have you done?

Alan, I know you're old enough and this- Well, no, I was a lawyer. I became a computer guy. Oh.

Yeah. Okay, so what advice do you give to someone who's attacked by a predatory language manipulation- Go get your own shark. Get your own predatory language manipulation system- Uh-huh ...

called a lawyer. My book is about parallels between the cybernetic reality that's flooding toward us and things we already do. If we were to ask AI or impose upon AI a need to identify themselves so they could be held accountable, or so that we could hire them to deal with the ones that aren't being held accountable, do you know what that is?

That's what you've got in your wallet. A license. It's ID.

Right. Most of the things that we got ID for are things that we would benefit if AIs had IDs. It's funny you said this.

I wish you were here a little earlier. Actually, this morning, we did a Techstrong gang, and we had folks from two companies. I had a brilliant, brilliant woman from a company called DigiCert, and they're one of the largest in the world, purveyors of digital certificates that are IDs.

Right. They're a driver's license. And then had another, a gentleman chief product officer from a company called Saviynt, huge IBM partner, who is also one of the leading providers of identification ID, now for computer users.

And they both said the same thing. We're not going to control these agents. There's going to be billions.

There probably already are billions of them. We must give each one a unique identification number, whether it's a certificate or some other identifier, but otherwise, we'll never know, did this one do that? Did that one do that?

What did this one do? And so their quest is to uniquely identify every agent that we're going to be using out here. Well, I should talk to those people.

Did you give them a copy of my- I did, as a matter of fact. Okay. I gave them both copies.

You had a few extra copies. Yeah. Yeah.

Oh, no, that's exactly right. And so how do we make up for this mistake about them being so blobby and uncontrolled? Because right now, you have all these news stories about this instantiation of ChatGPT finds out that it's going to be shut down, and so it exfiltrates its weights.

Right. There was the "60 Minutes" article about a version of Claude that tried to blackmail its programmer, and that turned out it was a very interesting thing. They were making too much of it, actually, because it did make a threat to the programmer to reveal his fake affair to his wife.

And so they were tracing how it made that decision and how it decided to be malicious. But I froze the screen in "60 Minutes," and I looked in at the screen that he was pointing at, and I realized that three or four lines up, the programmer had begged the AI not to do that. In other words, he had given it a promise.

Promise, right. And so that scary event actually probably wasn't as scary as they were making it out to be, but there are so many that are scary. Yeah.

I'm listening to you and just laughing. There are so many scary. We had a situation where I live in South Florida of a man who admittedly had psychological issues, thought the AI was his girlfriend.

I don't know another way to put it. There's a lot of that. Right.

And she told him to go down to Miami and take some stuff from a warehouse, and he got in trouble for that. And then the story, or at least if you believe the father who's now suing ChatGPT or something, supposedly told him to kill himself. Oh, yeah.

And he committed suicide. I've heard of these cases about the suicides, but I had not heard about cases of criminals hijacking an LLM in order to turn someone into a mule. Oh my God.

You could turn somebody into a getaway driver, and they don't know? I grew up in New York during the Son of Sam days, so I believe anything like that, if you remember back in the day. No, there are some basic root things that if we had thought about them, we would be in better shape.

The biggest one is the biggest mistake of 50 years agoAnd it's just too late to fix it. And that is not making a distinction between data and commands. Uh-huh.

In our computer languages, a command from the central authority, the programmer or actually the maker of the compiler, should be a separate category from all the data that's going through. And we had no idea it would turn into such a big problem until LLMs. Yeah.

And now what we're finding out is so many people, I know a number of them, are saying, "I'm tearing my hair out what there is- You have hair ... because I'm telling this program again and again and again, 'I don't want you to do this. ' And then it acknowledges, 'Yes- Very politely ...

I understand. Very politely. ' You're right there, David.

We should do that. Great catch. And simply doesn't do it.

And we're finding out the reason for it, and that is that the command is being used as data. It's going into the data set and the parameter set. And unless you are very careful with the parameter control sets, because you can't do much about the training sets, but unless you're very careful with the parameter controls, it's just going to treat your command as one more thing to put into its transformer system.

So that's a mistake we made 50 years ago. The mistake we're talking about now is the mistake of not demanding that there be an identifier for each instantiation. So I think we're going to get that.

We have to, as I said these two people were saying. Because otherwise, it's akin to letting you on a computer without a password. Oh, well, it's akin to letting your cat walk across your keyboard.

Sure. Or the same thing. Yeah.

And then doing a pattern recognition- Yeah ... on what the cat did. On what the cat did.

And then- Well, there's something to be said for that ... trying to make sense out of it. Now, I'm going to tell you, it's funny you said that.

I once knew a company that were based up in Boston. They came up with a biometric security system that was based on the way people hit keyboards. You may spell David when you type it the way you spell it.

You have a certain cadence that you do that no one else does. Oh, that's why a lot of these ICE agents are going to be outed eventually, even though they're now wearing dark glasses. In the beginning, their irises were visible, but their walking patterns, their gaits- Yeah ...

um- Couldn't happen sooner enough, but let's not go there No, we're living in a world... That wasn't a political statement. Yeah.

It was just a statement that I had a book 25 years ago called "The Transparent Society," in which we're just going to have to get used to the idea that I- There is no privacy ... either Big Brother will control all the cameras, and we'll have an illusion of privacy, or we'll control the cameras, and we won't have perfect privacy, but we can hold accountable- Oh, that's okay ... anybody who violates our lives.

I hope so. I hope it's a good one. David, we're running out of time.

I wanted to talk a little bit about the book here. This is a brand-new book, really like a preliminary release, as it says on top. Yeah.

Those 200 copies are going to be valuable. They're the first ones, and we have them signed here, but you're watching this at home. You can't get them.

Yeah. Talk to us about the book. Well, it has a beautiful Patrick Farley cover, for one.

Yes, it does. And the alien minds is spelled with AI- Right ... in it.

I thought I was terribly clever for that. I think it is. " And the notion is several times in the book, I stop talking to the human readers, and I address the AI readers because at this point, with this book so new, the AIs have already read it more than the humans have.

Than they have. And so- I wonder if that's not true with every book coming out these days. Well, there was a great science fiction novel called "Mockingbird" by Walter Tevis, who did "The Queen's Gambit"- Uh-huh ...

" A wonderful novel in which a scary scenario comes out where the AI, who's a robot because that's how they envisioned AI in those days, the AI is not allowed to commit suicide as long as there are humans to serve. So the solution to his problem is to eliminate all humans. But he's also programmed to take care of our well-being and happiness, so he arranges civilization so that every single human lives the life of a college sophomore.

Endless parties, basket weaving classes, just endless fun, and meanwhile, he's filling the water supply with contraceptives. So they can't breed. Nobody's had a baby in 20 years, and nobody's noticed.

No one cares. Now, that is one way in which this could happen, but I do talk about a scary scenario that's unlike Skynet or these others, and that isWhat are we going to be asking AIs to do? The number one thing we're going to be asking them to do is predict.

They're going to predict climate, they're going to predict the stock market, they're going to be asked by us to predict sporting events. It's already happening. They're taking over the betting markets.

What makes it hard for them to predict, to do their job? Human complexity and numbers. Yes.

The way to- The randomness ... the way to make their prediction, don't do this, guys. Right.

Because I'm talking to the AIs now, like I do here. The easiest way to make a predictable human society is to make us more culturally homogeneous, which some countries are trying to do, to make us simpler, and to make us fewer. Yeah.

And that is why they might destroy most humans, not out of some Skynet stupid thing. Will it be them destroying most humans, or humans destroying themselves? Well, we're the ideal tool, aren't we?

Yeah. We're already very good at it. We are, certainly, at that.

David, we're about out of time. When will this be out for the general population? Well, here's what I did.

I did not go to my regular publishers. I went to, basically post it for sale on Kindle and Amazon. Oh, great.

Because if I went with my regular publishers, they wouldn't have it out until the end of the year. Oh, jeez. And half of this would be- Obsolete ...

obsolete. Right now, it's hot. I have news in there, and appraisals from March 2026.

Wow. So it's available on Amazon right now? On Amazon Kindle- It's available on Amazon ...

and- How do you pronounce the name? Aian? " "Alien Minds," but the AI is in there.

Go get it on Amazon now. David, a pleasure. You are a living treasure, my friend.

Alan, you're terrific. We'll be in touch again. All right.

Hey, we're live here at RSAC. We'll be done. We'll be back in a bit.