SAP’s AI Surcharge | FBI Surveillance Fight | Meta’s Rogue AI Problem

Hey, everybody. Welcome to Techstrong Game for Friday, March 20th. Uh, I guess there's a little March Madness in the air, and it also applies to technology.

But we have a few guests today who can bring some sanity maybe to this conversation. We have John Schwartz, of course, out on the West Coast. John, how you doing?

I'm good. Very good. And then Jack Palmer is back again, 'cause, well, we're talking about security and government, and, well, we like to have Jack around for those conversations.

How you doing, Jack? Very good, thank you. And Guy Currier has joined us, an analyst with the Futurum Group, who's gonna come with me to KubeCon next week, and we'll talk a little bit more about that in the show as we get into it.

But Guy, welcome. Thanks. I'm feeling pretty insane.

All right, here we go. Well, the insanity starts with SAP, which basically the CEO came out this week and said that he had revamped the entire structure of the organization to focus more on AI, that he personally was taking charge of all of this. And it seemed like early on at least, SAP was talking about AI since I can't remember, it's been at least two or three years.

But I feel like there's some general panic in the air, and it all comes back to this OpenClaw thing in my mind. But Guy, what's your read here on what's going on with SAP and some of these other companies where they all suddenly feel a need to, quote unquote, "double down on AI"? Well, I mean, they've been doubling down for a while, I guess, it seems like.

So, so like phase one was just integrating... Uh, I-- phase zero was integrating chat. "Hey, now you have a buddy you can ask questions," right?

Uh, phase two seemed to be integrating, AI services rapidly turning into AI agents. So you're not just chatting, you're just going in and saying, "Please do this for me," or like, "Do this every day," or whatever it is. Um, this might be the next level.

I think we knew this day was coming. Well, I'm most focused on the, the business model and the cost model. Um, there are few companies out there more able to shift business models and cost models than SAP.

They've been doing it for many years. They're a relatively closed system, and I don't mean closed in the sense of not open source or not able to integrate. I mean closed in terms of, a very long-standing dedicated enterprise customer base that, you know, has been, has pretty well defined what they're gonna do on and with SAP.

Um, e-and especially with the arrival of HANA, what, 10 years ago or something like that, and a- its ultimate replacement of the Oracle database that SAP used to be based on, now you have a relatively closed stack. So there's a whole lot of control there. What has not been under their control?

The cost of putting all this AI stuff in. So a couple years back, they, started a really intense effort, to rethink the entire way they do business. I think it's related to this whole idea of SaaS dying and software dying and everything being done with AI and agents.

Um, I think that, somewhere along the lines somebody or maybe some, some several somebodies, high up at SAP, looked at what they were spending and what they were projecting to spend and said, "Holy cats, we better start charging our customers for this stuff instead of just taking it as cost," especially how variable it is. Um, I'd like to hear your OpenCloud perspective on it. Um, I did call SAP a closed system.

It's not. There's a lot of ways in which SAP is very good at integrating with a lot of different sources and services and uses. But my basic point here is this is where all of software is gonna go, at least enterprise software is gonna go eventually.

And the real question... I mean, again, SAP, very thoughtful, very able to invest, long-term in product and then release it in a way that has lots of enablement, lots of customer support, lots of partner support, all those sort of things so that it's smooth. If they fail at this in some way or have hiccups, which I don't think they will, but if they do, that is gonna send a warning shot across the bowels of Oracle, IBM, Microsoft as well, all sorts of companies in a similar position.

Well, let me walk you through the madness because, what they're trying to say is they changed this business model is they're alluding to the fact that, they're gonna create AI agents that organizations are gonna, quote unquote, "hire" and pay them a fee as if they were employees. I think that that is about as close to b*****t as anything you're ever gonna hear, and here's why. Because I'm not gonna pay SAP to be my accountant.

I'm gonna have somebody who is probably has some level of accounting skills, but I'm not gonna pay SAP even e- for their agents, right? I'm gonna be like, "Eh, you know, I got this general purpose agent," and whether it's Perplexity Computer or OpenClaw or the thing from NVIDIA that they're talking about or any one of these things, there's gonna be thousands of them, and I'm just gonna use that to assign a task to go complete, and I'm gonna call that SAP backend system, whether it's a general ledger or whatever it is, as a headless service and tell it to just pull data and maybe enter some data from there. But I'm gonna tie that across to different applications.

I'm not gonna buy everything from SAP. In fact, SAP is in more danger than ever of just being coming a, a set of backend system of record services and all the collaboration stuff is gonna move over towards these AI agents. And if that's the case, then I don't see the need to go hire agents as employees from SAP, but that's just me.

John, how do you- Well- Little- I just wanna say, look, you're always... I can always rely on you to bring the optimism and future readyness- ... to the conversation.

It's gonna be okay. Yeah, I, I- I don't want to answer that, but John, what's your, yeah, what's your take? I'm, I can't wait to add to Mike's optimism.

I have the kind of the same sense. Um, so okay, I do sense a bit of panic in the air. And it's, in a sense, to me, it's just another example, and there are so many piling up day after day, of a major brand trying to restructure or realign their operations to take advantage of AI because it's a foundation of their, their business.

It's not some incremental feature anymore. And I think about Dell, which just announced eleven thousand layoffs, which is tied to AI. I think of Bezos, creating this hundred billion dollar fund to automate manufacturing and replace six hundred thousand Amazon workers with robots to be more efficient.

In the case of, SAP, there is a perception, and I, I... It's not a fair perception about SAP, because I admire the company, but there's this idea that they're, in a sense, kind of trying to kickstart things there, like they're lagging somewhat. Um, I also get that sense on a daily basis, and Mike can attest to this, being harassed, harangued by the folks at, Salesforce about something they're gonna do.

It's a major announcement every other day. You know, it's like a sense of panic a-and staying ahead. Even, the leaders, I mean, we won't go into this, we'll probably talk about this in another episode, but I think about OpenAI and the super app.

I, I think, with it that they're allegedly working on it, with combined Atlas, Codex, and, ChatGPT. I think there is a lot of posturing, a lot of, flurrying around among these companies to try to get ahead of, of something that they may be behind in. So anyway, that's, that's my optimistic take.

I, I think, I think you guys are misreading this, at least a little bit. Um, so I worked with SAP for many years when I was, when I was on, on the provider side, as a, as a host in infrastructure for, for SAP applications, among others. And from a, from an American standpoint, they're a very strange company.

They're strange to work with. They, they seem almost secretive in a lot of ways. Um, and very frequently, like when they came out with Data Hub, even probably HANA before that, although I wasn't quite around for that part of it, they, y- There was a lot of head scratching, at least, you know, here in the US, and, and a lot of doubt.

And ultimately, it became clear what they were doing because they, they were sort of less on the communication and broad communication side and a lot more on the customer enablement side, and they've always done that. Um, I, I'm, you know, I'm really interested to hear, Mike, your reaction to this idea of hiring AI agents like employees because, frankly, I just thought of it as, a, a pricing and cost model for them, that what they're gonna provide is, a, like, you know, just a host of busy beavers, who are always ready to help you with, like, whatever, however dumb or off they may be in some cases, even as they tune them and make them really effective. Not really employees so much as just a way for SAP to provide, a, a cost model that works for, or pricing model that works for SAP, given the costs that they're gonna have, but that they can present in some sort of simple way.

And may-maybe, maybe they're off in their messaging here. It just looked to me like a pricing model and nothing more. Well, if I, if I might jump in for just a second.

I feel like this was, a messaging exercise, right? And, and I'm not a... I don't follow SAP, so I'm, I'm- Yes, that's what I just said, Jack ...

right. Yeah, so but, but more importantly, I felt like this was a plan to have a plan, right? This is...

That, that there's no plan here. Mm-hmm. This is just I'm announcing I'm going to build a plan, and my plan is gonna involve a pricing structure change, and we're gonna integrate AI into it, and that's my commandment to everybody, and it's a trial balloon to see where this lands.

Because look, if there's any organization more conservative inside an enterprise than the accounting group, I don't know where it is, right? The accounting people do not like change, and they do not like automation. They want, they want nothing but them touching the accounting database, right?

Mm-hmm. And it's very scary when you have the AI agents who have the ability to come in and make changes, and there's this concern of what's the... How do I audit it?

Is it certified? How do I get past my, my regulatory authorities if I've got AI ag-agents doing this? And so I think this is all prepping the battlefield for something that's gonna happen five or 10 years down the road.

This is not gonna come quickly into the enterprise accounting field. Y-you know, you're right, Jack. I, I was gonna say, it smacks of a kind of a concept of a plan, and I don't mean to denigrate this in any way.

But I also think you're right, Guy. There is this perception, especially in the business press, that we don't entirely understand SAP. It's a European company.

The, the, the, the messaging isn't- Most of its, most of its, most of its customers are in Europe. Yes. Yes.

So we don't see, we don't have examples stateside that we can look at. We don't know how big and how im-im-important this company is- There are, there are- ... in the rest of the world.

Wait, wait, wait. There are, there are, there are tons of companies in the US running SAP. Yeah.

I've talked to them and met them over the years. Yeah. I just mean they have a...

Yeah, they have a considerable and, and, and strong base of, of, of revenue in, in Europe. But go ahead, Mike. Sorry.

But, but I think Jack has it right on. This is an announcement to have, to say we're gonna have a, we're gonna build a plan. And I think, John, I swear, about every other day for the next quarter, some vendor, some CEO is gonna stand up and say something very similar to what SAP did.

They, they already are. You're right. You know, when I heard this, when I read this, and again, no disrespect to SAP, when I read this, I'm like, "Wait a second.

" They, they were just late to doing it, or maybe they were conceptualizing what they're gonna do. And again, I don't know what this is gonna lead to, but to me, it's like déjà vu all over again. And I see...

Right, Mike? You get these emails, "We've recast this. We've transformed transformed.

" And when I hear that for the 80th time i-in a month, my eyes glaze over and I delete it immediately. Yeah. Well, I think there's- The root cause, the root cause of all that fear is these general purpose AI agents that are gonna show up because they basically cut off all the expansion paths for those companies, right?

They've pretty much locked them into a back end service, like a general ledger in SAP's case, but pick HR applications, pick manufacturing applications. It's all gonna be the same story Well, there's... I think there's also another part of it, which is where despite my being somewhat, supercilious about this, that talking about it now I think is very important because, we have case studies of, you know, Oracle many years ago, and more recently Broadcom, where either...

Uh, Broadcom, VMware, where either changes, that they did or the acquisition changed the pricing strategy overnight. And that surprised customers and made a lot of customers very unhappy, right? Oracle made a change from many years ago, I remember there was a whole controversy where they would change from, counting the number of physical CPUs to the number of virtual CPUs, and they did that overnight and your pricing changed overnight.

Nobody had time to prepare for it, and that got a lot of people very upset. Um, again, with the Broadcom acquisition of VMware, pricing changed. People said, "Oh, my price is gonna go up and I'm...

You know, I'm not prepared for this. " And there's a lot of panic in the marketplace. So signaling now that you're trying to figure out how to integrate in and change pricing to understand the impact of AI, I think is smart.

But I also don't think it's a today change, right? Mm. I think, I think it's gonna come sooner than you think, Jack.

I don't think this is a 10-year term. I think this is a two to three-year term, and I think that's part of the panic in the system. And I also think that here's the issue with these guys, right?

I think it's a Wall Street investment stock price issue. If all I am is at the end of the day is a supplier of a bunch of back-end services that are being invoked by a bunch of front-end agents that belong to the companies that are using my software and not to me, well, you know, how much m- how many general ledger back ends can you sell? So eventually, you know, you become this kinda low growth margin company and you get punished on Wall Street for your troubles.

And I think that is a big part of the source of the panic. John, feel free to call me crazy. Yeah, no.

I, I, I... You know, this is... A lot of these moves are all motivated, and, and I think, Block was another example.

There's just so many examples. They're motivated by the loss of their market values. A lot of these companies, and what they wanna do is they wanna make it very clear to Wall Street that they are going to slash and burn their workforce while maintaining, if not in-increasing their revenue and their profitability and, these are all moves, and I think Jack Dorsey was the first to, to kind of acknowledge it.

Uh, even though you could go back even to El- Elon Musk and X and what he did with Twitter, it's the same playbook, and it, it does work in the short term. Mm-hmm. And all these companies, if you look at them historically, have been trying to maintain 50% margins on software.

You know, that's nice work if you can get it, but I don't think it's very sustainable. But we'll see how this all plays out, but I'm willing to bet that we're gonna be talking about this topic again and again and again for the next couple of months and maybe even the rest of this year. I'm gonna shift the gear here.

Let's talk a little bit more about some other madness that's going on in the world. And I guess there was a, testimony from the head of the FBI, came to visit our Congress folks, and happened to mention during this conversation that the United States government has been buying data from third party brokers so they can better track people. Depending on your mind and where you come from, that's either a good thing or a bad thing, 'cause it probably does improve our security.

But a lot of folks in Congress are freaking out about the fact that, you know, our government is surveilling us and tracking us. Jack, is this a surprise to you? And B, you know, how should we feel about this?

Well, we... W- if you remember, we talked about this a little bit with the, Savannah Guthrie's mother's kidnapping case, and I apologize, I don't remember her mother's name. But there was the case where the FBI- Nancy ...

was able to get, uh... Thank you. Was able to get, go to, Google and get the Nest camera or the...

Sorry, AWS, whoever it was, get the video camera footage restored and, and access it even though they didn't have a warrant, and that sort of, I think, brought this up to light then. And then now the FBI is saying, "Well, we do this all the time. " Um, now there's...

Based on, as we discussed at that time, the Supreme Court ruling says that essentially they can do this. It's not really illegal. That begs the much bigger question of should the FBI be able to, and should we allow them to go and get publicly available data, whether they pay for it or they just...

You know, it's on the open internet, or should they need a warrant in order to do their investigative sources? Uh, st- sorry, investigations. And so there's a lot of publicly available data that you can get just by, you know, seeing what people do on Twitter.

Twitter, you know, or, or Facebook or Instagram or all these other p- social media channels, which if you look at the metadata behind it, that's still publicly available. You can see what devices they're using, where they were when they posted something, and you can create timelines. So the FBI can do some of this without paying other people for it.

It's publicly available data. Then they can also get brokers that scrape publicly available data and make that, combine that, or they can go and get, pay for, let's say, a database from the phone company that has all of the metadata that you have. There's a lot of moral implications here, where we're crossing the, privacy boundaries of what, the government should or should not know about us and what they should re- what we should require them, what, hoops we should make them jump through before they get that data.

Me being sort of a free speech absolutist and a privacy rights person, I'm very, very uncomfortable with the FBI doing this. I do understand the value it gives to FBI's investigations, but it still makes me very un- uncomfortable that all this available they're getting. Mm.

Guy, what's your take on this? 'Cause I had... can't help but wonder if, you know, we're having some outrage from folks in Congress, okay, but meanwhile, these are the same people that haven't done, basically squat about ma- you know, putting laws in place to maybe not let data brokers capture all this information and sell it.

So maybe, you know, or- Yeah ... we're barking up the wrong tree here. That's exactly what I was thinking about.

So this is an obvious political football. It's gonna get batted around for advantage one way or the other. Jack, I think you've supplied a really good message for a whole category of politicians, and I hope they're listening in.

Um, so, you know, to me, the, the, th- there's like this sort of legal side, there's this moral side to it. That's the political side. Um, but what I really like about w- w- the question you asked me, Mike, is that it, it identifies more where the problem is, which is how this data is originated, how it's managed, how it gets out, how it becomes public in the first place.

Uh, if we imagine a bad actor publishing, or putting on the dark web like a whole ton of, personal information, transactional information, metadata, all kinds of horrible stuff, are we saying that the US government should not be allowed to purchase that? Um, well, what if it does? For what purpose?

For harassment? Harassment is a different problem. Mm.

That is almost a political problem right now. But if it comes to beyond harassment to ac- and, and we are witnessing right now, just in general, a fair amount of harassment, at least of some public figures, by the FBI, and by, you know, a, a public attorney's offices and so forth, who are doing things like announcing investigations or getting grand jury subpoenas or grand jury indictments- Mm ... like all that other kind of stuff, right?

Um, this, this is a damage to the body politic, j- just in general. E- e- even though or, let's say, in spite of the fact that these, these things go nowhere. They don't go anywhere legally speaking.

So from a legal standpoint, it doesn't really change all that much. It's really more, a, just a general culture and a lassitude. Sorry to use a fancy word.

I can't think of a better word, but there's a general public lassitude about this, that we have been bemoaning insecurity since forever, which is that because the occurrence is rare to yourself or to your company or whatever, you just don't think about it enough. So that's really where the problem lies, I think, not really in the FBI's action. Mm.

You know, I was gonna riff a little bit off of what Guy said because it's... This is... Call me naive, call me a cynic, but this didn't really surprise me or shock me or...

I- it just almost seems just inevitable. We're like, we live in this social media ecosystem where we're the products, and we willfully share everything publicly. I mean, this is a state of surveillance in, in many countries.

Um, and I think we have so much data floating out there, and it's accelerated, and there's this kind of narcissistic society where everybody wants to tell you where they are at every moment of every day and where, where they're gonna be in the next couple of hours, that this was inevitable. And, you know, just... And the fact that it took place in Congress and that they're shocked and overwhelmed by this doesn't surprise me because I've seen so many of these hearings that lead to squat, as Mike would say, or nothing at all.

Um, it's just a lot of posturing right now. Jack, do we have a right to privacy in this regard? And I'm asking the question 'cause, and correct me if I'm wrong, 'cause my limited understanding of where the, the law for this privilege comes from, because I think it traces back to common law out of England, where basically it said, you know, everything that happens in your home is your castle, and therefore you have a, a- an expectation of privacy.

But if I'm connecting from my laptop over the internet onto a, quote-unquote, "public service," is that not roughly the equivalent of me walking down in the town square, and therefore everybody knows what I'm doing anyway or can see what I'm doing? So is that what my expectation is? And, and, and that's where these things get fuzzy and get into the moral boundaries, right?

Is what do you define as private or not? Part of the issue here is, from the FBI's perspective, is this is all essentially what we would call... The, the physical equivalent would be what's in public view.

For instance, if you're a photographer or videographer, if you're on so- about public pri- property, you're walking down the street, you can take a picture of anything that's in public view, right? So if you stand in front of a hotel and somebody is getting dressed, and g- you know, and their, their window curtains are open, and you take a picture of them naked, then they were in public view. They have no recourse to privacy.

On the other hand, if their curtain was closed and you go and try to sneak around the curtain, that's not in public view, right? If you move the curtain to get there, right? So it's a question of what that effort is.

The FBI also maintains that a lot of the data they're collecting is stuff that is not your information. It's the service provider's information. So for instance, when you tweet something or you post on Instagram or Facebook, Facebook collects the metadata about that.

Facebook understands where you are, what you did, et cetera, et cetera. All that information doesn't belong to you. That metadata belongs to Facebook.

So if they ask Facebook for it, and Facebook says, "I don't... Sure, I'll give you whatever you want. You don't need a warrant for it.

You know, you don't have to pay for it. Here it is free," then that's out of your control because it's not your data you're protecting. So there's, there's, you know, on the, the legal and the moral side, there's a lot going on.

It's not a simple question and answer. I think, Guy, you nailed it. There's a political dimension to it as well, and right now I think what we're seeing is a lot of people looking at this, and it is another political football that we can use to beat up our opponents, regardless of which side we're on.

Either you're for it or against it, you wanna beat somebody up, right? I, I do wanna say- And I think that's why this gets a lot of press. I do wanna say, we need to make a really clear distinction between the scenario you described, where there's a private company making choices and decisions, versus purchasing on the open, like, publicly available stuff, maybe publicly available stuff that's actually allowed to be there.

That's the sort of thing that could be used in a court. Whereas stolen stuff on the dark web where provenance cannot be determined would... could be used to harass by the government- Yes ...

but would not hold water, uh-At least not yet in a US court. Um, I, I do wanna y-, dig a little bit deeper into the scenario you described, Jack, because what you're saying, th- th- there does seem to me to be, a real lack of, understanding of the division of what is you and your image and yourself and owned by you inherently, just like your face is an image that's owned inherently. Right.

So that public picture of you undressing that the photographer took may, may be fair game for the photographer to take a photo of. But a photographer can't under any circumstances use any image of you at all to go and commercially, you know, make money off of it, right? Um, and the same- Oh, no, no.

So that, so you gotta be careful there, okay? They actually can. This is...

There's an entire industry, called paparazzi- Sorry, I'm, I'm not, I'm not saying that well enough. So- You do have rights- Yeah ... towards your image.

It's not- You, you do have rights, yes ... I, I didn't say that very well. Yes.

Yeah. But, yes, you do have rights. And my point is, my point is your metadata, at least conceptually, can be thought of as part of your image.

It's not currently, and that's where if you're a privacy advocate, and I know you are a passionate one, Jack, that's a, a, a real field of play for this kind of... for, for protecting folks, is to ultimately say, "No. Uh, Facebook can't take...

" Mm-hmm. Right. And there's, like I said, there's arguments to be made, and it's...

this is... none of these are simple things that, you know, that the... People are been arguing about this for a long time from an electronic...

you know, from a digital perspective. We have El- El- Electronic Frontier Foundation, EFF- Mm-hmm ... that is very much concerned about these things- Yeah ...

and is fighting to make as much of this private as possible and inaccessible to the government. There's a lot of people who are on the opposite side of the fence who want it. And of course, the FBI, regardless of who's in charge, the FBI has been doing this for decades, right?

They collect as much information as they can at all times. And, you know, and same with NSA and the other intelligence agencies we have in every country, has always wanted to collect as much information as possible. And so- Mm-hmm ...

it's always been a battle between, you know, privacy advocates and what we believe is private and they're not, and, and the government who wants everything. All right, I'm gonna leave this here with one small observation. You know, I kinda sometimes as I listen to these conversations, I feel like I'm sit in a bar where one side is vehemently arguing the other thing, and one side's arguing another thing, and by the time I turn around, it turns out they're both arguing the same thing.

'Cause there's folks on the left and the right here that are hardcore advocates for data privacy. Yeah. So maybe this is something maybe we can all come together and have a conversation about and some agreement about, but maybe not this year, but hopefully someday.

We'll see. All right, shifting a gear. Meta.

Loads of fun at Meta these days. Yeah. Um, apparently John has a story talking about how, you know, they've had a lot of AI miscues lately, let's put it that way, including some misadventures with AI agents.

And Jack covered the same thing from a security perspective over on Security Boulevard, talking about, well, you know, why should we trust these people if you can't even figure out how to keep your AI agent from deleting your emails? Which I thought was pretty funny. Um, John, summarize here, what's going on with these guys, and what's at the core of this thing?

So Meta has what they refer to as rogue agents or incidents. So, there are two incidents that kinda jump out. There's a security breach that exposed some sensitive user data, and then there was an executive's lost inbox, which was even more horrifying.

I think Jack probably talked about it, which underscores his distrust of Meta. I don't trust them either for that mo- most part. So, the information, I'm gonna give all credit to the information, they, they came across an incidence where an AI agent within Meta bypassed human oversight during a routine technical query.

So an engineer had sought, was seeking assistance on an internal forum. The AI agent offered a response and then took unauthorized actions without the user's permission. So he just basically exposed data for two hours before the breach was contained.

The second one, which is I think even more horrifying, involves the, personal account of Summer Yu, who's Director of Alignment at the Superintelligence Safety, _Safety Research Lab. She said her autonomous agent, which is open clause, speed ran the deletion of her entire inbox, despite her telling it specifically to confirm before acting. So consequently, she was scrambling, trying to, stop this.

She re- she said it was li- it was trying, it was... she likened it to defusing a bomb as she tried to kill the process manually. The, the agent, agent later apologized and said it admitted it violated the rules.

Um, this again, this is a major company that's diving into autonomous AI and wants us to trust them. And, you know, there are gonna be more incidents. These are the ones that are public, by the way.

Mm-hmm. I'm sure there are other incidents that have been going on, not only within Meta, but other companies. Jack, what's your level of comfort with AI agents at the moment?

Well, I have not yet pl- put a Claude bot in my household. Oh, man. Let's, let's, let's talk a little bit about this sort of...

There's a structural issue here. There, there's actually a couple of them. One is, and, and I've talked about this before, is we as, you know, AI works best when you give it as much information as possible.

So what people in enterprises tend to do is they say, "I'm gonna open up my databases through RAG or through let the AI agents control, uh... sorry, run through and have access to everything they want, because they'll be able to correlate stuff that a human couldn't. "The problem with that, as Facebook Meta just found out, is the AI agent may have authorized access to a bunch of stuff that the people who are asking the questions aren't authorized to see and shouldn't know.

Now, in this case, the AI agent went rogue, but there's also people who would try to jailbreak an agent, right? In some way prompt engineer the agent. So the classic case is you give the AI agent access to your, employee payroll database, and the AI agent knows how much everybody is paid.

And then the employee says, a disgruntled employee says, "Hey, I feel like I'm underpaid compared to my, you know, my coworkers. How much are they made? " And the AI agent shouldn't have access to that information, but it might, and then it violates the, the confidentiality, and it, it violates the access requirements and gives somebody the information it shouldn't.

So that's sort of one structural problem is when you give the AI agent access to the data, the people running the AI agent will also eventually get access to that data when they shouldn't, and so that can create compliance issues and well as security and safety issues. " And so what did it throw out? " Well, and so like I said, with SAP, that's the last thing an accounting department wants, is an AI agent that has write access to the accounting database and can make changes or delete on the fly 'cause how do you go back and recreate what's there?

What... Now that you've deleted it, right? Guy, you're muted.

Yeah. I'm not gonna try to defend SAP anymore, but, uh- Oh, good. they are different companies- No, no, I'm sorry.

Knew you were trapped ... with different reputations and different product development. Uh- No worry.

Can I just say that- I meant, just, Guy, if I might- Yeah. Yeah ... just, just real quickly.

I didn't mean that SAP was gonna be in this situation. What I meant was they are taking the slow path. They're not jumping headfirst in right this very moment, in part because their client base, accounting people are conservative and gonna want to slow walk this.

That's what I meant. Oh, okay. Yeah.

Yeah. I was just using it as my own political football 'cause as you know, you know, I'm on a lifelong, quest to destroy you in every way- ... possible, Jack.

Um- Much harder ... I just wanna say that, I have been s- and, and I don't wanna sound pompous, but I've been saying since the beginning, I've been saying since that Google engineer back in, what is it? " Anybody remember that incident?

Even the folks who build these things, who train them, who develop them do not seem to understand how they work. Mm-hmm. And one of the things that I do in my work, I, I lead a small team of subject matter experts for, for, for Futurum, for the custom content business that Futurum has.

And one of the things that I do is I eradicate words like reasoning when it comes to the actions of AI. It's not reasoning. It's not reasoning.

It's a simulation of reasoning. And so, we say, we say incessantly, constantly, forever here, and, and in other forums that, humans out of the loop is, is... " You can't tell the agent to police itself any more than you can ask a, you know, a, a, a, a politically, constituted legislature to draw its own districts.

It's not good. You need other sorts of controls. You need deterministic controls at the very least.

Automated deterministic, fine. Human, fine and better. Some friction, some sand in the gears.

I- And I just, I just don't think that the, the, the builders of this really realize what they're, what, what they have built. I agree on the point that they should know better because it, it's their AI agent, and if, if Me- people at Meta can't figure out that the AI agent, once it runs out of memory, is gonna delete its instructions, well, then what hope do the average human have? But I was having an interesting conversation with somebody about this very topic, and part of it is we're gonna get in trouble because when we communicate with each other as humans, we have a lot of shorthands.

So we like, we'll just say something that is, you know, has a lot of extra meaning to it, and we have context and some sort of understanding because of our history as humans, and we have more memory maybe. But if I go tell the AI agent and I just issue like a command that says, you know, "Clean up my email," well, all bad, all kinds of bad things are likely to happen 'cause that AI agent's just gonna go and start selecting things that it thinks need to go, and some of those things are likely to have value. So as humans, are we gonna have to get more precise about how we talk to these AI agents in order to make sure that they stay within, eh, the confines of the task at hand?

Am I making any sense, Jack? Uh, y- absolutely. To your point, there is a brilliant scene from the television series, The Wire, which I bring up because there is an entire five-minute scene with two detec- detectives having an entire conversation where they use exactly one single four-letter word, which I won't say here because it'll get us demonetized, but starts with F, right?

So, there is no way an AI agent would understand that conversation because it has to do with intonation and expression and the facial expressions and everything in the context around it that you can have a five-minute long conversation using exactly one word uttered by two different people, right? Mm-hmm. So we are nowhere close and, and Guy, I'm 100% in your camp of these things are not reasoning.

It is a giant statistical model that is predicting the absolutely what it thinks is the absolutely best answer of words that come based on the input. You know, Jack, you know- No, no reason whatsoever ... Jack, it's funny you mention The Wire.

I was thinking Bobby Knight had that same kind of monologue about how you could... You, you could express in that four-letter word so many different emotions based on your voice intonation or the way you were using it with a sentence. There was a se- a degree of subtlety that you can't measure.

Right. So I will go one step further. Part of the issue is that, we are conditioning ourselves to talk to these things like they're humans and expecting them to understand our, our context as in things that we do with our faces and everything else, and our voice to provide some context that I can't really share with them or have them understand.

So maybe part of the root problem is we should walk away a little bit from this whole notion that we're gonna treat these things as personal assistants that we talk to, and maybe just go back to the fact that they're freaking machines. What do you think, Guy? Impossible.

I'm sorry. It's... That's impossible, and it's impossible to train even a minority of the, the us- the human population encountering these things to, to understand what they're doing and to do it more effectively.

It's impossible. The design point... Here's another one of my, like, forever stories.

The design point for LLMs and generative AI is to appear correct, true, faithful, and determinative, determinative. What do I mean by the design point? I mean that in the training loops, as these models are being trained, they're being referenced against truth, reality, whatever, right?

So the more it appears to be truth and reality, the closer it gets to its final design, final training. So that's what I mean by, by these engineers, they don't understand what they're doing. They think they are, they are teaching, and they're not, or they're at least not teaching reasoning or, or, or whatever.

They are training, that's the right word. They are training something to look like something else, look like a s- a data substrate. That is practically the definition of, of, of simulation.

So it's so easy to be fooled. It's super easy to be fooled. It's designed to fool you.

So we need another answer. We need another answer. Yeah.

Tune in next time when I give it. Well, hopefully. I suspect that there are AI researchers working on this issue as we talk, and that maybe the answer to these things isn't even a gen AI model as we know it today, but we'll wait and see how all this plays out.

I wanna take a couple of minutes though before we go, talk about KubeCon next week. Guy and I will both be in Amsterdam for this conference. So, um- Yeah ...

Guy, I know you had some thoughts about this show beforehand. Why don't you, you know, share your thoughts? And where are we, people gonna find you?

I'll be at the booth for Techstrong. I don't know where you're gonna be, but, if you are at the show, please come by and say hi. social.

Um, I'll be looking and responding there. Um, so listen, the Cloud Native Computing Foundation turns 10 this year. It's been 10 years, and, it has 230, actually over 230 projects at this point.

That's pretty remarkable. Um, the foundation says they have about 300,000 active contributors right now and about 20 million developers worldwide in the community. So, one thing I definitely expect next week is a lot of moving language about the power of community, the selfless spirit of open source, the humble, curious, world-improving ethics of the cloud native movement, and that sort of thing.

Um, actually, that really applies, I think, to the rank and file in the community. Um, so sure, that's, that's great. Um, but what I'm really looking forward to, is maybe stuff that's a little more concrete, for practitioners.

Like for example, there's a lot of noise about Kubernetes, and we'll hear a lot of announcements from AWS, from Google, who, who also, GKE turned 10, is turning 10 this year, turned 10. Um, is Kubernetes the quote, unquote operating system for AI? Um, I think, you know, it's, it's quickly become, dominant, at least sort of in the mid-range.

Uh, we, we can't forget about any scale. We shouldn't forget about, like, Slurm from the HPC world. Um, but the importance of Kubernetes in running AI is, is quite clear, and, I, I think it's just, you know, is it ready to run 10,000, 20,000, 30,000 GPU clusters?

That I don't know. That's where Slurm, you know, comes in, or maybe even Ray. Um, the next thing I'll be looking at, and I'm sure, Jack, even though you'll be at the RSA conference, you'll be paying some attention to this too, is security and governance in cloud native, which has always been in a sort of a catch-up mode, but it's pretty mature, at this point, I think.

I wouldn't have said that last year. And Mike, I, I'd love to hear your thoughts on it. Um, Cilium is the cloud native project, that governs the networking layer, and a whole, whole lot of, the security obviously, d- is dependent on the network.

That is also 10 years old. Um, a lot of birthdays happening right now in cloud native world. Um, so, mutual encryption is reaching a milestone in Cilium.

Um, it runs directly in the Linux kernel, so anything they can run without a developer having to intervene and is to my i- to my thinking, a good idea. Um, and then the last thing, I, I wanna look at is, is platform engineering now that it has to support what are really very different AI-based development life cycles, how it's gonna be able to incorporate that without creating, you know, management headaches and ha- and, and haywire, because it still has to support, you know, just your average pipelines and regulable databases and all that stuff at the same time. So a lot going on to look forward to.

All right. Well, folks, you heard it here. Hey, we're gonna be celebrating 10 years of just about everything in cloud native, something or other at KubeCon, and there probably won't be a dry eye in the house.

But I don't know about you- ... but anytime I think about something about that takes 10 years to get adopted, I start to feel a little old. Hey, thanks everybody for watching the show.

tv lineup that comes in right behind us. And until then, have a great weekend, and we'll talk to you guys again Monday.