Techstrong TV March 19, 2026

Hey, everyone. Welcome back here to techstrong TV. I've got two people to introduce you to today.

It's always a pleasure when we have two people on. Let me first intr- introduce you to Carole Winquist. Carole is, the CMO at GitGuardian.

And Carole, welcome to techstrong TV. It's great to have you making your first appearance here on techstrong TV. It's great to have you.

Thank you for inviting me, Alan. Yes, so I'm, as you said, the CMO of GitGuardian. I've been with the company five years, so I've seen the growth from, pre-Series B company to now a post-Series C, 'cause we just announced it.

So thank you for having me. Yes, and congratulations on that. You know what, Carole, be- let me introduce Dwayne, then I'm gonna come back to you a little bit though.

Um, also joining us is Dwayne McDaniel. Dwayne, welcome to techstrong TV. This is your first time on, since we have two first-timers.

Welcome. How are you? I'm doing great.

Thanks very much for having me. Uh, yes, I'm Dwayne McDaniel. I'm Principal Developer Advocate at GitGuardian.

I've been here about three and a half years. Uh, and I absolutely love engaging with the community. Uh, if you go to our blog or our YouTube, you'll see my face and my name over there a, a lot.

And I love keeping up with the trends out there, and I'm very excited to talk to you about our annual report today. Excellent. Before we do, though, we've got some other stuff to talk about.

Carole, you say you're at GitGuardian now about five years. Give people a sense of, you, you know, what'd you do before that? What'd you do, you know, six years ago?

What did... Give us a sense of your path, your journey to- Um- GitGuardian ... my journey is, being a CMO in, B2B enterprise SaaS software for too many years.

More than you want to admit. Yeah, exactly. And, before that, I was working in, business process management.

Uh, and I came to cybersecurity with GitGuardian, and it has been, an awesome learning, curve, I should say, and I love it. Uh, the product, is really, amazing and brings value to the community and to make a, a safer world. So it's a bit, yeah, quirky sometimes, but yes, we do our, our best to help.

So I'm- You know, but, but that's... And I've been in cybersecurity myself for twenty-five, thirty years, and people sometimes lose sight of that's why a lot of peop- a lot of us are in here is it's, there is a bit of a feel good. It's frustrating because progress is slow.

And you know the thing about security, right? When nothing happens, you did your job. But that being said, knowing that you're, you're on the good side, on the right side, is, is a good thing.

Um, I'd love to talk to you about your feelings around business process and SaaS companies with AI, but we'll save that for another interview. We got stuff to talk about today, but... And welcome.

Dwayne, let's hear... You're here about three and a half years, you said, at GitGuardian. Let's hear a little bit of your career arc.

Oh, believe it or not, Carole and I used to work for the same business process management- Really? years ago. Uh- Okay ...

but we did not at the same time. We didn't actually meet until I started- Oh, really? working here at GitGuardian.

And, just like Carole, this is my first foray into properly working in the security realm. I'd been talking about security off and on for the last decade of my life in developer relations across platforms and Git tooling. Uh, in fact, I first mentioned GitGuardian, after their first report because I started including them in a talk about Git hooks and keeping secrets out of your source code when you commit it.

Uh, and now I do that professionally, so I'm very, very proud to do that. Good for you. That's great.

What a great story. So you guys were both at the same company at different times. Yes.

They didn't know each other, came here and met. Yeah, exactly. Sounds like a romantic comedy or something.

A rom-com. But anyway, so look, you both have been excited to come to GitGuardian. GitGuardian is a...

I, like I said, I've been in security a long time. GitGuardian's a company that we've, you know, had seen the name, a lot of us. I bet a lot of our audience, being security people, have seen, you know, or heard of GitGuardian.

But I don't, you know, I always wonder, I always worry that people have heard a name doesn't mean they know the company, right? So Carole, as the CMO, I'm sure you've spent a lot of time in the exercise of, you know, who we are, what's our brand mean, what, what is, what is GitGuardian? So I'm gonna ask you, if you don't mind, to carry this one and s- give people in our audience, who is GitGuardian, right?

What, what does GitGuardian do? And, and can I brag a little bit, Alan? Oh, go.

Uh, that's my role as a CMO, right? So I, I, I should say, I, I can say we are the global leader of secret security and non-human identity security. We've been doing this since 2017, and if you ask us about secrets, I think we know, we know it all, right?

We, we, we've done it. We've taken the challenge to make sure company, can basically know where their secrets are. The one that are, you know, well managed in vaults but have maybe some problematics with their hygiene being long-lived or, over-permissioned, but also all the secrets that sprawl everywhere because they are hard-coded in code.

Everybody, I think, knows that that's happening, but also in other tools like Jira, Confluence, Slack. And, and this is a big problem as, as most of your audience should know, I think. And this morning, again, there was a leak, with the SalesLoft, hack-And Telus has been breached because, you know, the, the attackers found some credentials when they breached SailLoft, and now they're using it to breach into Telus and move laterally and find more secrets and enumerate secrets all over the place and just hack again another company.

So I think show-- seeing all these breached where secrets are the initial, element that help hackers enter but also move laterally, that's why we, we are, I think, bringing a real solution to the market. And, and not just because we help detect everything, right? Because you can detect, but what do you do, right, if you, if you don't have action plans?

So, we, we like to say that basically, GitGuardian really help closes the, the all these incident at scale. So we, we, we sell into very large company, but we also offer the solution for individual developers or, or company below twenty-five for free because that's our Good Samaritan, approach to the market. But for large and complex companies, we, we bring a solution that not only enumerates and, and shows the secrets, but also help remediate and put into action so that at some point there is a real management of, this problematics of secrets within the company.

Excellent. Very good. Dwayne, I'm gonna give you-- Carole was pretty, you know, covered it from A to Z, but anything you wanna add that, you know...

In your role, you're out here talking to people a lot, right? Evangelizing, listening. A lot of people think evangelists talk a lot, but a good evangelist listens a lot, right?

What are you hearing? What are people saying to you about GitGuardian? Well, pe- one, people really appreciate the fact that we're free for open source projects and for individual developers.

It's an enterprise-grade tool that anybody can use. Uh, the biggest benefit, I think, to the community is that Good Samaritan program, that Carole mentioned. Uh, so we scan every new commit that hits GitHub public, every single one.

Um, there have never been more. Um, last year, in fact, m- not to preview the, the report numbers, but a forty-three percent increase in commits scanned last year. That's just that more, much more happening on GitHub.

And if we find a secret, we email the committer right then and there. It's a fully automated system. We've sent out more emails last year than we ever have before, to, again, to kinda tease the numbers coming up.

Um, but that is the biggest benefit and why most people know us. They've gotten an email, and you see it out there on social media. People say, "Just got an email from GitGuardian.

" Some people get angry that, you know, they leaked the secret in the first place, not at us, but, just the fact that it exists. And, and that's how most people first find out about us. And then when they realize, hey, that same powerful engine that we use to detect secrets, not just the ones with the prefixes, not just the ones that match a, a regex, but contextually does this high entropy string allow access to something, you can hook that up to literally anything with text, be it your developer machine, be it your production environments, and now because we integrate with the vault systems and the identity managers, literally anything inside your ecosystem.

And once you have that true visibility into the access mechanisms, backfilling the rest of the information and finding out what state it's in, giving you a way to actually tell if you are governing your secrets at scale, that-that's what we do, and that's what people know us for out there. Excellent. Thanks, Dwayne.

That was great too. All right. com is the website, and that's probably the best and easiest way to kinda on-ramp onto Git, whether you're looking as it, at it from a net or onto GitGuardian, excuse me.

Whether you're looking at it as an enterprise potential customer or even just, as you say, Dwayne, an individual or open source project who, who wants to take advantage of, of the free offering here. Um, Dwayne, you also made reference to this survey or annual survey report which recently came out. Uh, since you mentioned it, I'm gonna ask you to kick it off.

W- tell us about it. So, th- that same engine that reports on if you leaked a secret or not, well, we obviously are collecting that data, and we've been doing this now... This is our fifth annual report.

So we look over the course of a calendar year what has happened on GitHub. Uh, this year, I'll just say the giant number, we found twenty million six hundred and forty-nine thousand secrets added to GitHub just in the year twenty twenty-five. That's not cumulative.

That's not the total that's on GitHub. It's much more than that. Uh, this is a thirty-four percent increase year over year.

Last year, re- revised numbers from last year, is down, twenty-one million added in twenty twenty-four, eighteen million in twenty twenty-three. Uh, go back to the twenty twenty-one numbers using the exact same methodology, exact same scans, 'cause, you know, public GitHub and our records of it, eleven million. Uh, so overall, we've seen a hundred and fifty-two percent growth of secret leaks.

Now, to put that in perspective, that's not all just, hey, this is all new devs. Uh, active devs in that same span has increased ninety-eight percent on GitHub. Hmm.

It's-- Last year was the biggest increase ever on GitHub's platform of new developers, amount of code we're pushing, amount of everything we're pushing. But that's, like, just part of the story. Um, what we're pushing is changing.

The, the complexity of the code, the complexity of all of these non-human identities, these workloads that need to work together and authenticate back and forth to make up our apps, to make up our pipelines, to make up-Literally all the ecosystem in tech, it's just never been this complex, and there's never been more people doing it at the same time. It's kind of this perfect storm of... Because we're using AI, just first time it's come up this conversation, we're using AI so much, and AI is trained so much on existing code.

I would say it's, it's built on the, back of code on the internet, and I don't know if you've ever seen the internet, but it's kind of terrible. We did it kind of wrong. " And we think that's a good chunk of why we're seeing that increase.

And you see people like me pushing code, right? So you have now in my team, in my- Well, now y-you just hit it on the head. I...

So yes, it's, it's frustrating, I think, to see that... L-let me back up. Let me give you the good news.

There is an ever-growing minority of users out there who are getting savvy about secrets, right? Through GitGuardian's efforts and, and so forth. But at the same time, we are being overwhelmed.

We're seeing this in AppSec with the amount of vulnerabilities or potential vulnerabilities being found by using AI scanning. We're seeing it in places like GitHub and other places where we went from a universe of maybe, maybe forty to fifty million people who code in the whole world. I think that's about...

" 'Cause the whole concept of who, who is coding and what is a, what does being a coder or a developer actually mean has changed. Carole, when you and I are developers, and I, I was busy this morning on, you know... Don't even ask.

But, it amazes me because I never considered myself anything more than some hacker, and, and not even a hacker. I'm a business guy, but you can ma- Anybody can create anything today. So, but the repercussions of that is, Dwayne, exactly what you're describing.

We're seeing the amount of code being generated exponentially go up, and with that not, you know, the amount of secrets going up, the amount of, I'm afraid to say it, but probably the amount of vulnerabilities and insecurities out there are also multiplying. 'Cause as much as we... Look at, at enterprise levels, I guess we're scanning.

We're using tools like GitGuardian for secrets. We're using AppSec tools to, to look at our code. But when you, when you ten-x the amount of people developing code- Yeah ...

this is what you're dealing. I mean, I, I don't know another, you know... As much as I'd like to sugarcoat it, there's no sugarcoating it here.

So- And, and we're not going to stop it, right? So we- Oh, yeah. What are you gonna lay down in front of the railroad tracks and hope the- Exactly.

So- That's crazy. I'm starving ... we just need to help.

Yeah. And we'll- We need to help the enterprise. Yeah, we need to help, and we need...

And not only... I mean, helping the enterprise is great. It keeps the lights on, right?

But even helping these open source projects, even helping the army of, let's call them, civilian developers, right? Who are d- now finding themselves, you know... So not from the IT department even, the people in the HR department, in the marketing department, who are making their own apps all of a sudden because they can.

They need help too, and it's almost, you know, it's the shadow AI thing where the IT team who may have GitGuardian available doesn't even know that Carole and her people just made a quick little app for some little project they're doing, and they're running it, you know, on some Mac Mini somewhere or something like, like a lot of people are doing. It's a crazy time. It's a crazy time we're living in.

So Dwayne, you, you mentioned some great metrics here. I always like to ask people when we, when we get into talking about reports, what wasn't on your bingo card? What, what didn't you see coming?

I, I intellectually kind of thought we were gonna see a giant massive spike in AI services and whatnot, but what actually rolled out even shocked me. Uh, the... It's not just that we're developing code with AI, we're never building, building more stuff with AI, the AI infrastructure.

Uh, as y-you know, with... When you're building AI, the cost of the tokens is just one part of your ecosystem. It's just one part of the total equation.

Um, there's-- I had never... Before we started putting this report together, I had heard the name OpenRouter, but I am not an AI developer. I am not building, with multi models and, like, bouncing around models trying to find what's the best model for this use case.

Um, but OpenRouter, we saw a forty-eight x increase in number of leaks from twenty twenty-four to twenty twenty-five data. Uh- Really? It...

Forty-eight x. It's the fastest-growing detector I think we've ever seen, I'm pretty sure that we've ever seen across any report. Um, and then you look down that list, and the next fastest-growing was DeepSeek.

Next after that was Brave Search API, which is the go-to default MCP server for so many projects out there. So we started seeing this pattern of it's not just we're building more code. What we're building has literally shifted.

Now-The good and bad news of that is it's not bringing a whole brand-new branch of vulnerabilities. These are the same vulnerabilities. The OWASP Top 10 is still the OWASP Top 10.

All the other security reports you read of, like, what we're doing with AI, it's still misconfiguration. It's still leaked credentials. It's still broken access.

It's still all the other problem- It's more of it ... vulnerabilities. Yeah.

We're, we're- A lot more of it. We're making these mistakes a lot faster now, and there's just more people making them. And we're building- Yeah ...

making it with different technology, slightly different technology. So let me be the optimist. Look, we're, we're seeing a lot more of it 'cause AI is, is developing a lot more of it, right?

This is AI generated. But we're also... Part of it may be also that we're using AI to discover it, and our ability to scan and find these things with AI is, is, it's not part of the problem 'cause it's not a problem.

It's never a problem finding more vulnerabilities or more, you know, potential bugs or security, it, you know, ways. But it, it's overwhelmed. It's overwhelming.

I, I wrote an article on this, like, earlier this week. It seems like two weeks ago already. But it's overwhelming, right?

So, like, the developer, professional developers now, it's not so much that they're actually coding anymore. They're, they're managing. The-- it's about governance, right?

Governance of, of, of all this code, you know, securing all this code is overwhelming our abilities, our resources. We, we just c- you know, you can't, you can't 10X the amount of code that you're putting in without 10X-ing the governance or, you know, be really efficient- Yeah ... 5X-ing the govern- w-without, you know, some big increase.

And, and we leverage this, I think. I mean, yes, y-you're talking about, we are talking about the risk that AI brings, but AI is also on the side of the defenders, right? We, we use it to improve our products.

Uh, at GitGuardian, for example, we use machine learning since quite a long time to basically, fine-tune the model so that it detects better, but also help on severity scoring, on contextualization, on analyzing the code so that the AppSec team, they, they can make decision, on severity based on actual facts and not random just, checkers or just a, a little bit of info. And we know there are fewer, far fewer AppSec people than developers, and we need to help them. We need to help them figure out the 10 secrets that really, really, really are the hottest, most dangerous one first, and then they can go to the 10 next and the 10 next, right?

So... And some of them, some of the, the, the secrets can be just closed because, I don't know, they, they, they're not valid anymore, and they are on a test environment, so let's just, kill the incident. So yeah, that's how we use AI on our side.

Um, of course, we, we, we, we leverage this, all the time because we have to go fast too to, to help, and, and not only, helping on, you know, the environment, the software development life cycle, but we are now moving also towards the, the laptop of the developer because, you probably heard of the Shyulude and, and all these new, attacks we've had, in the past six months where the, actually the, the endpoint, the, the laptop is the target because hackers know there's a, a realm of, of secrets, and they can harvest, and they can, then use them for further attacks. So we, we are now protecting this endpoint too, because we are a strong believer then, you know, the developer, the AppSec team, the IM team need to work together to actually fix the problem. Yeah.

No, it's forcing us. It's f- look, you need AI to beat AI at some level, right? Exactly.

Yeah. But it's also... It, it's like the snake that swallowed the rat, right?

This rat has to work its way through from the developer to the, to the tester to the AppSec person to the deployment to the SRE. We're gonna see this disruption wave, if you will, just, you know, work its way through the, the whole process, and it may not be a bad thing. You know?

It, it, it could work. I, I definitely see a reckoning happening out there. Uh, and one other fact to kind of put this in perspective.

So part of our platform, we, we do validation. We do, non-intrusive calls to see if the thing works. And we took a subset of our findings from 2022.

We did this in the 2025 report, so 2024 data. Took the 2022 numbers, like, about 11,000, and we said, "Hey, these were valid in 2022. " And 70% roughly came back as still valid.

Wow. And we did that again this year, same data set, 64% come back valid. Uh, secrets that we found valid across 2025, end of year, 77% stayed valid.

Like, w-we're talking about pre-AI boom, or recurrent, the AI development boom. So this is a known problem. This points to that larger issue of governance.

But now with attacks shifting the way they are, and if there's a credential, it will be abused, and we just have to assume that now. It used to be if it gets pushed, we're gonna assume it's gonna be abused. If it's in plain text, in something someone could break into, it's gonna be abused.

It's forcing this reckoning of governance that, okay, we're going to have to deal with this now at scale, and identity in general at scale in a way that we've never really had to before. It's really forcing the issue, which- Yeah ... I think is a good thing ultimately.

Well, yeah. I mean, no one wants to do things because there's a gun to your head, right? But, but sometimes it takes that sense of urgency to get people to, to act.

Right. There's no both ways. Hey, guys, we're, we're probably over time here.

For people who wanna maybe get a copy of the report and see some of these like mind-blowing numbers for themselves, what, what's the best, what's the best place to go? com, and then you'll find the report advertised on the homepage or, you know, it's there. Excellent.

2026 S- St- State of Secrets Sprawl. And it's sprawling it is, sprawling it is. Yes.

Carole, Duane, thank you so much for coming here on Techstrong TV with us today and talking secrets a little bit. Um, continued success. Keep up...

Look, as I s- I... We were talking off camera. You know the thing about security, right, is when nothing happens, you're doing your job.

Uh, when it comes to secrets, security, and stuff like that, so many of us take for granted that w- we're, we're secure, or we haven't, you know... Or maybe we're just the zebra who didn't get eaten by the lion that day, right? Different zebra.

But, you know, the work that you guys do in, at GitGuardian is good work. A- as you said, Carole, that's the reason, one of the reasons you're there, right? It's, it's, it's being one of the good guys.

Keep it up, and we appreciate it, and have a great day. Thank you, Alan. You bet.

Thank you. All right. We're gonna take a break here on Techstrong TV.

We'll be back in just a moment. Hey, everyone, it's Alan Schimmel from Techstrong. You know, we're gonna continue with this fantastic series we're doing of sessions between some of the leaders at Microsoft, as well as analysts from the Futurum Group.

In this next session, we're lucky to have Clay Wesner. Clay is the partner for GPM Power Apps Studios at Microsoft, and Futurum analyst Keith Kirkpatrick. In today's, session, it's really a customer success story where Clay, joined by Keith, are gonna delve into a real-world customer story, in this case, Wells Fargo, offering a blueprint for leaders ready to scale success in the age of intelligent apps.

You're gonna see how Plow- Power Platform is being used to modernize complex regulated workflows with Copilot Studio agents and Power Apps. This session will highlight architecture, business impact, and lessons learned from deploying intelligent apps at scale. I think it's really a great session you're gonna enjoy.

Let's go to Clay and Keith. Hi, I'm Keith Kirkpatrick, research director with the Futurum Group. I cover enterprise software and digital workflows.

Hi, my name's Clay Wesner. I look after our low-code developer experiences on the Power Platform. Well, thanks for joining me today, Clay.

Maybe, Clay, you could talk to me, though, a little bit how Power Platform can actually help these organizations balance that, that agility to handle these types of, of scenarios with their compliance needs that often come up when you're dealing with things like banking or insurance or, or any one of these regulated types of processes. Yeah, absolutely. And it...

You know, we, within the product, we sort of refer to this as managed platform because it is very much a feature of, of the platform of how you can govern at this scale. And, and this has come from, you know, not just, us deciding exactly what's gonna be in there, but really f- folks and customers leveraging low code over the last 10 years and evolving to have a really, really strong governance. Because I, I think we learned very l- early on in the journey that if those guardrails are not there, people are just inclined to wanna turn it off.

Um, and, and we're very much... I use the, the word guardrails deliberately, and a lot of the things we do in a managed platform is focused around how do we give you the right control so you can still enable these types of tools, whether it be building apps, building automations out at scale, but do it in a way with the, the right sort of controls and guardrails on it. And so, like, examples are things like data loss prevention.

So I can set rules around what connectors and what data you can access versus someone else. And so I can also say how many people you can share an app or a workflow or something with. And so I can sort of mitigate the risk that you might be able to, to have working in low code versus someone that's received more training or onboarded to the platform.

And so typically what we see customers do is sort of implement this zoned approach of, you know, their, their zone one is everyone in the organization, and they say, "You can build apps for personal productivity. You can connect to your office data. Um, you can sort of work with those well-known sources, and you can go and share apps and flows and agents with up to 10 people," as an example.

"But once you wanna go beyond that, we want you to engage a little more with IT. We wanna make sure things are supported. " And then what typically happens is we'll then have a zone two, which is potentially some more sensitive data, potentially-You know, a-ability to share with more people within the organization.

" And then that final zone would be your IT, your dev center, who's working with, like, your really critical data around things like finance and, and HR. powerautomate and start building, and they're not going to fall into a trap there. They're gonna fall into the pit of success because we've, we've put those right guardrails on what they can access and what they can do.

If you look at not just if we're talking about, let's say, agentic technology, but just everything, if you look at the development of the smartphone, everyone expects sort of a consumer-grade experience throughout all facets of their life. And, and I guess that, you know, do you see that sort of pushing or helping to evolve kind of what c-customer success might look like, you know, not just now, but into the future? You know, again, earlier in the, the low-code journey, it was always IT departments, development teams that were looking at the low-code platform.

And more and more these days, as we're talking to customers, it'll be their employee experience team. You know, it will be folks f-responsible for actually healthy and, and, and productive employee experiences. And that's what I mean.

It's not just about cost saving, but, you know, it's about bringing the right tools in. There's the SNCF, the French railway, are actually a really good example. They run Power School, which is an onboarding school for the whole Power Platform when any new employee starts.

And this is becoming a really, really common practice that more and more folks are, as they join an organization, they're getting training on these tools, not as something they have to use to do their job, but as a benefit to them to be able to do their job in a more productive way. And I think, again, the, the consumer push and acceleration of AI is, is just accelerating that within the enterprise as well. Right.

When you're talking about human in the loop, you raise a really good point, because ultimately, this is still new technology, and you wanna make sure that particularly in, you know, you're dealing in a commercial environment, that you don't want this agentic technology to sort of run wild or unchecked. So I'm just curious if you could talk a little bit about, have you seen other examples where customers have actually deployed their sort of checks, and, and balances to make sure that their technology does what it's supposed to? Yeah, absolutely.

And there, there's a couple of ways we're seeing folks doing that. One is just in how we define and build the agents and the tools themself. While that agent has the ability to issue refunds, it can only do them up to a hundred pounds.

So it has very specific guidelines built into it that once it goes over certain criteria, loop in a human. Send them an approval workflow so that they can approve this, review the details. So that first one is just very structured, giving the agent details.

Um, the other side and, and this is where we've sort of really seen how apps have evolved in the last couple of years. If you've been looking at what we've done with Power Apps, we've introduced this concept of an agent feed, which is really about in the same UI that you would come into the app and, and do your day-to-day work, you start getting this feed of activity from the agents that are in your digital team effectively. And so you can start seeing where they're completing actions, where they might need assistance, or where they're getting blocked.

And so what we're starting to see there is even our UI patterns of what we traditionally thought an app was is starting to bring in this agentic behavior to give, you know what I mean, that human in a loop and that oversight capabilities. So I still want someone to have a really clear view of what tasks are being completed by the agents, what's being completed by AI, and in that view, get... be able to get into the reasoning, understand the logic and sort of the thought process that the agent followed behind it.

So it's not a mystery of why something progressed or why an action was performed, but as the, as the human responsible managing that team of agents, I can effectively go in and see why it did something that might be then become a teaching moment for the agent where we correct that behavior or change it for future cases as well. Well, you know, it, it's really interesting you mentioned sort of the generational shifts that are going on. You know, we're seeing the, you know, entry of these, I guess you'd call them AI natives, coming into the workforce where they don't know anything other than a world with AI.

Uh, and I guess, you know, that kind of begs the question, you know, we've heard so much about AI in the past, you know, particularly the last couple of years. Can you talk to me a little bit about, you know, s- what role can AI actually play within customer success? Because, you know, it's a wide, you know, AI has so many capabilities, but I'd just be curious to see if we could boil it down to this function.

Yeah. I-- And I think it honestly depends on the customer and how they're approaching it. You know, one of my favorite examples of, I think, sort of scale and pace, PG&E here in the United States, they're a big Power Platform user, and we, we talk about scale.

I think they estimate since they started their journey in twenty twenty-one-Along the lines of like thirty-eight million dollars in savings that they accrue to the power platform like huge in, in, in terms of scale. Um, but so much of actually what they've implemented is not just you know, cost efficiencies. They introduced an agent called Peggy, and they actually have a nice little avatar for Peggy that they, they introduced across the organization.

And Peggy now handles... it's between thirty to forty percent of their IT help desk calls. So built in Copilot Studio, Peggy has access to their knowledge base, all their policies and documentation.

And just Peggy, one agent, they estimate, saves them about eight hundred thousand dollars a year. And it's, it's absolutely transformational. And so even with the savings they were getting on the power platform between apps and automation, there is a limit, there's a limit to how much productivity that that can drive agentic, you know what I mean, tools and, and, and what people are able to build in Copilot Studio has really just broken through that, that barrier.

And you know, you look at, again, someone like PG&E, when they implemented Peggy, it was very simple, looking over knowledge bases, access to information. It helped a large volume of sort of tickets that would come through the help desk that used to be a human replying to an email or replying to an IM. Those humans now are actually providing much higher quality support on a, on more technical cases.

They're not helping someone log into Citrix for the first time or, or point them to something that's really well documented. Peggy's able to do that. But then they've also continued to evolve it over time.

And so again, one of my, one of my favorites that Peggy can do is getting folks that get locked out of their SAP accounts. One of the most common things that IT apparently happens thousands of times. Um, and now Peggy, using an integration between Copilot Studio and Power Automate, can actually open up SAP and go and unblock that person's account for them after they interact with her on Teams.

And so this was something that was critical to an end user to get unblocked really, really quickly. Peggy's able to do that for them fast. But it wasn't high value from an IT support team and what they were really providing, them going and opening up an account and unchecking a blocked checkbox.

And so I feel it's a really good example of where they started simple. They focused over sort of knowledge base examples. They evolved it into actions.

But it's something where they've gone for a, a high volume, you know, cost inefficient area. They've applied agentic AI to it, and that's something that go back three or four years ago would have been an extremely expensive tool to go and implement. Leveraging LLMs and leveraging Copilot Studio, they've been able to do all that in low code, which is super impressive.

I, I'm curious, you know, one thing, Clay, that you alluded to earlier is if we think about how apps were pre-- you know, previously developed and rolled out, it was IT who kind of managed that. Now it sounds like what you're saying is we're getting to the point where, you know, business leaders or even folks who are, are working within departments may be able to actually launch apps or launch agents, obviously with that human in the loop and with those, you know, specific guardrails. Are you seeing any kind of patterns emerging in terms of, you know, customers who've successfully scaled these, this agentic automation for more of a grassroots approach as opposed to springing from IT?

Yeah, you're absolutely right. I mean, we sort of see an approach from both directions, and some, some customers very deliberately approach it from one or the other to start with. I actually feel like all the examples I've, I've, I've sort of talked about today do quite well balancing both spectrums and both ends of the spectrum, sorry.

And I think that's where you start getting the real value multipliers. PG&E, great example. I talked about Peggy earlier.

That's an IT or centrally led tool. It was about optimizing a process within the IT team. But at the same time, they have thousands of developers across their organizations.

And when I say developers, I mean low code citizen developers that are enabled to go and build apps, to go and build agents, to go and build automation across their, across their team. And they've sort of very deliberately focused their center of excellence, their digital transformation team on a few core objectives. So that's the team that sets their governance policies, makes sure it's scalable, and then they also support and train those different sort of divisional leads across the company.

PG&E actually, again, I think they're on the spectrum, the end of the spectrum where they're doing this, you know, in a really amazing way. They have a conference every year called Level Up Now, where they actually get together all their citizen developers and, and those divisional leads from across the company to come together, share stories, share learnings, and sort of explain new technology. But it starts becoming a real cultural tool in that they're enabling people to go and solve these problems, make themselves and their teams more efficient.

And there's, there's reward that comes from that. You know, they're getting folks together, they're getting a lot of learning. And so I think, you know, while lots of companies are enabling citizen development, the ones where we see it's truly being successful, they're bringing this level of evangelism to it.

Well, Clay, maybe you can talk a little bit about some of these platform features that, that are kind of critical for managing customer success initiatives, because it really seems like, you know, there... To help organizations, you know, address all of these issues. There's, there's obviously the human technology component.

I would also say there's just the practices and, and, and sort of learnings. We actually have some good documented platform guidance out there of, like, what are the best practices in, in thinking about this zoned approach that I was talking about in, in how people, can sort of apply different levels of control to different parts of the organization. I would say then we start looking at the specific technology.

One, a lot of those guardrails just light up directly in the product. So as a new citizen developer, as a maker, when I go land at any one of the power platform tools, I can get welcome guidance with links to internal learning, explanations of where I can go to support. I get routed to my own personal developer environment.

So I actually have a sort of controlled, dedicated environment for me to go explore in, to experiment in. I'm not sort of working in prod. Um, I, I have the ability to be controlled.

And then things like pipelines, which effectively are a low code ALM tool, so that once I do build something, I can either use it for my-myself and my personal environment, but if it gets to the point where it does make sense for it to be deployed somewhere centrally leveraged by others, I can go through an automated deployment process where the right checks go. I have an AI advisor that reviews my code, makes sure my apps are secure and performant and accessible, and then get the right approvals before that gets deployed. And it's really that mix of, you know, we wanna democratize, we wanna make these things available to everyone across the organization, but then have these right built-in tools so that you don't have to go read a wiki to find out what's the process that you should follow.

It's built in to the developer tool. So I kind of just as I start building, get guided to the right environment, I get guided to use the right data, I get guided to share it and deploy it in the right way. And all of that we bundle up and sort of leverage within that managed environment, which gives the, the admins, the IT, the central digital teams that control centrally to sort of set up those tools and that content that they want available across the organization.

It sounds like all of these tools, you know, really underscore what you were talking about before, which is this culture of trying to utilize technology in a way where it's deployed at the right time, in the right space and with the appropriate guardrails, but while still fostering a culture of experimentation and, and ensuring that people feel empowered to use these new tools. You're absolutely right. Like, the cultural-- I think when we talk about your, your first question about, like, what's the new definition of customer success, you know, I think it's the customers that have implemented the right culture, and it feeling like it is a culture of empowerment and experimentation, not, you know what I mean, not something that they have to fight really hard to get access to.

Because that's where a lot of these examples where we have customers turn around, they've built something that's ended up saving them millions of dollars, it came from the expert that was involved in the business process. It didn't come from a central team. And to get that creativity and get that ideation, you need to give people access to these tools.

" Uh, and I think, you know, so will users, so will makers, they will find a way. And to restrict these tools to, to hide them, folks will go find a tool on the web that can help them be more efficient in their job. The companies that are doing this right are making it part of their culture to provide those tools and just really enable people from the get-go.

The technology is probably going to be more accurate over time if you're talking about trying to, you know, really assess, you know, images and differences between them. But, you know, one of the other things I'm really curious about is how can agentic AI and, and all of this technology be used in regulated industries? I'm thinking in particular financial services, banking, insurance, where, you know, there's a lot of complex process, but you also have to be mindful of all of the, regulations that are, attached to those industries.

Yeah. It... And it's actually quite surprising, I think, in, in this technology shift with AI compared to when we moved to the cloud, compared to the internet, compared to a lot of the others.

I think actually the regulated industries have, have actually been, quite a lot of the front runners on this. Um, you know, EY, for example, built PowerPost, which helped them with their financial processing, sort of end of month processing. They built this as a, as a sort of typical low-code application.

They're already looking at how they bring agentic checks into it to make sure that things are being posted in the right period, that they write-- they have the right information. Again, time-consuming sort of manual checks. Wells Fargo have rolled out agents to more than four thousand branches, you know what I mean?

A huge, huge number. And they targeted a process that was around their branch forms and procedure management. And this is something that was particularly time-consuming.

So if you went into a branch and said, "I need to set a power of attorney," or, "I need to open an account," and, you know, under maybe a non-traditional circumstance, there's a huge amount of internal documentation around those procedures, the right forms, the right information to collect. And before, that would mean as a customer is standing there with the branch member, they're looking up that information, trying to go find the right procedure, going and writing-Going to find the right form. So a heavily regulated scenario, but also really impacting a customer who's literally standing in front of you waiting you know, maybe on their lunch break, trying to, trying to get through the bank really quickly.

And so they rolled out an agent, you know, across all their branches to actually manage that forms and procedure scenarios. And so that now in the branches, those employees are jumping straight onto an agent talking about the scenario that the customer has and working with this agentic AI to basically get guidance on the right forms, the right procedures to follow. Even in these regulated industries, they're seeing the value in AI, and I think it's more about how they do it, making sure they have the right checks in place, making sure they have the right guardrails rather than what they probably would have done five years ago, where they just tried to turn it off.

You know, we, we talked a little bit about, you know, potential friction there, but are there any other sort of potential hurdles that organizations need to be wary of, and, and what, what's sort of your take on a solution? Like most things, we talked about human in the loop. You know, making sure you introduce this technology in the right way to organizations is really, really important.

I mentioned EY earlier. They were really, really successful in after building PowerPost, which helps them manage their, their sort of end-of-month financial processing. It simplified it.

It brought in some agentic behavior to valif-validate quality, and it... They had like huge gains in efficiencies in both. I think it was seventy percent in, in sort of the time, or ninety-five percent in lead time to get things posted and about a thirty-five percent cost saving for them.

So like real, real sort of impact to the efficiencies of their users. But what they did really well was once they built that tool, they told that story, they evangelized it. And so they helped people understand that this is how this technology was helping them.

This is how it was implemented. And that not only made obviously people a lot more receptive to onboard and leverage the technology, but it also started driving this ideation of other things to go improve within the organization and using similar technology. A lot of these companies are not coming in and doing a full low code approach of apps and agents and automation and reports all on day one.

Where we're seeing folks be really successful is they're leveraging the composability of the platform. You know, they're starting with, for example, they might have a, a legacy application that's inefficient for a user. So they go and use an app, they build more efficient, streamlined UI over the top of that.

That's an incremental solution they can deploy, they can get out to their users and start seeing benefits. Then on that same app, they can go and add automation. They can start getting approval workflows.

Then they can start bringing in agentic AI, getting that automation and that AI behavior incrementally building these solutions over time. And it's very much, you know, intentionally how we've designed the platform in that these are not all or nothing solutions. And you know, back to our earlier conversation, pace is extremely important these days, and people don't wanna go do a twelve-month waterfall project of every requirement met.

They wanna find ways to incrementally build. And by leveraging a platform that has common governance, these tools are designed to work together, apps with automation, with agentic behavior integrated into Copilot with that unified platform. So essentially you're setting up a framework to enable organizations to really drive these best practices in terms of making sure that, yes, you are implementing new technology, but you're doing it in a thoughtful way where you have the right checks in place and, you know, you really are making sure there's, you know, other things that, that you need there.

You need the audit trails. You need to make sure that, you know, when you do a project, you're going back and you're actually assessing, you know, does the technology achieve the goals that we set out to, set out to when we deployed it. E-exactly, and I think it's that, you know, there's two parts to it.

One is that being proactive, so as you're releasing a new app or a new agent to the organization, do you have the right controls around it, the right guardrails from the beginning? And again, our goal is let's have the right framework, the right tools, the right guidance to go really enable that and let an organization tailor those guardrails to sort of accommodate their level of risk, what they're, they're comfortable with doing. But then on the flip is make sure we just have the right visibility, the right auditability, so that as you're leveraging AI more and more within the organization, it's really transparent- Mm-hmm.

about what it's doing. You know, I think one of my favorite things with, Copilot Studio, and Pets at Home is a great example of this as it's interacting with customers on customer service. You can go into any step through any sort of run or action the agent has performed and see-- understand its thought process.

Why did it do this particular step? What were the inputs? What were the outputs?

What were the reasoning? Um, and not just understand it, but then also help teach it for, to handle sort of moments, at a different way in the future. And I think having those, those sort of tools from a governance perspective just built in, again, you know, we talk about it being unified for the developer, unified for the end user, but also for the, for the admin, so that they're doing in this sort of a central and controlled way.

And even then, whether you're building an app, an automation, an agent, you know, you've got that composability across the platform. But I don't think admins really want a super composable admin story. They, they want that to be a lot more unified and, and controlled.

So, you know, it-it's bringing the, the blend of those worlds of let's bring together multiple technology, multiple tools, but make sure then you sort of have one central view of, of how it's all coming together. If you want to really drive the use of new technology, you need to do it in a very stepwise fashion using a platform that allows you to unify people, processes, technology. It doesn't make any sense to try to do it in a very disjointed way.

You won't have the governance required to do it safely. You'll confuse people in terms of which tool should I use, which approach should I use. Ultimately, it really does matter to make sure that you have a unified way of approaching the implement- implementation of new technology.

It's also really critical to make sure that as you go about your journey, whether it's implementing low-code processes, implementing agentic technology, to have a clear understanding of your business goals. What outcomes do you want? How are you going to measure them?

And then how are you going to take all of these different learnings and then streamline it so you can actually apply it and scale it over the enterprise, not just for today, not just for tomorrow, but well into the future. And finally, I think the most important thing that kind of resonated with me today is you need to look for a trusted partner, trusted technology partner to help you through this journey. Agentic technology is very new.

Low code, yes, it's been around for a while, but, you know, there are still quite a few pitfalls that can be out there. To, you know, to go it on your own can be very, very challenging because you have all of that risk of potentially opening yourself up for errors, missteps, and of course, there's that, you know, we talked about it a little bit today, regulatory concerns. It makes a lot of sense to, to partner with a company that has experience with other enterprises to deliver these types of benefits using that new technology.

Hey guys, thanks for the throw. We're here with Ryan Macready, who's Vice President of Marketing for Liquibase. And we're having a chat about a new report that they have about, well, the state of database management and automation.

Ryan, welcome to the show. Thanks, Mike. It's great to be here.

Mm. Appreciate it. I think we have a newfound appreciation for databases again.

I think for a while there we took them for granted, but in the age of AI, suddenly everybody's paying a lot more attention. And, well, to be honest, we don't really manage them in a way that a lot of folks would say was, automated or even remotely automated. Maybe they're just manual most of the time.

But what's your assessment of what's going on here, and what are you seeing in this report that you guys did? Yeah. Um, you know, so, for those that aren't familiar with Liquibase, we have our community, which is wildly popular.

You know, we help, companies automate database change, in our community edition. And then, our Liquibase Secure platform, also provides, automated database change and self-service for developers, as well as the control of policies and security compliance a- across our 65, databases that we support. So really creating a standard of excellence.

And, you know, what we're seeing in our report, we, you know, we put out a survey for the last couple of months to our broader, broader community, which has now over 100 million downloads of our community edition. And, it was actually really compelling what we're seeing, and we're seeing it also show up in, a lot of the, prospect, conversations that we're having. 5%, or let's just call it 97%, of our community, has some form of AI touching production databases.

Um, obviously, there's a fair amount of risk to that. And Amazon just, you know, had a power outage the other day where I think there was something similar that happened where they had some, AI assist that caused some downtime. And, you know, in, in parallel to the amount of AI or LLMs potentially touching, production databases, we found that only roughly 27 or 28%, of the community has some form of enforced governance.

So really controlling what goes into the database, what tou- touches production, which of course, you know, is... There's a cause for concern there, especially as it relates to AI investments and, uptime and really reputation. Uh, 'cause what, what we found in a lot of our conversations and, you know, we work with nine of the ten largest banks in the world, a lot of heavily regulated, industries.

Um, what we found though is for the, you know, the larger community, if you have variation at the foundation level, at the database level, you obviously have your data models that sit on top of that or your data platforms, and then you have your AI models on top of that. If there's variance in the foundation, that's actually gonna impact the top, which, you know, every company in the world has some, some investment right now, in that. And, we're finding that, the database, and specifically standardization of database change across these environments is becoming critical to reduce the variations in their AI models.

Um, so it's actually a, a really incredible time here at Liquibase. Is it your sense that people are looking to automate this more, and how do I do that? 'Cause I feel like there's more at stake than ever.

The database is kind of serving as the memory for the AI system now, and it's also the mechanism by which I might apply some governance policies to AI agents that, shall we say, have a voracious appetite for data, for better or worse. So how are we rethinking the whole approach to databases? Well, so, you know, in our report, we found that, you know, most companies kind of on average, they have five different database types.

Let's call it, like, data lakes and warehouses and, you know, your Oracles. They have a, a lot of different database types. Some companies have ten, and then the amount of change that's happening is pretty substantial, too.

Uh, you know, the lead companies, it's twice a day. Uh, I think on average, usually it's, like, once a, a week, couple times a week. Um, but that's...

You know, you think about trying to standardize that change or that amount of change across those different types, that's a lot of nuance to get your arms around, as an organization. Um, so, you know, what we allow our, our customers to do, and we enable them to do, is allow developers to self-service that change. Um, and then we're able to have automated guardrails, in addition to that.

So we have custom policies that we provide as well as, a form of schema lineage so they can see, you know, the who, what, when, where, and why of change. And, we provide that for auditability purposes, but also for, their AI models as well. Um, which in a lot of senses, I think pretty profound, where some companies are still trying to do this manually through DBA tickets and scripts, which effectively does not scale.

Um, so we really empower DBAs to take control while allowing developers to self-service. That, to us, is effectively status quo. Where we really shine is when you're able to standardize all your database change across your enterprise and have the consistent policy and the visibility to ensure, you know, really this level of standard- standardization, you're creating, let's call it, a center of excellence for our customers.

And again, we're doing that for, nine of the ten largest banks in the world today, a lot of retail, healthcare, et cetera. Uh, all the-- all of which are very regulated. So in the report itself, what surprised you, or what leapt out at you the most that you didn't expect to see?

Well, you know, I think there's, there's two bits of it. There's the piece where we looked at the community and kind of said, "All right, what's the community doing? Um, what are their challenges?

" Um, and then we also looked at our customers. We surveyed our Liquibase secure customers, and they're on polar opposites of the spectrum, as you can kind of imagine. So when we spoke to our community, you know, I think hearing that, like, well, you know, you're...

have some type of AI assist, agentic DDL or LLM touching your production databases. Like, that's a... I didn't expect that number to be so high, actually.

I thought that that would be lower. Um, so that was somewhat of a concern. Um, the amount of change, you know, was, like, all right, that feels about right.

Like, we know that your elite teams are kind of releasing daily in some cases. Um, certainly there's a need for innovation that is always pushing developers, which is critical, and we of course support that. Um, I think the, the thing that was really interesting, though, is that the community said the number one thing pre-preventing our AI initiatives from happening or really putting our AI initiatives at risk is data quality.

And that comes down to the variance at the foundation influencing what's happening with the AI models at the top. Um, and I think that, to me, is, like, very profound and, you know, really compelling, in the research. And then when you juxtapose that to the Liquibase secure, customers, you know, really being a Liquibase secure customer means, you already have governance turned on.

So, you know, right out of the gate, you already have some form of policies already set where you have guardrails. Immediately, you have visibility. Uh, we have structured logging that you can send right into your SIM or, we work with you to create different dashboards for, reporting purposes and whatnot.

Uh, as well as, you know, various, kind of like risk scoring as well as drift detection. So there's these things that our, our customers have, really that governance right out of the gate, and, and I think it was like ninety-nine point two percent of them have this immediately. Um, so they're in, very different states, and you think about their environments and, kind of the maturity of accomplishing their AI initiatives.

Uh, and this is something I believe every CIO will care about as they're under pressure to accomplish these things. To your point, as we get further into this whole AI era, are we establishing some greater appreciation now for the fundamentals of data management that, you know, arguably we've kind of ignored for far too long? One hundred percent.

Um, you know, you see it, y- we definitely see it more so, I think, on the Snowflake, the Databricks sides of the house where it's very much the Wild West. You know, we've been working in the application CI/CD, side of the world for a really long time. Um, that is a very...

let's just call it, it's been around long enough where there's a lot of process typically there. But for these, these newer platforms, these newer data platforms, it's data, data engineers effectively not having a DVA always trying to manage all of this. And of course, there's a lot of risk associated with that.

Um, but, you know, from a, a CIO or CDO's standpoint, just that level of standardization is almost incredibly impossible to achieve without a Liquibase. Uh, you, you can't do it through humans. Um, so I think, you know, as the need for speed increases, and obviously we're seeing that pushed 10X by AI, the need for control increases, and ultimately that's what we're helping, some of these top enterprise companies achieve.

You know, and to your point, it seems like, you know, we all complain a lot when the AI is wrong or the output's off base. But I can't help but wonder, to what degree is that really the fault of the AI model or is it just simply we were unable to get to the right data at the right time and the right place, and therefore, the AI model then went off the rail? Yeah.

Cer-certainly you got... You know, there's, there's modeling things that need to be accurate, but I, I... You know, it's really easy to go point to the model.

Where we're finding is that the underlying infrastructure has to be working in parallel, or those models will never be accurate. Um, and I, I think as companies continue to walk down this path, and you could argue run down this path, they're probably gonna find that out the hard way, through downtime and audits and everything else that comes along with this. Uh, there's certainly a lot of pressure on these AI initiatives, as well as, you know, you could argue there's more investment ever in them.

So just think, we're in a really interesting time in technology, and, we're seeing, you know, the, the need for control over your infrastructure, to be as accurate as, you know, really your, your models are. So, it's an interesting time. Are there things that you see people doing today that you kinda just shake your head a little bit and go, "To your point, that's gonna be the school of hard knocks"?

Well, you know, we, we talk to a lot of community customers, and, you know, what, what we find is... Or, or, or maybe they're using some type of, database change management tool. Um, but they're really invested in the speed part of it.

They're like, "Well, we have to, you know, automate this change," and that's really important. We believe in that. I mean, that effectively is our religion as kind of the creators of, of, you know, that.

But, what, what they don't always understand is the inherent risk that comes without having controls. I kinda said, like, you can automate m- more database change, but ultimately that creates more risk. At a certain point, that tips over, and there's real consequences that comes along with that.

Could be a fine, it could be a security threat, could be the reputation. Um, I mean, I think Cloudflare is a really great example. They had, they had...

They took down half the internet in the fall, because, a database, changed. And, you know, there's things like that, and they're probably one of the most resilient companies on the planet. Um, so this can happen to anybody.

Uh, so I think it's really, you know, thinking about, all right, so how are we gonna make sure there's automated guardrails? We're not prohibiting speed. We're really helping our developers self-service, ensuring there's visibility, ensuring there's lineage, ensuring that we can go accomplish our initiatives so, you know, CIOs can continue to build, you know, what they're, being held to go build and doing that safely and, upholding, you know, the reputation of the company.

Uh, we're helping our customers achieve that. I also feel like we're overestimating the ability of our platforms to handle all this data as it comes in because not only are we accessing more data than ever, but the AI agents or whatever it may be will be creating more data than ever. So do we kinda have to have a fundamental rethink of the underlying in- infrastructure?

Yeah, absolutely. Um, o- of course, you know, as AI, whether it's AI assist or a- agentic AI is, Copilot, for example, supporting more development, the amount of code that it creates is pretty significant. And if they go q all- QA all of that, obviously takes time and, just in terms of process, I think people have to really rethink that.

Um, but again, it's not, it's not achievable through humans and tickets and documentation. It's, it's achievable through automation. Um, and I really believe that's policy as code, and it's ensuring that you have the right lineage as part of that.

Um, so that's, that's what I think companies will, um... We, we saw that this last year. Um, we actually, you know, had some pretty incredible achievements this last year, in terms of growth.

A lot of it was supported by exactly what we're talking about. Every company in the world has some type of modernization initiative. Uh, definitely the, the Fortune 1000 and above, which is largely who we work with.

0, you know, internal development platform really, where they're pressure testing all their infrastructure to support phase two. So, it's an incredible time. Is it your sense that organizations are being proactive about this, or are we gonna stumble our way into this and usually until there's something bad happens, and then we're gonna wake up and go, "Hey, you know what?

We forgot about some of these fundamentals"? Well, the stumbling's already happening. Um, you know, just last year alone, there was a few instances, a few headlines where we saw, agentic AI take down some production databases.

Um, you know, it's March, and there's already been multiple stories of this. The, the... I think the most recent one was, you know, the Amazon outage that happened, believe it was two days ago.

Um, so these things are certainly happening, and they're happening to, again, some of the most resilient companies. You know, Amazon's one of the most incredible companies on the planet. Um, so I just think, everyone is susceptible to this, and it's something that, you know, whether you're a CIO or a CDO, you really have to get your arms around quickly.

And, again, it's not something you can solve through documentation and humans. You really need automation to, achieve this level of-... kind of compliance and governance at scale.

Ultimately then, you know, if you get your crystal ball out a little bit, where do you think we're gonna be a year from now? What, what's, what's kind of like, you know, the, the, the journey ahead look like from where you sit? I believe regulation is gonna have a big impact on, AI initiatives and the trust of these models.

We're already seeing a few AI mandates. Uh, I know there's a couple in the EU. Um, there's been guidance put out in North America, too.

I believe this is gonna put more pressure on leaders to really, ensure that, customer trust is supported, they really have, you know, really the proof in place to show that they're in control. Um, that's ultimately what it comes down to, and I think that's, you know, a year from now where we're gonna be at. There's gonna be so much pressure on leaders to achieve this, because of regulation.

All right, folks, you heard it here. I think we can all agree that AI provides great powers, but as always, when you get great power, there's more responsibility. Ryan, thanks for being on the show.

Thank you. Appreciate it. All right.

And back to you guys in the studio. Hey everyone, welcome back here to Techstrong TV. Let me introduce you to our next guest.

He's been on here before. It's Graham Naray. Graham is the CEO of Oso.

Graham, it's great to have you on, man. How are you? Hey, I'm doing great.

Thanks for having me. Took us a little while to get going, but we're going, Graham. Um, as I mentioned, you've been on here before.

Um, you know, you're in business long enough, you get to... I- w- we have this family connection with your dad, and, it was really... I got a kick out of seeing you achieving success and starting a company like this.

And, and... But give people a sense, I know, you know, this is where you are now, but you've actually had a, quite a career. Give people a sense of kinda where you've been, how you got here.

Sure. So before starting Oso, I worked at a company called MongoDB. Um, when I joined there we were still pretty early on, so doing about a million in revenue.

Um, by the time I left we were doing about 250 million in revenue. Um, it was really cool to play, you know, a small part even in that, in that story. I sort of spent the first half of my time there building out the go-to-market side of the business, so first marketing and then a function to scale the sales org.

And then I spent the second half of my time there working as chief of staff to the CEO, a guy named Dev Ittycheria. He and I kind of made a trade. I told him that he could have my life for, like, two to three years, as my wife will attest, and he took...

You know, he definitely had it. Um, but in exchange for that I wanted to learn all the things about building and running a company, and that, and that's what we did. So in that period of time, we launched MongoDB Atlas, which is now the company's main revenue driver, helped take the company public in 2017, built the first product growth team, and then when I left, to start Oso, he became the first investor.

So Graham, that brings us to founding Oso, right? Yeah. I've, I've founded four or five venture-backed companies, I've co-founded.

I've been involved in it. For anyone, if anyone who's ever founded a company knows that it's not something you just, like, wake up in the middle of the night and decide on a whim that you're gonna start a company. It's something where you've gotta have a passion f- for what you're doing, that you believe the mission is important.

Y- maybe you gotta be a little crazy. But talk to us about what drove you here to, to start Oso. Yeah.

I mean, to be totally honest, look, I, I love infrastructure, I love security, and I love developers. And we were working on a different product at the time, actually, but it was through conversations with our customers, that it became clear that there was this area of the stack that was kind of untouched, hadn't really been worked on in, like, 50 years, and that's permissions. So just, like, to step back, right?

So for those who are, you know, not as, familiar with the domain, you know, you have, authentication. You can think about that as, like, the login screen to your app. But permissions are sort of the behind-the-scenes machinery that decide, you know, if you wanna click a button, if you wanna pull up a page, if you wanna pull some data, are you allowed to do that or not?

And it turns out that for the last 50 years that's all been done custom, and increasingly now companies like, you know, 1Password, Brex, Vanta, ZoomInfo, Productboard, and tons of others are using Oso to do that instead. Um, and now, look, I've been spending almost seven years working on this domain, and if I can be honest, like, I remember going for a walk with my friend once, during COVID in the Arnold Arboretum, park in Boston, and I was explaining to him what we do, and he finally got it, and he goes to me, "Oh, permissions. " Uh- ...

and I kind of feel like I've been in this, like, pseudo-unsexy domain for the last six or seven years, but now, but now with the shift to agents, all of a sudden everyone wants to talk about permissions. Here we are, Aaron Levie tweeting about permissions, Dharmesh, CTO of HubSpot, posting about permissions, and I feel like everything that we've been doing has kind of prepared us for this moment, which is, which is kind of cool. Well, you know what they say, the wheel of karma goes round and round, right?

And the wheel's come around to you now, so good for you, man. That's, that's... It's true, though.

I, I, I will tell you, I had a moment... So I spent the weekend digging in a Perplexity computer. Uh-huh.

And yeah, I could tell you all about permissions, you know, 'cause I-- hooking into WordPress and hooking into this and using that, and, and it's all about the permissions, right? 'Cause it could hook into anything, but you need the permissions and authentication to do it. And yeah, I get, I get that.

Um, Oso, give us the website. O-S-O dot... com.

com. Okay. Yeah.

com is owned by someone in China, and they want a lot of money for it, so I said no. com. The guy, the guy wanted about seven million dollars.

Yeah. And that's- Not happening. Yeah, no.

com. Um, but yeah, a long story. Anyway, so Graham, how...

Give us a little... So, so you've been at Oso now, what? Would-- Is it three years?

Four? No, we- Two? No, no, no.

This is-- We're coming into our seventh year. Um- Seventh. I knew it- And so this is what I mean ...

'cause I knew it was four. Look, I've, I've met with over two thousand engineering teams over the last several years to talk to them about this one problem. And, you know, in, in the domain of, like, just B2B SaaS applications, that problem was one thing, but the shift to agents has dramatically shifted this.

Uh, it has dramatically sort of, like, changed the dynamic. Let me tell you sort of what I mean. Um, we all know that apps are over-permissioned.

What do I mean by that? Like, you have more permissions than you need. This is like everyone has experienced this, as like, ah, just, you know, give them that role, just make them an admin, whatever, so they can get their job done.

And this is a kind of a bargain that we've accepted, this, like, gross over-permissioning because, it was in the name of user productivity. We don't wanna get in people's way, and fundamentally, we could get away with it because, one, we're trusting users, whether or not we should. We say, okay, users have judgment.

They, you know, they wanna keep their jobs. They're fundamentally accountable. Um, and two, there's, an upper balance the amount of damage that, you know, you or I, Alan, can do before the security team notices.

The problem is agents break this bargain, so agents don't have these constraints. Agents are not trustworthy. They can be tricked.

They can do silly things. They can hallucinate. Uh, even when they think they're doing the right thing, it might be the wrong thing.

And two, they operate at machine speed. They could do way more damage in five minutes than you and I could imagine doing in a span of, you know, five days. And we see this already happening with incidents popping up.

You know, there's an incident from AWS a few weeks ago where their coding agent decided that the fastest path to, resolving the developer's issue was to destroy and recreate the environment, which created a thirteen-hour outage. I don't-- I'm not blaming AWS. I'm not pointing fingers at anyone.

This is a, a problem that's endemic in our industry, and we view those kinds of incidents as the canaries in the coal mine, and that's sort of what, what, why we think this is an exciting time and sort of why we think the, the research that we're publishing is relevant today. I love it. Excellent.

Excellent. So speaking of research we're publishing, you guys recently, I guess in partnership or in conjunction with Cyera, another company we follow a lot, released this study of how enterprise permissions are actually used, are being used- Yes ... you know, now in this new agentic age.

Uh, and, and some of the... You know, I, I mean, I didn't read the whole report yet, under embargo, but I got the highlights, and some of it, I mean, kind of raised an eyebrow or two, right? But, you know, I didn't see that coming.

But tell us, Graham, you know, what, what, what, what for you are the key findings? Okay. Well, in some respects, it's like it's saying what everyone already knows.

It's like saying to dentists that not everyone flosses enough. Like, anyone in security is intuitively gonna know and understand the findings of the report, but it is quantifying something that I, to the best of my knowledge, no one has ever done before. I've seen it in surveys.

I've seen it in, you know, annec-- so the, the anecdata sense. But what we did that's different is we analyzed production permissions usage over two point four million users. Um, and what we show, which won't surprise anyone in the security industry, but shines a light on a problem that we really need to address now, is that ninety-six percent of permissions that people have, they don't actually use.

Said differently, people have 10X more permissions than they actually use on a day-to-day basis. And again, this was fine for humans because we had the trust, and we had the time constraints. But now you imagine taking an agent like Claude Code or Cowork or whatever and exposing it to this grossly over-permissioned surface area, and you could be in for a world of hurt before you know it.

So this is the first finding, is this, you know, ninety-six percent of permissions. Um, I'm happy to, you know, elaborate or answer any questions from here on. You caught me- Sure.

No. I, I... So look, first of all, let me raise my hand and say guilty, right?

I'm, I'm good for this too. I, I go and, and, you know, set stuff up and then forget I did it or I just... You know, I thought I needed it, but I really didn't need it.

And, you know, i-i-it's almost-- I don't wanna say it's human nature, but it's definitely... You know, it's the same thing that drives cloud engineers to not shut off an instance when they're done messing with what they were doing, right? Yeah.

It, it... We just... And, and what it really, in my mind, it's a form of debt.

It's a form of technical debt. Yes. Right?

Which is kind of that whole DevOps thing we, we talk about, right? Yes. And technical debt.

So-Big picture then, unused permissions, technical debt, yes, no, sometimes? Oh, no, this is, this is like security debt. And by the way, I d- you know, I...

We're all guilty of this. There's n- there's no... Anyone in the industry who claims to me that they have implemented least privilege at their company, I'd really like to see the proof.

The problem is not a human issue, the problem is not that security teams don't care or didn't know, the problem is that the juice was never really worth the squeeze because there were other things that were bigger risks. And- Yep ... in reality, this isn't a human issue, this is a systems issue.

So our friends at Cyera, you know, one of the things that they share with us, and that is included in this report, is that 80% of SaaS permissions are managed statically. You know, if that's the way the system works, you're creating a really high burden for someone who, you know, for security teams who are already way understaffed to go and do something about this problem. Um, and so this is not- But let me stop you here a sec, Graham.

When you say managed statically, what do you mean, for our audience? What I mean is, the way this typically goes is you start your job, you get a bunch of roles, and that's it. Someone...

If, if you, if there is supposed to be any sort of change, someone physically has to go and spend the time to manage that change by hand. Yes. Okay.

And, and that's, you know- So it's, it's, it's manual. It's manual, and for companies that o- you know, even small companies wind up with over 200 SaaS applications. You know, large companies like in, you know, the, the biggest ones have over 7,000 applications.

These are the customers that we're speaking with. You can't manage that stuff by hand. That's crazy.

Um, but, but not only that, you now have... So in addition to, you know, em- employees not accessing 96% of the permissions they have, they also don't touch 91% of the sensitive data they have access to, and almost a third of them have the ability to destroy or exfiltrate sensitive data. So again- Crazy ...

acceptable for humans, but you wanna hook up Claude Code or ChatGPT or whatever to this grossly over-permissioned service area, and an agent at that point, a tricked agent, an agent hallucinating, an agent doing something silly that it thinks is the right thing to do, now has what feels like carte blanche to do something very destructive. And where, you know, PwC is saying that 88% of executives want to accelerate their, their AI spending in the next 12 months, but, but I don't think any, you know, most of them are not clear on what's going to happen when the, you know, the rubber meets the road. You have these incidents like what happened with AWS and others, you know, people getting freaked out about OpenClaw.

These, again, these are just canaries in the coal mine. I, you know, I, I, I told my team when we, when we started this research that I would shave my head if the number were less than 90%, and as you can see, Alan, I still have my hair. Still have your hair.

Okay. On this interview I'll tell you, I will shave my head if there is not a massive, massive issue like the AWS one, but like way bigger in scope, before the end of 2026. You know what?

Okay, we're gonna hold you to this one, Graham. My... I could shave my head, but it really wouldn't do any good.

But for you, it's gonna mean something. So we- we're gonna come back and, you know what? If we, if we see an incident, we're knocking on the door with our clippers.

Um- We'll bring the clippers ... ex- yeah, that's a bold, that's a bold marker you're putting out there, my friend. Oh, I'm, I mean, it's easy.

This is, this is like child's play. By the way, I haven't... I've buzzed my hair only once in my life, so this is something I actually care about, and I'm fairly certain my wife won't be happy if I have a buzz cut.

I don't think- So I have real skin in the game here, Alan ... this is, this is real skin. So let me ask, is there anything in this study that, Cyera and Oso partnered on that kind of r- you weren't surprised at?

I mean, you were sur- excuse me, that you were surprised at, like you didn't see that coming, didn't have that on the bingo card? Um, honestly, no. And, and I think that's kind of like the, the proof of the pudding is in the eating.

Again, like anyone in the security industry, when I talk to them or I've given them a preview of this, they're like, "Yeah, of course. " This is, this is not... " However, I think this is the first time, or one of the first times, in history where you can draw a straight line between security work and a business outcome, which is to say that every board and every CEO, you know, from, from like, you know, Fortune 1 on down, has an initiative right now to drive adoption of AI.

They also need to protect their businesses. They have, you know, massive, massive businesses to protect, and if they want to do, one, drive adoption of AI without putting their business at a massive risk, they have to address this permissions issue. And so our hope is that this drives conversation in the industry for security practitioners where they can, you know, both...

where both we can have discussions on more productive ways to solve this problem for agents, which of course we're doing at Oso, but, you know, I don't claim to have all the answers. " Excellent. Graham, we're almost out of time here.

Um, I wanted to... So th- this study is out now. If you're watching this, the study's already been released.

com or where- Yes ... where do you go for this? com.

We'll have a banner at the top of the site. Excellent. Hey, man-Continued success with Oso.

Keep up the great work. You are in the, the permission nirvana space right now, right? Where- Who knew?

Who knew? Go figure. Yeah.

Uh, good for you. Do say hello to your dad, though, too, Graham. I will.

I will. Thanks, Alan. All right.

You just watched Graham Nora, CEO of Oso, here on techstrong TV about this new release, a new study done with, Oso and Sierra. Check it out. We're gonna take a break on techstrong TV.

We'll be back in a minute. Control, this is Agent Dev. I'm in position.

Copy that, Dev. Stand by for go. Standing by.

Hey, everybody. Welcome. Welcome to another episode of Agents of Dev.

Well, I'm just... I'm Mitch Ashley. Good to be here with my co-host, Brad Shimmin.

Welcome, Brad. Good morning, good afternoon. Hey there, Mitch.

How you doing? I'm doing well. You're on the East Coast, so you know, by, by as far as I know, it could be afternoon there.

I know it's not afternoon, but it feels like it. The day's been long already. Too much cloud code.

So I, I blame, I blame the time zone change, spring forward. It's messing me up. That definitely messes us up, especially our dogs.

They're, they're always like- Mm. "What... So when is t-time to eat?

Just, like, tell us. " They did not agree to this. Yeah.

Yeah. Yeah. Their, their representatives in Congress did not vote for this, but okay.

Whatever. Yeah. So, let-let's jump right in.

Um, we have some good call-out stuff, so jump in. I think you have some information about talking about what? Teradata?

Yeah, right. Uh, one of my favorite companies, 'cause I, I, you know, think this is... Anyone who's been in the, you know, data professional space for long knows who Teradata is and, and they, you know, I think, have done a great job of transforming themselves into a modern open data platform provider.

And they- Mm. like everybody, have realized the, importance of bringing data to AI, and particularly in support of agentic, you know, workflows and building anything agentic in nature. And, like everybody, they've said they've, they've cottoned on, to the idea of vectors, ve-vector stores and embeddings and the importance of, of that for semantic search.

Um, but unlike everybody, they also recognize that, it's not the only type of data, it's a representation of data, and that data might be structured in a relational database, it might be unstructured, video, audio, et cetera. It might be semi-structured- Mm-hmm. in JSON.

And, so they, they've updated. They've been working on this for a year, but they've, they've brought a pretty big upgrade this week, or this past week, probably just this past week. Week or so.

Week or so. We won't go there. Yeah, it's, it's hard to know because we, we record- We don't even know if it's morning or afternoon more this last week.

It is all the same, actually. Yeah. So yeah, they have this really neat idea, that I've seen from other companies like Oracle, and they've done a good job with this, and, and that is a hybrid search capability that, basically lets you do, like, traditional keyword matching and semantic search con...

in a conjoined manner. Instead of having to- Mm. you know, go get something and then pipe it on.

You're, you're basically treating it as a single retrieval so that you can bring to your agents, a context that is much more accurate. And what do we get from that? Less hallucination, so- Yeah.

Uh, that is my, my callout, my tip of the hat, goes out to those guys. Let me ask you, so 'cause you, you look at all the databases and, and I, I don't spend time in just kind of the database world. Uh, have they all sort of adopted the unstructured data or w-we'll look at anything.

mv files, we'll look at your JSON, we'll look at your structured into tables and columns and rows and et cetera. What, what's the state of the industry? Ugh.

Yeah. So the NoSQL versus SQL still kind of persists, but it's not with the same fervor that we saw earlier and all the, you know, full stack, app dev, you know, folks kind of said, "Hey, we want it, we want this," and MongoDB shot up. Right.

Uh, it's, it's, it's I think more of, a case of, you know, optimizing for the use case, and you may find yourself in need of a, a graph database, for instance, that can scale to a certain degree, and you may not get that degree of concurrency and, and lack of... and latency, you know, guarantees if you're running it as an add-on or a plug-in within, Postgres, for instance. Interesting.

Um, so maybe you want a pure play. So it, it, it, it's this specialization versus, I would say, not commoditization 'cause it's not that. It's, it's...

or even consolidation. It's, it's more what Oracle calls converged database and most humans call multimodal databases - Mm. Okay ...

wherein they can, they can handle all the different indexing simultaneously together. It's kind of the idea that we're seeing play out right now with the, the whole open, data lakehouse. You know, once, once you slapped on ACID, you know, c- you know, guarantees for transactions- Mm.

can it really not to do anything? It can do everything. So- Mm-hmm.

Mm-hmm ... just let's, let's just, you know, throw all our data into these open data lakehouses- Yeah ... and work with it.

But not for every use case. Yep. Interesting.

Yeah. It's a, it's a new era. Um, I, I think it's a nice time to, to be building right now simply because we don't have the same sort of, in, you know, technical debt, or inertia- Mm-hmm ...

that's associated with, you know, making a heavy investment in a single database management system with an emphasis on the management system. Excellent. Well, my, my call out is, Open A- A- Open...

I can't talk. OpenAI, is set to acquire a company called Prompt Foo. First of all- Awesome ...

whoever named that company gets the T-shirt of the week for sure. Prompt Foo. Right.

Your- You know, I'm thinking- Your Prompt Foo- Okay, now we can have- ... beats my Prompt Foo. Well, then we're gonna have Prompt Foo Fighters, of course.

I mean... Exactly. So anyway, you know, what, what, what jumps up to me about this one is that Prompt Foo is basically a security agent or technology, if you will, about making sure that the interfaces to the LLM, what agents are doing as they're going through the development pipeline, going into CI/CD processes.

Mm. Uh, trying to constrain, I would, in my terms, constrain from a security standpoint, what the agents, what the calls, whether they're APIs or, you know, direct prompts going through code, however that's working. Could be through web scraping, whatever method.

Um, how that's, how, how... If it's being done securely. Um, now what those- Mm ...

constraints are and how you control those, I don't know yet 'cause I haven't had a chance to really kinda dig into it and understand how Prompt Foo works. But it... " You know?

" This is bar. This is bar, not Foo. Yeah.

Yep. But it al- it also looks at, at RAG flows and behaviors of agents, things like that. So it's, it sounds like a really interesting company.

Definitely gonna dig into it more. And, what's notable is, you know, p- the companies that the big players acquire says a lot of about where they're heading and where there are gaps- Yeah ... or where they're looking to accelerate, right?

We could build that ourselves and, you know, you know, why couldn't, Claude or, or, Codex write this up tomorrow? Well, it's a little more complicated than that. Well, I guess, I guess we're gonna get into that.

But that, that's sort of the leapfrog. So it's not just the, the Ciscos and the Microsofts and, and Googles of the world that- Yeah ... do acquisitions as part of their innovation strategy.

You know, even the- Yeah, there's a lot to acquire ... quasi tech startups. Yeah.

Right. Some of them with excellent guerrilla marketing that might earn them, let's say, $5 billion, in an acquisition higher, as we've spoken about with a certain clawy, crustacean. Um- ...

it's, it's a, it's a weird world right now, is, is it not? " Or, "Tired... " Mm-hmm.

Well, we're in a- You know, it's like being a- ... we're in a world where we can do that, I mean- Yeah ... in theory.

Not everything is accessible through generated code. Well, yeah, you can do it. You can, you can one-shot pretty much anything agentically these days.

Clearly, we've seen that from Anthropic with their, compiler. So, but the, the point is it's... that's very different than maintaining that software over time.

And I think that's, that's where the rubber really hits the road. Well, that's the perfect lead in. I didn't bring up Foo Fighters, Prompt Fighters, Foo Prompt Code Fighters, whatever they are, acquisition by OpenAI, for that purpose.

But it, it, it's funny, you and I have kinda come to the same place independently about, what about, what about the whole... all of the debt that we're creating, and are we kind of- Oh ... are we sort of creating a false sense of security through AI-generated code?

'Cause we don't really know the quality of that code. Just because- Yeah ... it got generated doesn't mean it's a- adequately tested, it's adequately secured to the point of acquiring companies like that.

And, I did this, analysis on kinda where we are with doing AI development- Mm ... of AI, more AI-assisted than agent, agent-centric yet, 'cause that's where most people are. They're still in that earlier one-third of the adoption phase.

And especially for, for junior developers, it's really easy to say, "Hey, press the button. It's g- generated the code. Good.

Check it in. It runs. It seems like it works.

" There's a kind of a gap, I guess you can say, in AI-generated code verification debt, right? Yeah. Yeah.

That isn't being picked up and recognized by everyone. Now, m- more senior developers are gonna know, yeah, but okay, let me go make sure this is, this is g- not only gonna function correctly, but it's code I want. Well, I, I wanna ask, do you think that even the most seasoned developer isn't going to fall prey to this verification debt w- that we're, we're starting to talk about here?

Because if AI makes you 50% more productive, for example, the org- the organization that you work for isn't just gonna get 50% more goodness. It's- Mm-hmm ... it's gonna get more pull requests.

It's gonna get more documentation, more POCs, and more gambling against the future. You know? Mm-hmm.

And that the gamble is this isn't gonna go wrong. And all of that is, is like... there's not a one-to-one correlation between, you know, saving time, and increasing output or value, and I think people really kind of lose sight of that.

And so I, I love what, we're hearing from some, some folks who I, I like to read, like Lars Jensen and, and Kevin Brown-Uh, were the two, two guys recently that have been writing about this, lately, and I was just citing, an idea from there with a fifty percent increase. And I think they're absolutely right. It is gambling against the future, and it's the same gamble we take with any technical debt, except I think with, agentic code, you have a much greater risk of, of the Rumsfeldian unknown unknowns than you do with traditional technical debt, which is gonna be, you know, just increased inertia, or complexity leading to more friction, less speed, you know, more headaches, but you see the headaches.

Mm-hmm. Well- With this, you don't. And to, to that point is sort of the gap that you miss, whether you're more earlier in your career developer or let's say an experienced developer under pressure, on time pressure, right?

Okay, well, I'll, I'll take that, right? I'll worry about it later. Right.

Right. Is i-i-it isn't just that it's implemented and functional. It's like, how is it implemented?

Because when I wanna go change something to that, and that's what I've noticed really- Mm ... in the development process is, well, wait a minute, because that's not quite what I was looking for or how I wanted it to come out- Yeah ... in terms of an output structure or whatever it might have been.

And you realize, well, okay, to make that change, it's actually a pretty substantial change to code. And, you know, you can design code to be resilient and not require as much change, you know, whether it's through submodules and structuring and kinda organizing how you architect the code. Or- Mm-hmm ...

you can just kinda let AI do it. And I think we're, we're still in this ear-earlier phase of, yeah, but what's it generating, and is that good? Because if tomorrow Brad comes up and says, "Hey, we've gotta change the signal process this way for our reports," and it's a re-architecting, and you're like, "Well, if I would've known that, I wouldn't have done it that way originally, and maybe I don't want to do this now," right?

That kind of a question. So there can be some pretty big consequences by not knowing what the technical debt is, I guess is my point. That's right.

Yeah, I mean, when you, when you build a product and you have a traditional product, and, you know, you have a very clearly, you know, defined, you know, set of outcomes that you're, you're trying to get to, and you therefore architect it to do that within whatever constraints you've already set for that software. Mm-hmm. It has to be available at such time, has to be secure for this, blah, blah, blah.

And you ha- you know all those things, and so you build that into your spec that you use to build that software. And I, I've noticed with agentic, development in particular that v- ideas like spec-driven development are, are game-changing, absolutely game-changing. However, they don't work the same way that they do with traditional software development in that I, I feel like they're much more compartmentalized and therefore constrained, almost like, you know, a cart horse with, with blinders on that doesn't see the, the, you know, people yelling at the side of the horse- Mm-hmm ...

because we're afraid the horse will bolt. Mm-hmm. Don't startle the developers is what you're saying.

Right. You don't wanna startle the agent is what I'm saying. So don't let them see all the things that might, might influence them or might scare the c**p out of them.

And, and I feel like that's what w- the current ideas around spec-driven development really are doing, are putting blinders on, on the horse because we're afraid of it bolting. So here, here's the dilemma with that is my belief proven, proven multiple times. You know, sometimes when you have a theorem, you look for proofs even if it's not always true.

But- ... throughout my career, I, I've learned that, you know, f-fifty-one percent of planning only occurs in what you write down in a spec or a plan or a doc or whatever. Yeah.

The other forty-nine percent comes from implementation. You learn... That, that spec is woefully inadequate when you've completed it, right?

And you're ready to- Yeah ... okay, let's go into implementation, whatever, however you approach you're taking, whether it's with AI, AI or not. And that's one of the benefits of using AI is you can get to an implication, implementation place, let's say not production ready, but you can get there very quickly.

You can run parts of it very quickly. You can, you know, we used to block a spec out. Here's a, you know, a module we're gonna just do a dead call to or whatever, um- Mm.

Mm-hmm ... so we can start to work with it. We, we can do that very easily with, you know, with AI.

We can say that's a later feature, right? Da, da, da. We'll worry about that next.

Um, so it, it accelerates that learning curve or understanding curve of, let's say, intent curve of what you're really looking for because you're- Yeah ... learning through implementation. Um, and that's what, what I don't like about the, I'm in planning mode or spec mode or definition mode, like pick, pick your favorite.

Claude kind of gets into this question and answer. Do you want to do A, B, C, or D? Mm.

I feel like I'm in, you know- Therapy? Yes. Yeah.

Should, should I lie down for this question? Um, I'm like, yeah, okay, I, I get it. I, I s- see where we're going.

It's definitely helpful. But that's... Planning and implementation aren't, aren't, you know, co-linear.

They don't sit there- No ... and like I finish one and I start the other and- ... forever that shall be the process.

You know, you see where I'm going. On to the next. All, all, all of my specs that I've completed with Conductor in, in the Gemini, you know, experience, let's call it, have been archived.

And do you think I think about them? No. Do you think the, the agent thinks about them?

No. They're just archived. Somebody's like, "Hey, don't we have an ar- don't...

Isn't there an artifact for that? " Right. Right.

Didn't we do this t-three months ago? Oh. Yeah.

It's interesting. I, I'm-So, so here's the other side of the coin is, all right, if it's s-easy enough to generate new code, it's easy enough to make changes, so what if it's wrong? So what if it's re-architected badly?

Not, not, not for performance- Yeah ... or for scalability, but just in terms of maintainability of code. If, if creating code, the cost, the time, the effort, the complexity of doing that continues to drop, drop, drop, drop, becomes less- Yeah ...

consequential, okay. So what? Maybe I do re-architect half the application in making some changes.

And as long as I've got a way to test it and verify it and make sure it- Well, yeah. Are you- ... works, is secured and all those things, do I really care?

But, but the, the, just that's just it though, right? If, if- ... if your agent has agentic process can do that, do you really think that all the unit tests that it's going to come up with are representative of- Mm-hmm ...

the system as a whole, or are they specific to the task at hand? It's a blinders, right? I hope it's the latter.

Exactly. Exactly. Well, and that's, that's, that is...

that's a really good point, which is it's not only edge case, it's the making up our, our own unknown unknowns. What about, what are the things that could possibly happen, right? That we sit...

You know- Yeah ... I like to tell, tell myself and tell teams of what are the assumptions we're making? And let's, let's go break those assumptions.

Let's say you can't do this backwards. Well, what if you could do it backwards? Let's try it and see what happens and see what breaks.

And, and not just for fun and games. It's those are the things-- that's how you start to uncover some of those- Oh, I love that ... weird conditions, you know?

And that's where innovation occurs. Well, they are different, aren't they? Both- Yeah ...

feel like you're saying, and I think you're right. Yeah. Yeah.

Uh, through, through, consequential change and impact becomes opportunity from an entrepreneurial perspective. Mm-hmm. I just read this on an article.

I know Alan Post wrote about the, the tech media industry, the whole disruption with how Google's, not to get too far afield here, Google's changed the whole SEO calculations and algorithms and how traffic's being distributed. I'm like, yeah, they're... so they're rewriting the, the rule book.

Screw that. Tech media rewrite the rule book. Let's go rewrite it ourselves, you know?

Don't, don't wait for Google to figure it out and tell us what we have to do. Now, we're still gonna have to do part of that anyway. But don't just be a f- a follower to- Yeah ...

to, you know, waiting for the crumbs to drop off the table, right? Yeah. Under the table, over the table.

Always be over the table. Exactly. And before we move on to, from this topic, I wanna- Mm-hmm ...

I wanna bring up something that, is, has been personally bothering me lately. Mm. Okay.

This is, this is time for Brad's diatribe. No, it's not that bad, as, as last time. Should I lay down again?

Is this, is this consult the tape time? No, no, this is, this is vertical. Yeah.

Couch? Okay. This is, this is not that bad.

Okay. Um, but I, you know, in, in thinking about the verification debts, that, that load of verifying is, so hard and it's getting, it's getting more and more onerous. I feel like every time I build something that I'm...

I, I, I don't know how old our listeners and viewers are, but hopefully they're old enough to, to know what microfiche is. Um- Oh. Yeah.

And- It's not a, it's not a species that swims in the ocean. It's not that kind of fiche. No, it is not.

No. There's a very strange spelling. Uh, a- and it's on, you know, emulsion and it's, it's basically something you need specialized hardware to use.

But the way you use it, and what you're using basically is an archive of a lot of information- Mm-hmm ... housed in a very small footprint. And to interact with it, you would typically in the past, and I think they still exist in some places, like libraries, if those of us who know what that is, and in newspaper offices back in the day.

Mm-hmm. You would go into a room, and it would have all these huge, huge monitors, and you would s- basically scroll through, you know, at high speed. It looked like warp speed.

Literally scroll. Like all the stars. Before they were electronic, yeah.

Yeah. Yeah. And it, it was just overwhelming, is the word I, I would use to describe interacting with that.

And I feel like when I'm validating- Mm ... the code or even the documentation of what's written, I, I, you know, am in, again, back as a child, you know, going through one of those microfiches trying to scroll to find the newspaper article that I'm looking for and know that I haven't missed something important. I don't know that I haven't missed something important.

Mm-hmm. All I know is that I don't have the cognitive ability to, to actually keep up with the amount of information that's scrolling past as I try to get to what I think is the outcome. So I feel like I'm, I'm, like, mentally just crushing my, my ability to function as a human being in trying to keep up with this.

Well, it, it's, I think you're hitting on a really important point. Where as, as anything accelerates, and I think we're already here in many, in many other areas, but as we, as we... the velocity increases of code, of security vulnerabilities, of incidents that happen, of operations problems we have to s- to address or, or just debugging, you know, software and creating code.

Mm. As the velocity of that increases, there, there'd be a point there won't be enough humans on the planet to, to observe and look at all that stuff. Maybe we're already there.

No. Like, it, it is impossible f- you know, you're, you and I are banging our heads up against an impossible problem we actually can't solve. We can't look at all of it.

We can't understand all of it. Even with the best microfiche technology scrolling as fast as it could. I remember when, dial-up modems went from 2,400 to 9,600.

I can only no longer read it. It scrolls by too fast. We're kind of in that era of, you know- Yep ...

So something has to change. You have to do it differently. And that's, I think- Well- ...

what you're pointing to is that set innovation opportunity of like, okay, where's the verification debt bots, agents, cycles? Well, let's ask that question. Okay.

Let's, let's ask that question, what is that? Because what I see in the industry right now is a technique that we, we've known for a long time, and we use to a great degree, and we, we call guardrails, which is- Mm-hmm ... prototypically, you know, having an adjudicator model, let's say for instance, evaluating or, you know, having, what would you call them?

Uh, battle robots, uh- Mm-hmm ... rock 'em, sock 'em robots. Battle bots.

Yeah. Uh-huh. Yes.

To, to sort of battle over quality and completeness and a-accuracy. And so what I call all of this isn't guardrails, but epicycles. Hmm.

You know, the way that we very long ago tried to describe the retrograde motion of planets by just drawing another circle on another circle on another circle to try to explain what the heck was going on. And so I, I honestly think that as long as we have enough compute and the models continue to improve, that, yeah, I think that you could close in on Zeno's paradox increasingly getting closer and closer by simply adding more of these epicycles to a given process. And that coupled maybe with advancements in, in, like I was just talking about with Teradata for one, you know, adva-advancements like that, that improve the context that agents are, are looking at.

Mm-hmm. We can get toward a point where maybe you and I don't have to scroll the microfiche. Mm-hmm.

" You know, let's step away from the, the microfiche viewer. At some point, yeah. Let's, let's change lenses and look, look through the telescope now or whatever.

So- Mm-hmm ... so I think, I think I have a similar perspective maybe described a different way, which is we, we tend to think about linear things. I'm gonna go get a, an agent that does code security.

I'm gonna go get an agent that does guardrails for this. I'm gonna go get an agent that does... So I, I think w- the future we're headed to is kind of a crowdsourced agent model, which is, it's just like y- if you and I were on a development team of, you know, let's say a dozen people or, or larger, it could be 500 people for that matter.

But we all- Yeah ... come to this from a different perspective. You know, I'm looking at all this from how we're gonna scale and maintain this.

You're looking at this from- Hmm ... data quality and how do we do data protection or whatever perspective you're bringing. But you're also bringing a lot more than that.

You're bringing a whole bunch of experiences that helped you develop that perspective. So in a crowdsourced model, it, it's almost like a- Hmm ... I don't know the right, right mathematical term, whether it's a Monte Carlo exercise or what, whatever it is.

But- Well, those are really great- ... through that crowdsourcing, you, you get- ... great algorithms ...

to the answer. You know, you get to the answer. Yep, yep.

Which is, okay, yes, we've... All these factors have been considered, and the gotchas and downsides of scalability have been addressed, and none of them are 100%, none of them are at 20%, and let's say aren't important. There's some, okay, great, we've reached an equilibrium, not a three-body problem equilibrium, which is a whole different other issue.

Yeah, very different. We can never- We have, we have to, like, treat the world like it's, it's a two-body max problem. There you go.

Yeah. Or we can't solve it. If we get three-body code, we're in trouble, and nobody will ever know where anyone's gonna go next.

Um... Mm-hmm. But it, it's that sort of a, of, of a shared mindset across human computer AI agentic- Hmm ...

work that comes to the right conclusion. And so that way you have the diversity of thought or diversity of background skills, models, whatever it is that's bringing that, and that's what- Yeah ... we do as humans on teams.

Now, we're, we're less... We can't handle the volume that we could in an AI world as humans, but we could with AI. So long story short, short, I think that's sort of a manifestation of what you're talking about, how that looks like.

Yeah. And with the... I, I, you know, I, I don't even know if we need AI to do this.

I think maybe we need- Hmm ... to stop chasing determinism and start treating our agentic systems like the weather, and just use ideas like, you know, Markov chains, hidden- Hmm ... Markov models to, to- Mm-hmm ...

" Mm-hmm. " Mm-hmm. I mean, maybe, you know, I'm not talking about at a macro scale, maybe at a micro scale as to, you know, if we step away from the microfiche viewer, and we have these very complex autonomous systems building, documenting, and running themselves, that maybe that's the best, most efficient way to accommodate change and to anticipate change, most importantly.

So maybe we're talking about, you know, v- just a very different idea of what software actually is going forward. I don't know. Yeah.

It's not- Interesting ... what we're doing now, though, I'll tell you that. By the way, Courier producer, excellent producer, Podgap pro-producer just messaged that our audience, 63% of our YouTube audience is between 25 and 44, which sounds about right, right?

They're not only gonna all gonna be in there. They're 26-ish, he says. Yes.

Yes. So that, that's, that's super helpful to know because, you know, if you're speaking to someone who is four or five years into their career versus 15 or 20 years- Hmm ... into their career versus 30 plus into, into their career, you know, they're, they're coming at it from different perspectives, skill levels, experience levels, all that kind of thing.

So that's part of that collective, right? And that's the other thing about- Agree ... development is-Earlier in your career, you don't have all the faculties of that judge role that you perform.

You've seen some of my posts about this, and that's where, mentors or other folks can help speed the development of that by working with other people, to develop that skill. And I think you can in accelerated fashion. So- I applaud IBM, for instance, in bucking the current trends toward, you know, cutting off the, the people coming into the, the industry and instead are, are actually prioritizing bringing new people in because I think they recognize what you're saying- Mm-hmm ...

as being, you know, critical to being able to have continuity as a business moving forward. You cannot break the v- the supply chain of knowledge and insight and expertise. So love it.

It, it- We need to do more of that. It's the, you know, w-what got you here isn't gonna get you there problem, right? So you can't hire s- people to be trained the same way you or I...

I mean, you, y-y-you-- the people coming into the industry five years ago weren't trained the same way you and I were when we started- No, right ... on relative points in our career, right? Things are different.

We, we didn't have LeetCode. No, we didn't have open source or I, you know, what's a repository? You know, there was a lot of things that didn't exist when I started.

Um, but the point being is, if you're hiring and if you're seeking a job for how we developed code five years ago, that's still happening in organizations. But I think the reason why people aren't hiring entry level is 'cause they don't know what the next kind of person they're going to need and develop and grow as they use more AI. That's right.

So look at the skills, and that's what I'm trying to point out in this analysis, of using, you know, like who, who's the implementer, who's the judge, who are the functions that, you know, are this kind of function versus task functions. And someday th-those could shift around more, right? Could also move into AI.

But I think that's that forward-thinking you're talking about with IBM, which is they've got to be l... I haven't asked them about this, but they've got to be looking at this like, "Hey, wait a minute. If we cut off the supply chain now, at some day we won't have any more people like we need, and we'll be in the same problem- Five years ...

with COBOL that we had- ... you know, ten years ago of they're all retired or died or whatever. We can't find any more.

We've got to retrain the next generation. So kudos for thinking systemically, not short term. Yeah.

Agreed. Okay. It's time for the drop.

All right. Um, I'm gonna, I'm gonna throw one out here to start with. one01 or whatever number it was addition of, of the of the code.

I'm like, I'm not sure about that numbering scheme, but whatever. It doesn't matter. The, the...

with the thing of there's a bunch of features in it. A few, well, there's things like hooks, a-and none of these are necessarily new things, right? There's hooks, there's, some behavior controls.

But seriously, hooks, hooks has just been added. Yeah. Just, just been added, right?

Okay. So- Okay. Not new.

Just, just checking. A-and as I looked at it, and I was talking to a reporter about it, I'm like, yeah, these are... y-you could look at this as run-of-the-mill kind of features.

Maybe some are catch up, maybe some are a little bit new, maybe a lot new. But when you look at how they're implementing it, they're taking some of the control plane, like, I need to manage what AI is doing. I need to constrol- control.

I need to make AI accountable for what it's doing. Yeah. The starts of that, you know, I've, I've thought of it mostly about how you do th-do that in the broader environment of in development- Mm ...

and test and CI/CD and into production. And they're pulling some of that into the IDE itself. So, you know, I immediately thought of, you know, two thousand and one and the Starchild and the evolution of humans, and now we're- Yeah.

Yeah ... into the third generation of, of AI... IDEs.

Maybe I'm thinking a little too far ahead but it could signal s- of a change of this move from code centric to directing, judging, and controlling and, and what happens with AI more than just a set of tools. I've got five more slash commands in my IDE. Okay, great.

So what? Now you got five more things to do. I mean- Right.

Right ... not so what. They're all good.

But it isn't just adding more features, more ornaments on the Christmas tree. It's about changing how we manage the whole process. And I thought that's what's distinctive about this VS Code release.

So it's gonna go by the wayside. Most people say, "Oh, yeah, yeah, they're catching up here. " I think there's more to it than that.

Yeah, I love that. It... By the way, is it still written in Electron?

I don't know. Is it still an Electron app, I mean? It's probably been rewritten ten times by now by Boris and everybody else.

But rew-been rewritten. I, I don't know. Call, call me when it's, when it's native Go or Rust.

I, I, I don't know. So- I doubt it's been rewritten. I don't think so either.

That's, that is like the definition of inertia. Uh- There you go ... absolutely.

I mean, it is a behemoth. Oh, yeah. Yeah.

Yeah. So my... That...

And actually that figures into my drop as well. Um, I... my preferred editor IDE, as you know, is, is Zed, which is written in Rust.

Um, but, uh- You have, you have, you have principle scruples. It's in Rust. I do.

I, I stand- Okay ... behind them firmly. Okay.

Um, but, but, I, I look at it as more of a text editor than, you know, a, a, an agentic IDE, the same way that I don't use Warp, the, you know, terminal emulator as an agentic terminal. I, I want my terminal to be a terminal. I want- Mm-hmm ...

VI, you know, motions to be what I use, and I want just my Zed editor to be all those things, not a lot more. Mm-hmm. And, you know, shiny toys, you and I are victims of shiny toys all the time.

But-I re-I stumbled across one that I really like as, as like an actual agentic harness, one that would... I would consider my, you know, IDE, in that as you're, as you're talking about, is what I use to state my intent- Mm-hmm ... and to shepherd, that intent to, to an outcome.

And that is, Pie, just it's ca- the... from the Pie Nano guys. Uh, and it caught my attention, and this, by the way, it's, it's, it's written in TypeScript, so I...

don't think I'm against TypeScript with my hatred of Electron. Um, but- You'll over- you'll overlook it for, for the time being. I will, absolutely.

Just semicolons aside, okay. I, I, I can't get over that one. But, at any rate, their tagline, it, it really caught my attention.

" I just adore that. It's a... It's like, it's like the anti, you know, the anti-VS Code 'cause, 'cause like it- it's basically completely extensible.

You, you can write extensions, you can write skills, you can write prompts, you know, template, templating prompts, and themes and all that stuff. It... But it's not...

All that isn't baked in. Um- Mm-hmm ... and for ins-instance, they don't even include an MCP server, and they do that on purpose.

Uh, I, I would imagine probably because there are better ways to do this. Um, and, I, I love just the simplicity of it and the extensibility of it, and the fact that y- I can basically use it to write itself, to, to build- Interesting ... itself from within it.

And that's something... Uh, you know, I've, I've... This goes back to what we were talking about earlier, earlier with spectrum and development, and I'm sure everyone here has heard of this GitHub repo called Superpowers.

Mm-hmm. Right? Which, which I think is- If you haven't, check it out ...

over, like, seventy-five thousand stars. Yeah. Oh my God.

Yeah. Yeah. It's, it's really cool.

Um, but what caught my attention with that is, is, within Codex, so OpenAI's, you know, UI, or ID, whatever we're calling it, agentic harness. To install the superpowers, you don't... You're not, like, doing a curl in a shell, you know, pipe.

You're basically... I, I wrote it down. Um, this is what you just type.

" Hmm. That is how you install it. Hmm.

Okay. Is within the agentic harness itself- Interesting ... as a English statement.

Interesting. Boy, no- Right ... dash dash parameter, dash dash parameter.

No. And if you, if you look at the install, file, which is, a markdown file, you will see that it's, it's, you know, a lot of text and a lot of e-explanation along with some bash commands, you know, laid out in there- Let's dig into that ... to run to install this thing.

We should... In an episode, we should really dig into superpowers in GitHub, some of the things that it does- Yeah. Yeah ...

for folks that haven't used it. We could maybe do a live, a live demo for a live, failure of a demo. Well, well, I am gonna do a little, little bit of, though I'm not qualified to do it, therapy for you to end this episode.

Hmm. And I, I think one of the reasons why you may be so colon, semicolon adverse goes back to Pascal days of ending all your- Okay ... statements with semicolons.

I remember those. That's going back a ways too. Uh-huh.

Maybe it's a little PTSD happening there. But I don't know. I'm just saying.

It's possible. It could be. It is quite possible, man.

It is quite possible. Yeah. All righty.

Well, Brad, as usual, it's, great fun. Enjoy talking with you and doing this together. We, uh- Always ...

we hope that everybody, all of our twenty-five to forty-four-year-old audience members- Twenty-six-ish ... twenty-six-ish, enjoy tuning in, and we sure enjoy doing this with you. Send us your feedback.

com. And, be sure to like, follow, share with your friends. We appreciate everything you do.

If you have ideas for topics, guests, things like that, we do have guests that come on. Um, both vendors, but also, you know, there are other, other experts that we'd love to have come talk with us too. Yeah.

So, I'm talking to Tracy Reagan, one of my, long-term colleagues in the, software open source world, about having her maybe join us for a episode, Brad, so... And she said- Oh, I'd love that ... " So we just have to schedule that.

So we'd love to get more guests on too. Thank you for listening. Thanks to Corey and our fabulous team, production team that helped bring this to you today.

Take care, and we will see you soon. Control, this is Agent Dev. I'm in position.

Copy that, Dev. Standby for go. Standing by.

Hello, everyone. My name is Dave Moreira. Happy to be here.

First time as a presenter here, but I'm no stranger to Field Day. I've been a delegate a few times in the past, few years ago. Um, so I'm very happy to be here.

And, something that, that I've learned during that experience is that you guys like to dive deep right away, and that's exactly what we're gonna do, right? Um, so today we're gonna talk about one of my favorite features for VMware Cloud Foundation 9, which is, VMware Advanced Memory Tuning with NVMe. You may have heard about it or not, but my goal here is so that you understand what it is, how it works, and how it benefits our customers in a way.

Um, I call it the beyond the DRAM barrier. I wanted to call it the RAMpocalypse, so maybe I should just- ... trademark that from now on.

Uh, but you, or most of you are aware of what's happening with the prices in RAM, right? So this is very timely subject, and, hope you get a lot out of it. Yeah, when we're talking about this feature in particular and how it fits within the whole VCF ecosystem, right?

We have a lot ofUm, components within VCF, right? We have the compute component, storage, networking, which is vSAN, NSX, et cetera. So the compute component is what we, you guys know as vSphere, which, which is vCenter and ESX.

And yes, we changed the name back to ESX from ESXi, 'cause we like to change things up. So this feature is part of that core compute component. Say that fast five times.

Um, so it is included on the core vSphere area, right? So whether you're running VCF, VVF, it is already part of, the ecosystem of VCF. Right, so we got that out of the way.

So what exactly is memory tiering, right? So I have this quick diagram here. What we're trying to do is leverage less expensive devices to act as memory, right?

Um, so in this case, we're gonna take high-speed NVMe devices and pair that up with DRAM. DRAM is always gonna be that tier zero, what we call tier zero, the first point of entry for V- memory pages. And underneath that, we're gonna have NVMe devices to kind of back that off, right?

Uh, but from a VM perspective, we're creating a logical memory unit per se, and presenting that to VMs, containers, VKAs. Whatever you're running on VCF, that's what we're presenting to VM. So from a VM perspective, they have no idea what is happening.

They don't know there's NVMe, or they don't know there's DRAM. They don't know what type it is. They just know, "Cool, I have a bunch of memory, I can use that.

" They worry, a- again, they consume that logical memory, and I do want to point out that this is a native solution, so we don't have a separate appliance anywhere. Uh, we're not taking additional resources, a ton of resources anywhere else, so it is part of vSphere, which is our core compute component. And the goal here is to lower the total cost of ownership for...

What we're seeing here is the cost of DRAM. So even before all this shift to HBM memory happened for AI, we started working on this a few years ago, and we did see that most of the bill of materials, about 80% of it, was the cost of memory, just, you know, a few years ago. Now, that has shifted to almost 96%.

So when you go buy a server that is probably now $200,000, 95% of that price comes from memory alone, right? And this is what customers are looking at, and they've been keeping me pretty busy lately asking, "Hey, up to 40%," which is what we're doing here, right? So instead of buying one terabyte, I can get away by buying five 12 gigs of memory, for example.

So if we dive a little bit deeper into the NVMe side, you may be asking, "Well, okay, how does it work to kind of, kind of be connected over the wire? " So the answer to that is, it needs to be a direct connected NVMe device dedicated only for this purpose. A lot of customers think, "Oh, I'm wasting space.

" No. No. If you think about it, it's going to affect performance.

We don't want anything to affect that I/O path, so we're dedicating a device just for memory. We're creating a partition, dedicated for memory alone, and, you know, we can scale up depending on, the ratios, and I'll talk about the ratios in a second here. " And that's a valid point.

0, we have, support for hardware RAID, so you can buy a controller. Uh, even Read Rock is supported as well. And you can have two devices working together.

So the memory pages, they go into NVMe, are gonna be mirror that way, so if you lose one device, you have a backup there. Any questions so far? I'll throw out one.

Yeah. Um, one that came to mind was, speaking of the DRAM and, like, the cost and everything, so, like, what percentage DRAM reduction are customers realistically seeing right now? So I mean, I'll explain how it works later so it, it may more sense, but we're seeing about a 40% reduction in cost by doing this.

Um, so alone, a- all of it comes from the memory alone. Not only that, but also you double your density- ... of the VM.

So you can bring 2X amount of VMs, and because now you can push the, the DRAM and NVMe together even more. Thank you. Good question.

All right, so that's what it was. Uh, but how does it work? Where's, where's the magic here, right?

Um, so what we're seeing, I'm, I'm gonna put this up, right? A lot of customers are seeing this, and even internally, we're, we're consuming a lot of memory. Think about those workloads.

I used to be an admin back in the day, and my SQL Servers, every week, kept asking for more RAM. " Like, you know, one terabyte of RAM when active memory was very small. So what we're seeing here is that we're memory bound on the host level, but also we're seeing that the CPUs are being starved for memory.

There is a, a... They need to communicate to each other, and they need each other to work. And CPUs in this case are not really able to push any more because there's no memory available, so that's another problem.

Number one problem here, we run out of memory. Uh, the other problem here is that we're not really using resources efficiently, right? So we're wasting space.

We're wasting not only resources, but also power, cooling, rack space. If I have a server that is fully popular- populated with CPUs, all the DRAM slots are populated, you know, how can I move forward? I need to buy more servers, and doing that now, it's, it's very, very pricey.

So if we dive into that, scenario, right, where we're using most of our memory and a little bit of the CPU. We have an algorithm, it's proprietary, that is every few cycles it's looking at the activity of the pages, not only if that page is being used, but how often, how recently, what pattern. So we looked at all these things, and we make decisions based on that, and we classify those pages into hot, warm, cold, very cold, right?

Every few seconds and, you know, based on that information. And, and then we bring NVMe, and then we make the decision of moving some of those inactive pages down. " So, yes, vSAN, I always say back in the day, very similar, right?

We, we push down some of those, data that is not being used, push it down to inexpensive devices. So that's exactly what we're doing. We take NVMe, push the pages that are not being touched down to NVMe, and DRAM is gonna keep those pages are active.

So from a VM perspective, they're always going to talk to DRAM because they want that data fast, they're gonna get it fast. When a VM comes down and say, "Okay, I need that page that is cold," we're gonna bring that up to DRAM and we're gonna read from DRAM. So we're constantly moving up and down.

Is it per VM or is this host-wide or how can that- Yes and no ... allocate? Yes.

So the way we can configure this is, by cluster, per host. Um, we can disable the VM. So if a VM if, for example, is latency sensitive, you probably don't want this, right?

Just in the chance there to, latency there, so we can disable the, the VM from having memory tiering. Okay. Good question.

So Dave, this... And I'm Jack Palmer from Paradigm Technical. This seems like standard ca- standard memory tiering and caching, and you're just, what's...

I, I don't understand what the, the, the, the secret sauce is here other than you're pushing... I mean, what's on NVMe? What's, what's the storage mechanism that's on the NVMe bus?

So you're thinking about, swapping, is that what you're saying? You, you, you're thinking- Well, yeah. I mean, you, you, you've replaced...

You, you've described what is essentially a memory cache, right? Yeah. And a cache replacement algorithm.

Yeah. You know, classify what's hot, cold, warm- Right ... in the cache.

Yep. And we decide what we evict out of the cache and how we bring it back in. That's different.

Yeah. In, in, in way, yeah. Standard system memory caching, right?

Right. So you're pushing it... You keep saying you're pushing it down to NVMe.

What's the backing store on NVMe? Is it SSD? Is it DRAM, SRAM on, on an NVMe card or?

It's, uh... So it's a PCIe NVMe device, so it could be any form factor that way. Uh, but it would do require certain endurance and performance classes to, to meet the requirement for performance.

So it's just a standard backing store. It could be an SSD. Yeah, yeah.

It could be, a DRAM, a slower DRAM or something that's somewhere down in the NVMe side of the world. Yep, yep. Okay.

Thank you. Uh, Denny Cherry from Denny Cherry & Associates Consulting. Um, what's the class...

What, what are you considering when you're looking at the, the, the pages? Is it reads? Is it writes?

How, how are you deciding what's hot and what's cold? Uh, both. So we look at, like, the activity based on how recent it was read or written, how often was done, the pattern, all those things come into play to make the decisions.

Okay. Right. Um, now there's a...

Expand here a little bit more on that. If the DRAM is being used maybe only 50%, we don't want to move pages just for the sake of moving pages, right? Because you know, that's additional resources, maybe a performance hit.

But so we have to pass a threshold, about 70, 75%. When we have memory pressure, then we start proactively moving pages out, right? So we're not, being reactive here.

We're proactively thinking, "Okay, we're getting full. " So just a quick question. Leon Talera from InfoCert.

Um, we are enabling DRAM using this method, this, technology. Enable to see larger memory than, larger capacity than you can have with a single RAM, or it is another- Yeah. Yeah, we'll get to the ratios in a second.

But yeah, we're starting with a 1:1 ratio. Oh, yeah. So you're, doing 100% more right off the bat.

Wow, okay. Thank you. Yep.

Uh, so again, you know, this is presented to a VM as one logical unit, both, both of these tiers. And once we move pages down to NVMe, make room on DRAM, then we can push more. And here's where I talked about the having 2X density on the VMs.

5 more, VMs here, and then we can push both resources at the same time. Because we have memory available, we can actually utilize more of the CPU that was just sitting there idling, right? So this is, this is what we want to see.

Uh, you know, we pay for those resources, we wanna use as much as we, we can. I mean, that's the whole idea of virtualization, right? Good questions, by the way.

Uh, you also may be thinking, and you touched on that, about swapping and ballooning. That's still in place, right? So these methods here, we still have TPS, which is memory sharing, compression, ballooning, swapping.

That stays in play, in place, but these are more reactive, things that we do down the road. If there's a lot of pressure, we're still gonna do swapping and ballooning eventually. Memory tiering is gonna come in right after TPS and help us to be more proactive, gives us more space, manage those more intelligently, and those resources, return those resources back to the host.

But yeah, if we have more pressure, if we're pushing overcommitting and doing things like that too much, then we're still gonna have those, reactive measurements to, to gather some of those resources back again. Right. Yeah.

com. Um, can you talk a little bit more about ballooning? I recognize all those other ones, but that's interesting.

Yeah, ballooning, it's we pr- pretty much have a driver within the guest OS- Okay ... that it starts to inflate. Um, so it's kind of reclaiming some of those, pages back.

Ah, okay. Right? Um, but what happens there is that we don't, we don't classify those pages.

So because it's reactive, we just... If it's active or cold, it doesn't matter, we're trying to reclaim those, those memory back. Ah, okay.

Right? Thanks for the clarity. Versus the tiering where we actually classify the pages and we, we don't wanna mess with the hot pages.

Gotcha. Yeah. Good question.

Thanks. Thanks for the clarification. All right.

So we talked about what it is, how it works. Uh, now I wanna talk about some of the operations. The, and one of them is the ratios, how to configure all that stuff.

0 U3, which was tech preview, very limited. And the ratio was, a four to one, a DRAM to NVMe ratio, meaning that you could only expand twenty-five percent with NVMe. You know, it's small, but twenty-five percent more is more.

It's okay. Now, we went GA with VCF 9, and the default now is a one-to-one ratio, so you have a hundred percent more out of the bag, right? So you have, for example, a host with two hundred and fifty-six gigs of DRAM.

You buy a drive that's at least the same size, and you get twice as much. You get almost half a terabyte. What if, you know, we go and we buy a bigger drive?

So, if we don't change the ratio, we're still only going to use two hundred and fifty-six because we're basing everything off the DRAM. A couple things happen here. Number one, you know, with, the NVMe controller, we do wear leveling, so we're just, you know, distributing those reads and writes across the drive, even if it's bigger.

Uh, so it's going to extend the life of that drive a little longer. Based on, on our testing, endurance and performance classes that we require or recommend, those drives should last about five years, so having a bigger drive doesn't hurt to ex-extend it a little more. But also it prepares you for the future, right?

So we know your workloads are great for a one-to-one ratio, but what if the active memory is really low, right? I can actually push this even more and change my ratio to a one to two. Uh, so I get three hundred percent or two hundred percent more, giving me almost a terabyte.

Uh, and I didn't have to do anything. So I had the same drive from example two, you know, five twelve. The partition was already created.

All I did was go to vSphere, change my ratio from a one-to-one to a one-to-two, and now I have, you know, an extra two hundred and fifty-six gigs of memory from that. Uh, so that is very... allows you to be very flexible expanding.

So just imagine doing this with DRAM alone, right? You have to take some DIMMs out, put bigger DIMMs in, et cetera. Uh, so it's a little harder.

So this is very flexible and allows you to go up and down as necessary. How far can you push that? So we can go up to four X.

Okay. Um, so, so there are some workloads that we noticed, for example, VDI, right? Some of them are not doing much.

They're just like kiosk somewhere, so we can push this up to four, four X. The maximum, partition size is four terabytes, for now, so hopefully we can expand that later. I was talking to our customer yesterday.

They have huge, infrastructure, so they wanna push that up even more. So, yeah, four terabytes maximum partition size and four X, memory increase here. Uh, and also last example, you know, it doesn't matter what the ratio is.

If you don't have a big enough drive, obviously you can't make space out of nothing. So I bring this up just to... for those that are watching, you guys, it's important to properly size, think about the future.

You know, what can you... to, to invest here, invest here. Um, don't try to save money on getting a smaller drive.

You know, you're saving a lot of money already with DRAM alone, so don't try to cut corners here. All right. Another, another question I get is, does this work with everything else, right?

Um, when there's a new feature and we say, "Oh, this doesn't support vMotion," people don't... are not very excited and, you know, rightfully so. Uh, vSphere's been around for over twenty years and, you know, HA, DRS, vMotion, that's something that is very important for admins.

And, you know, being an admin back in the day, it's, you know, I can see that. So yes, we support HA, DRS, fast suspend resume, which is what we use when we hot add, hot remove devices. So yeah, definitely in.

I talked about RAID being supported. And vMotion's interesting. Um, and I say interesting because, you can configure, an entire cluster, but I also said you can configure per host, which means that you can have, within the cluster, you can have a host that con-configured for memory tiering, but some other hosts are not configured for memory tiering, right?

Why? Because maybe you have VMs that are latency sensitive and, maybe monster VMs that we don't support there yet, but we want to put them on those hosts, right? So there's, there's some flexibility there.

Uh, but vMotion knows that, okay, this host has memory tiering, this one doesn't. I can still move stuff from DRAM to RIMA-- DRAM and, you know, if I have memory tiering, I can do the, the tier out capability. If I don't, then I just keep it on DRAM.

So it does know. It's aware of that. I have a quick question.

Yeah. You mentioned all the features that are supported- Yeah ... but you didn't mention any features that are not supported.

Features or, um... The unsupported stuff is mostly, VM profiles, so security VMs. Uh, so if you think about, TDX, SEV, they encrypt the memory, right?

So we can't actually see the memory pages to classify them. So that's one thing, you know, I've been g- been, been getting a lot of questions about lately. Okay.

Uh, those are not supported. Monster VMs, latency VMs because of performance. And, there was a couple more, but yeah, w-w-we're going down the list and trying to, to get through those.

Uh, so version... This is the first version. There's about maybe five VM profiles not supported, but next version will be less and less and less.

Okay, great. So for those type, for those, my graph, for those types of VMs, do you have to... If you had to configure to enable it at the cluster level, would you have to then go disable it on those specific- At the VM level.

Yep. Okay. Yeah, you can go to the VM level or have dedicated host, for those.

Okay. Yep. So- And- Oh, sorry.

I was just gonna ask, and then what happens if the NVM tier becomes full? Like, does it fall back to compression or does it get swapped? Yes.

Yep, it goes down the list of, reactive measures there. Yep. But it, it sounds like this is going to potentially conflict with the guest OS's own concept of memory management.

Are there specific use cases that are should or should not be used for this, u- that you should not use this or should use this for? Um, so some that we've seen so far, like in, in-memory databases, obviously that's, something I wouldn't use for at least, yet. Um, there, there's some.

There-- We have a list on our performance guide that I can share. Right. But what's...

I mean, what is your... How do you un-how do you understand what the operating system is doing in measuring, in, in managing its own memory pools versus what you're doing? It seems like you're in conflict.

That if the m-the, the, the guest OS, right? If you're running a Linux system, it's got its own concept of all the memory, the physical memory that it has. It may not be, you know- Mm.

-it's virtualized physical memory, right? Mm-hmm. And now you're virtualizing physical memory again.

But that OS has a concept of what it thinks is going to be used, and it's doing memory ca- page replacement based on what it knows- Mm-hmm, mm-hmm. -is going to be used or not used. So in a normal situation where it's not just a, you know, an in-memory database is just a pile of memory, but for a normal application running here, you're, you're potentially in conflict with what the OS is doing.

Um, yes and no. I mean, we do it at a higher level, because that's transparent to the guest OS, pretty much what we're doing, right? So we're trying to be proactive.

If you know... I mentioned the patterns and all that, those things, right? If we know that page will may be read soon, we may able to move it before it gets called out from the VM.

So there's, there's a lot of... I can't talk about the algorithm itself. But yeah, we, we do it at a higher level and try to be proactive about what the VM is trying to do, if that makes sense.

I can share w-more info with you later. Thank you. Thanks.

Uh, the other question I get is, security, right? It's a SSD drive. I can walk into a data center, pull out a drive, and get some, some memory pages out of that, you know, point taken.

So we do support encryption. Uh, we support it at two levels. " So all those VMs within the host are gonna get their pages encrypted when they move down to NVMe.

Also, we can do it at the VM level. Uh, so we can go VM by VM, enable mem, encryption for NVMe if you want to. Now, this also works with all the other encryption stuff that we have, VM encryption, vSAN encryption, vMotion encryption, all that stuff.

There are separate layers everywhere. Uh, it works, again, at a different layer, and it is supported with the entire ecosystem. Uh, configuration.

0 and, and forward, right? We are leveraging VMware configuration profiles, which is kinda like host profiles, but for the entire cluster. Uh, so here what we're doing is just, just going in, enabling memory tiering, and, you're passing that out to all the hosts.

You can o-obviously get some, host overrides if you don't want to, to do that, but it would be... That's, this is the easiest way you can do it. You can do, PowerCLI, ESXCLI commands as well if you wanna, if you wanna go that route.

Uh, so the advantage of using VCP, configuration profiles, is that you set it, and then it lets it do its thing, right? 0, it does require maintenance mode, reboot, and, but it will roll through all those, based on whether you have vSAN or not, what VMs needs to be moved, et cetera. So once you set it, it does all the stuff for you.

Okay. Uh, some failure scenarios. Uh, obviously with RAID, if one device fails, it will go to the other device, right?

Pretty straightforward. But what if I'm using a single device? Uh, which, you know, some people don't like to use in controller.

I understand why it's an additional cost. Now you get a new point of failure there. Uh, you know, some- there are some other cons in there.

But if you have a single device, which we support, an HA event will be triggered on the VMs. And I say VMs because only the VM pages are going to NVMe. All the p-the pa-memory pages for the kernel, they stay on DRAM always.

So if we have a failure on a single device, the host keeps running fine. The VMs may or may not have an HA event, and I say that because they will fail when the VM tries to call down those pages that are inactive, right? So, as soon as you have missed, lose a drive, some of those VMs may go down right away, you know, they reboot somewhere else.

Or, you know, ten minutes later, you may see a couple more, or an hour later, or never, right? If that VM gets rebooted before any, any of that. So, yeah, it's not a full crash per se, but it kinda stagger i-in a way, depending on the workload, what the workloads are doing.

All right, and I wanted to leave you with this. There's a lot of information here. The, the wipe-performance white paper I was talking about that has a ton of information, it's here.

Uh, so on the performance, there's a blog and a white paper. Definitely, take a look at that. I have a series for, a, a blog series for deploying considerations.

I talk about what devices to use, why, sizing, what qualifies a workload to be, compliant for this, et cetera. So definitely, a lot of information here. Hands-on lab, if you wanna kick the tires, configure it, you know, play with it, you know, definitely, take advantage of that.

All right, so to recap what we talked about, you know, main driver here is to help our customers lower their TCO, and we're seeing this up to forty percent even, this was even before the RAMpocalypse, right? So, we're... A lot of our customers now are very active, actively looking to deploy this, you know, to save a lot of that, that cost.

VM consolidation, again, when we saw that, that diagram, moving pages down to NVMe allows us to free up DRAM, be able to utilize more of that CPU resources. So, we have more, more VMs running on the same host, saves us an extra, some extra money there by not having to buy additional servers. And yes, being able to use all the resources more efficiently, DRAM, CPU, NVMe, all that stuff.

Any last questions before I get cut off? This... I wanna make sure I heard this right, that this feature is both in VCF and VVIA?

Yes. It's a, a vSphere component pretty much. Okay.

Yeah. It's at the core. All right.

Uh, so with that, I wanna thank you all. Great questions. I appreciate it.

And, if you need more information, happy to help. Thank you.