Techstrong TV June 16, 2025

Hey, everyone. It was a tough time for Google Cloud. Hopefully it's better now you're watching Textron Yank.

Hi everyone. Happy Monday. It's Alan Shimel here for Techstrong Gang.

Um, we've got a lot to talk about today. Of course, the end of last week was a, was a bit of a bumpy ride for our friends at Google Cloud. We've got some platform engineering news to talk about it more, but we've got a great gang of people to talk about it.

Let me introduce you to them. First of all, for the first time, I think on the show with me is a, a friend of mine for many, many years. She is a multi-talented, faceted individual, Dr.

Dr Stacey Thayer. Stacy, it's so good to have you on here. Um, it does my heart.

Good to see you. How's everything? I'm excited to be here.

Thank you, Alan. Excellent. Also new to our tech strong gang, uh, gang is Jack Gold.

Jack, welcome. And you know, I don't know if you've had, tell people a little bit about yourself. Yeah, thanks Alan.

It's good to be here. Uh, I'm a, uh, industry analyst. Been an industry analyst for about 30 years or so, and covering a pretty wide range of stuff.

Worked, uh, for some big firms for a while, and decided about 20 years ago to go out on my own. So I've been doing that as an independent industry analyst for that period of time. Fantastic, and welcome to the Gang Jack.

Great to have you here. Our next guest is, well, she's been a gang member since day one. She's the, uh, deploy hub, CEO Open Source ambassador extraordinaire, the one and only Tracy Ragan.

Hey, Tracy. How are you? I'm doing great.

It's good to see you again. Good to see you all. And then last but not least, batting cleanup.

He throws right-handed bat's. Left-handed plays shortstop. Occasionally.

He's up, what near Poughkeepsie, but Harris in New York, our Chief Content Officer, Mike Vizard, And, and coming off a lovely Father's Day. So happy Father's Day to you too, Alan. Yeah, absolutely.

Happy Father's Day. All those dads out there and all people who stand in for dads too. There's a lot of people who aren't necessarily dads who take that dad role on.

And, and, um, thank you to all of them. Anyway, Mike, it, it was a busy weekend. It was a busy last week and it'll probably be a busy this week.

What are we, what are we kicking off with? Yeah, it's starting around Thursday afternoon, east Coast, I might think. com.

We had a few of them down for a couple hours 'cause it affected WordPress and all kinds of organizations were reporting issues. And Jack, you know, we're now here on Monday, and we're talking about hindsight, but I wanna start with you on this. Do we still have these kind of single points of failure?

I thought we were better than this at this point. Well, we'd like to think we were better than this at this point. The the fact is that we're not, uh, one of the issues that people fail to understand is that even though you may not be, you personally may not be a direct client of Google Cloud, a lot of the infrastructure that you use is a client of Google Cloud.

So that if Google Cloud goes down, even though you're not using, you know, Google meets or whatever you, you think you're using, if you're using Spotify, if you're using, if CloudFlare, if you're using a bunch of other stuff that's powered by the components of Google Cloud, you're gonna feel that effect. And so that's really what happened here. It was, it was an escalating problem that started at Google Cloud, but then quickly spun out to all of the other people that use Google Cloud services for, for things like identity, for instance.

Uh, that then brought down all of these services that, that people are used to that, that don't say Google Cloud on them. So this is not unique to Google Cloud. It's going to be unique to any of the services, the cloud services we use that are built upon other infrastructure kind of invisible to us as end users.

Uh, it just happened to be Google Cloud at this point, but it, it's been Azure in the past. It could be AWS uh, it could be almost anything, Even Oracle. So, so Mike, I, I got a few things, Ted.

com, it's not gonna work for you anyway. That's not our website. com.

But, and, and Jack, we're one of those companies that though we don't directly pay Google Cloud for our hosting, our specialized host provider actually uses the Google Cloud, cloud native, uh, format. And so it affected us. But Mike, I don't buy everything Google gave as a timeline here because we, we had a problem with Streamy Yard on, uh, Friday morning's Textron Gang, which was, we called it Thursday morning at about 10 o'clock in the morning, and it was already, stream yard was already being affected.

And, and it was pretty much down most of the afternoon because of this Google Cloud thing. On top of that, CloudFlare Flare had some real issues. Now, CloudFlare, of course, is, you know, I don't know, 26% of the internet goes through CloudFlare or something like that.

But I didn't realize they were that dependent on a single cloud provider, namely Google here. I don't know if, if the CloudFlare incident is separate and apart seems too coincidental. But I think that contributed to the overall malaise delays of, of all these sites, you know, is cascading pretty badly.

You know, it's, it's a domino effect, Alan, right? Uh, you know, once one domino falls at the Google site, it, it's just a whole series of other dominoes that are dependent on that first domino. And whether it's CloudFlare or whether it's others, whether, again, Spotify, you know, pick your favorite, how many sites use, use Capcha.

Um, you know, that's a Google service. So, um, it, it's really an issue about interdependence that we don't see. It's all invisible behind the scenes, but it's, it's really there.

Tracy, is this bad engineering or is this just unavoidable nature of the thing? Well, I think it's a reflection of how important API management systems are, right? You know, um, this was, I I think they changed some kind of a quota and their API management system that brought it down.

What a simple update to create a massive outage. So we've talked in, in previous gang, um, discussions, we've talked about APIs and we used to talk more about them, and I think we've forgotten about them somewhat. So this is a reminder how important those API management systems are.

And it even goes further than that because we still have issues with API security. This was simply a, it looks like a, a lit, literally a change that kind of like the, when, um, when all the Microsoft servers went down because somebody updated something that changed a parameter that gave everybody the, the, the, the screen of death, right? The tombstone.

Um, but yeah, api I, APIs are important and these management systems have to be treated with some care. Mm-hmm. Stacy, explain the psychology of this to me a little bit.

'cause every time there's one of these incidents, everybody stands around and says, well, that's a wake up call for sure. And then about 24 hours later, everybody rolls over and hits the snooze button and forgets about it. Um, is there something that some flaw in our souls as it people that we can't quite wrap our heads around the fact that the cloud is a single point of failure?

Or, or are we living under some sort of delusion? I mean, I think it's, it's a matter of, well, how bad was it? All right.

What did that, Rick, how bad was the pain? How bad was the hurt for, for the company that I worked for? It was okay, we couldn't get into a couple of Google meets or, you know, so what did we have to do?

How is there something that we couldn't get around? And I think for most people, okay, it was a couple of hours, and then we go on. Now, if it started happening more and more, if there, if the failure was more, more painful, then I think people start trying to make change.

But people are so reluctant to make change to try and get backup systems to try and diversify that. It's just much easier to put all their eggs in one basket. 'cause you think it's Google, they've got, they know what they're doing, they're never gonna let us down.

Or whether it's, you know, something or Microsoft or something else. And you've, you've chosen and that's who you stick with. Uh, but that to me just shows the flaw in that kind of planning that when you put all your eggs in one basket or when you rely on something so heavily, or ignore it, or just put your hand head in the sand and say, it wasn't so bad that we just go with sometimes the path of least resistance, whether we, even if we shouldn't, I'm trying to assess the total cost of this thing, and somebody will of course, naturally throw around numbers with fees in it.

But, um, to, you know, Stacy's point, uh, so I couldn't log into WordPress and I couldn't load a story for an extra day. You know, what did that really cost versus what did it really cost? Somebody like, uh, Spotify that maybe has a business model that's based totally on Google Cloud, and maybe they should be a little more, uh, aggressive about single points of failure than we need to be necessarily.

But Alan, what's your take on the math here? You know what, this is like talking about ROIs of security, right? I I never believed in them, even though I sat and did these ROI calculators that we gave to the sales team to use.

I mean, I look, you, you take your best guess, basically, and you, you know, and there's some, it's more, uh, subjective than objective. 5 x, right? And then you want to carry that across, but at the end of the day, no one's looking for an exact amount, right?

They, they want a ballpark number, usually a big number that does end with a B, you know. But if you look at things like, you know, over the years we've all read the Verizon data breach report and, and some of the other security reports, they'll tell you that network downtime roughly equates to YZ amount of dollars per hour. And, and you could use that as a, as a, you know, a starting point.

I, I don't think you could ever get your thumb on a, on a much tighter than that. But suffice to say it's a big number, it's a major pain in the butt even for us, because Yeah, okay. All of our sites were, it wasn't just a few.

All of our sites were down for a decent amount of time that it's hard to put a, a real number on it, but it definitely cost a significant mon of money. What might be interesting, just to, to build on that a little bit, Alan, from a, from a cost perspective, is if some of these folks that are so dependent on Google decide to, to go sue them, uh, you know, I could see that happening with, there are a lot of lawyers in the No, I, I, I don't think that look is America. Anyone could sue anyone, right?

But mostly if you're a Google customer, they have SLAs in there, and, you know, even with a five nine SLA, that's still a certain amount of time down a month. And if you sue them, and if you read the fine print of your SLA, what you're entitled to is Google was down for two and a half hours. Two and a half hours is about 150 50th of your monthly bill to Google.

So they're gonna refund you $2 and 53 cents. Very sorry. Doesn't, does it make up what your real losses are?

Yeah, I don't disagree, but you know, I, we're, we're probably gonna see some of that stuff anyway, just because, I mean, look, you, you saw this right? Delta, remember Delta sued, uh, CrowdStrike over the outage last year? Yeah.

So you, the thing about it is, is I, you know, I, when I see things like this happen, when we, when these come up and Google's done great, they have, I mean, I don't know if they've ever had this kind of, uh, of press in the last, I don't know, five, 10 years. They, they, they're actually a pretty consistent company to be relying on. But what if this is critical infrastructure?

You know, we, we we're talking about, you know, you might not have been able to get to a website or you might not have been able to hold a Google meet. Um, that's not necessarily, as Stacy pointed out, super critical. But what if this was our electrical grid Or a hospital?

Exactly. So how, how, what, what can we learn from this, um, to prevent critical infrastructure from having similar problems? I think that's really the que the, the conversation that we should be having.

Not all outages are created equal, and not all networks are created equal. Certainly. But what we learned, Alan, was the dependency across these systems.

Oh, absolutely. But there's a, there's a different mathematics involved when you're talking critical infrastructure, right? Over the years, I, I used to do some work when, when it was it still secure?

We used to work with the good folks over at NERC and ferc. I dunno if you guys know NERC and ferc, those are the people who are responsible for running our nuclear energy, uh, nuclear energy plants. And I will tell you, they, they have real failovers and Plan Bs and options, right?

You should, if it's truly mission critical, you've gotta have contingency plans for when stuff hits the fan. And I, I think you do have that. And a lot of these mission critical, I, I frankly was surprised, and I'm not banging them in particular, but I was very frustrated with Streamy Yard yesterday, right?

We had live events going on that I had a stream that was pretty mission critical to my business. And, and the fact that they're solely dependent, it seems, on Google Cloud. And because Google Cloud was down my streaming, my streaming ability was nil.

You know, there's another piece to this as well. I, I, I don't dis disagree with anything you just said, Alan, about backups and, and having secondary, uh, infrastructure that you can rely on. I, I, and by the way, I, I would imagine Google and, and Microsoft and AWS have that as well, but one of the pieces that is apparent to me is that at the end of the day, someone screwed up.

This was probably a human error. You know, we don't have yet all of the details, but it's very likely that someone hit something or did or hit a button, or they shouldn't have, you know, the CrowdStrike example that you gave earlier, it was a human failing. And so, no, no matter how good our systems get or can be, we, we still have to deal with humans.

And the humans getting in there and messing something up is a real problem that I don't know how you fix. You know, we, we let AI run everything, I guess, but that doesn't work either. Before we end this conversation, I do wanna call out one thing.

There is an organization that benefited handsomely from all this. It's called Starbucks. And there was more people in Starbucks yesterday than any other day recently, because, well, they couldn't log in.

Well, but even if you were at Starbucks, you couldn't get on Streamy Yard. You still had your CloudFlare, I'll tell you two organizations that benefited from it yesterday. One's called AWS and one's called Microsoft Azure, because that's how people act, right?

Oh, they're down. I'm moving my stuff too. AWS or I'm not using Google Cloud anymore.

Yeah, We went to Zoom, right? Like, yeah. You know how that goes.

But as Jack said in the beginning, you know, the let he without sin throw the first rock into the glass house. I thought the other thing in the, and some of the articles that I read when I dug down, I said that the way they fixed it is, was they bypass the per the, the, the changes that they had made. Um, instead of correcting the changes, they, they, they kind of did an emergency fix.

And reading the article, the whole thing said, said to me, why, why don't they have test impairments? Well, how did this happen? You know, internally, something really got whacked out.

Let's just say it got whacked out, because normally you would've changed these, you know, you would've updated this management API management system in a, in a different arena before you pushed it out to the entire world. So it ki it, it was a kind of a weird one when I, when I was reading what, what caused it, because it was, I don't know, I don't know what they could have changed. I have no idea.

Maybe, uh, they changed, you know, how much data could be pushed through who, who knows what they updated? But how did, how did you break that? Right?

How did that happen? Human error. Yeah.

Human error. I guess it's, I guess it's, thank God for load balancers, right? 'cause we could shift things around, But what did they do?

Like bring down the number of concurrent connections or something. I mean, Oh, yeah. I mean, I, I, I'm not saying they're lying.

I just think there's more to this story. Oh, thank you Alan. That's probably what I was alluding to.

Yep. Um, we gotta take a break. We're gonna jump onto our B block in a second.

You're watching Text Trump Gang. Hey folks, welcome back. And we're gonna return to a topic we've talked about in the past.

It's called finops. It's the idea that we're gonna take our finance people and our IT people putting together and manage our cloud costs better. com talking about there's a greater need for urgency from a consulting firm called Do It.

That is the author. And by the way, we do take contributions to our websites. And if you're watching this, we'll consider making a contribution from the content side.

We'll make you famous. But Tracy, the author, is making a case that says that there's just simply not enough urgency out at the DevOps engineer's point of view. 'cause everybody's kind of focused on just making the thing run, and the cost of it is maybe secondary.

And my question to you is, a, do you agree in being, if so, what's to be done about it? So, you know, DevOps teams have been pushed for years now, um, uh, pretty much in two areas, not even innovation, agility and velocity. Uh, how do you push things out to end users, get new features out to end users as fast as possible?

Um, which is important because that's what drives business. That what keeps, that's what keeps us competitive. Um, it, it is, it's, it what, it's what we do, right?

That's what DevOps do. If we DevOps engineers make it impossible for us to make quick changes to software. Now, when we think about finops being part of DevOps, or having these two organizations talk more, which they do, by the way, we're now talking about more accountability.

And sometimes it's hard to be completely accountable when you're trying to move fast and agile. So we're shifting Now, if this is the case, uh, and I do believe that we should be adding more to the DevOps process, we should be thinking more about security. And this is another part of the finops discussion.

You know, what is operational impact? What is the change going to do? How, you know, is there some regulatory issues that we should be exploring?

And what does it cost? This is something we've really never thought about in DevOps. It's true because we have been pushed by upper management to get fixes out fast, not to be accountable for the cost of it, or even regulatory compliance in, in many ways.

We talked so many years and years and years, we had this, this discussion about separation of duties. And we tried to work on that. Now we have generating SBOs.

Sometimes we get that done, sometimes we don't. Most, most workflows don't even have that. So adding cost is going to be an issue for us.

And mainly because DevOps pipelines, as I have talked about many, many times on the show, need to evolve. It's hard to update DevOps pipelines right now when they're so scripted, and there's thousands and thousands of them. So we have to do this in a different way in order for us to be more accountable.

We're not gather, you know, Orillia and Deploy was probably the only evidence gathering store so that you can start doing historical trends and look at, uh, potentially add cost analysis in there. So we don't have the tools to actually incorporate finops and accountability into existing DevOps pipelines. I'm not saying that we shouldn't do it, I'm just saying that I kind of doubt if, if, you know, the C-Suites gonna say, yeah, put a bunch of money into that, because OPS can start managing costs.

I don't know if DevOps needs to Theoretically. So Jack, whose responsibility is it to manage the cost? Is it gonna be to Tracy's point, the centralized IT team led by maybe a CIO?

Or is it gonna be the software engineers? And I don't know, I mean, you've been at this for a while, but is there a change in the tenant of the conversation about cost in it these days? Uh, there's always a conversation about cost in it.

The bigger problem in my view is that it's really hard to retrain people once they're used to doing something a certain way to do it differently, right? And so if you are going to get DevOps folks, it folks that have been concentrating on getting things done as quickly as we can, 'cause we have to support the organization and who cares about the cost to start thinking about, well, if I do X versus Y, I'm gonna spend an extra million dollars. It's gonna be a real problem, um, to get those folks to think that way.

And I, and I think management has a role in, uh, and a responsibility, uh, on that side. But retraining is going to be the, the, the key issue, uh, going forward. And frankly, you know, the, the folks that we train, you know, the, you go to school to learn software engineering, they don't teach you about the costs involved.

They teach you about here's the best way to code, here's the most efficient use of the CPU. But by the way, they all have cost issues with them. But it's not really about how do I do this?

The, the cheapest, fastest way. Compute cycles for most companies are kind of invisible. You know, they cost something, right?

But they're invisible. There's so many of them. We don't have to be efficient at what we do with them.

And I think that's the real situation here, is how do we evolve the thinking in, in, in the DevOps space to also consider, if I do it this way, what's the cost to the company? So I, Mike, I, you know, you know, there's an old adage. You keep banging your head against the wall.

Either the wall goes soft or your head goes soft. And that's pretty much what we're up against here, right? Let, let's take it from a people perspective.

And Stacy, I'd be interested in your take. And I don't mean to just pick on DevOps. This isn't a DevOps issue.

This is an ops issue. This is an IT issue. People are under such pressure.

Get it done, get it done now. Make it fast. Make it work.

We're gonna replace you with ai, right? You gotta make this happen. They're not necessarily rewarded for making it happen the cheapest.

They're rewarded for making it happen the fastest. And so if you wanna now kind of change that equation and say, it's okay not to be so fast. 'cause I want you to be cheap.

They look at that as, oh my God, the bing counters are in charging it, right? We, we, we go through that a little bit ourselves here. I had someone write me tell you, you still using your password manager?

We can shut that down and save money. You know what f you? I need my password manager.

I'm not using I'm, I'm not going with that. When, when your quest for saving a dollar starts cutting into productivity, you got problems. Now, I'm not saying there's not waste, but almost since the very first day that the cloud was the cloud, someone spun up an instance and didn't shut it down when they were done.

And nothing has changed since then. I've got one answer for you here. I feel like I'm talking to Dustin Hoffman in the graduate.

Got one word for you. Automation. We need to automate finops so that it's not necessarily the Porsche Schook DevOps engineer or the ops person who's responsible for, for counting the beans.

We need to automate that so it gets done in there and take it off these people's already crowded overburdened plates. There's even a bigger piece. I think, sorry for interrupting Ally.

I just wanna ex expand that. I, I don't disagree with you, but if you're gonna automate the process, you need to automate the entire cost process. It's not just DevOps, it's all of it, but it's also the end user environment, right?

You need to, you need to, to, to, to follow the, the dollars all the way through. You know, what I think of, of, of concepts, you're giving these developers what feels to me like a ball and chain and saying, here, run with this and be as fast as possible and as agile as possible. And it's like, if you're, you're carrying a bunch of weight and you're, you're not as able to be that nimble, it's going to slow you down.

It's going to, to run into that ability. And they're also not necessarily exercising that muscle of thinking that way. And so when you say automation people, it's good to to, to train them to have that, that, uh, idea, whether it's a notification of here's your budget and you're exceeding it, and, you know, what does that look like?

But again, it comes down to that, that pain point and what, what's prohibiting you from moving forward in the way that they're designed to, to do their job? And are you holding them back into to what's the advantage of that? What are you really paying for at the end of the day?

And does the, does the cost outweigh the benefit? Not always. I'm not entirely sure I agree with, I'm not entirely sure.

I agree with the premise that I have to have a trade off between fast and inexpensive, or at least some modicum of cost control. But these things are not mutually exclusive in my mind. But Tracy, Well, the thing about it's is that we're already managing costs.

We don't just not look at cost. You know, IBMA couple years ago bought cube cost, right? To track the cost of, of containers, uh, and their Kubernetes configurations.

But that data is not connected to the DevOps pipeline. So, and, and most of these DevOps, CI/CD tools, they're, they were, they're, they're job schedulers. They're just job schedulers.

And they have a log that goes with it. They don't have the ability to track data. They don't version data.

They don't even give you historical references other than where a log might be. And if a build was successful, that's what they, that's what these tools are doing. So we will fix this.

I believe that MCP servers are coming faster than, um, than we all imagine. And I do believe that DevOps and, and CI/CD as we know it, will be replaced with MMCP servers. So that, right now, today, if somebody, uh, is using cube costs from IBM, they could connect it into their DevOps pipeline.

So for every time a container gets pushed out, that version of that container, the cost of it is being associated right to the DevOps pipeline. It will happen, but we can't do it right now with our technology. And it's not that we don't want to, it's just not possible.

It literally is almost impossible to manually update all of these scripts. It cannot be done anymore. Plugins were, are, are a huge, huge problem in the DevOps world.

And to try to build a plugin for, uh, every time you wanna manage something or add more accountability like an SBO m or trying to track the cost of a container, and it's, and it's test or production environments, it's not, it's not possible to do. So it's not that we don't want to, it's so we don't have good tooling right now to allow us to be that agile ourselves. We've been, we've been supporting agile development, but we are not agile DevOps engineers.

Our, our DevOps pipelines are not agile. So I want to see this happen, and I will, I, I do believe that it will be, um, plugins will be replaced with the MCP servers, and we'll have context windows that will connect the pipelines, and we'll have workflows that AI is managing. And then we can say, I'm gonna add, I'm gonna add cube cost to my pipeline and it's not gonna impact my pipeline.

I'm not gonna, I'm not gonna take a chance of breaking my pipeline by adding this one particular connection. I think we may be too much in the dirt here with, with this discussion. So what I mean by that is, look, I, I, I don't disagree with anything you just said, Tracy, but at the end of the day, if I add a component, whether it's a, you know, a, a a cloud component, a local component, and you know, a bunch of code, what I really wanna know from a finops, uh, uh, uh, view is how does that affect my entire business?

It's not just it anymore, right? If I add a component, does that increase my sales? Does that reduce my sales?

Does it make my people more productive? Does it make them less productive? I'm not talking just about the dev folks, the DevOps folks, 'cause they're certainly, that's important.

And, and managing that cost is important, but you really wanna look at it from a, a more holistic viewpoint and, and just tracing, uh, you know, the Kubernetes components isn't enough. It's how does that o overall affect my business? That's a really hard problem to solve, but that's where the real benefit would come to most organizations.

In the meantime, we're gonna get a bunch of AI agents that are wagging their fingers at us every time we overspend. Right, Right, right, right. Or just turn off your machine, you know, the agent should come in and say, sorry, and the, the quarter you put in the slot, you know that, that time is over.

Can't use it anymore. Something to be said there. All right.

Hey, let's take a break on this one, guys. I think we've, we've kind of covered it all. We, we'll take a break here on the gang.

We'll come back and let's go to our C blocks. You're watching Text Trunk Gang. Join Cruise Con Virtual on June 17th in 2025 for breakthrough strategies to address advanced threat intelligence, proactive incident response, exclusive bonus material and regulatory adaptation here from our keynote speaker, Admiral Michael S.

Rogers, former director of the National Security Agency, and an outstanding lineup of industry experts as they navigate emerging threats. The core principles of crisis management and the evolution of CISO Leadership. Register now for free.

Alright, we're back in. We're gonna talk about something that matters to everybody. It's called money and salaries.

And once you get paid for your job, and we have a pair of posts. com, talking about, on average, the DevOps engineers are making about 140 KA year. Now, that varies widely depending on where you are in the country and what kind of job you have.

So the average is just that, an average. com talking about how well platform engineers are making more than your average DevOps engineer. And maybe this is the new job title to get and go after and have.

And, um, this whole space seems to be evolving. Alan, I know you covered this space closely, but what do you think's going on here in terms of the hierarchy of IT folks and who makes what? You know, it's interesting.

org site, which is a huge community site. Yes, platform engineers is the platform engineer is a new hot title, and people are paying a premium for it. DevOps engineer.

And whether you believe that that's the right title or not, but DevOps engineers have had surprisingly great staying power making, also around one 40. There's not a huge gap between the platform engineer, and I think that was one 50 maybe. Um, I, I think the, the, the fate look, they're both pretty well compensated positions.

I think the bigger issue is how many openings are there for platform engineers? How many openings are there for DevOps engineers? The interesting thing about platform engineers, it's a new title with an old job, right?

We used to call these people CIS admins. We used to, you know what I mean? They, they, uh, be it ever, no matter how it smells, it would still smell well, no matter how it looks, what it still smell is sweet or whatever the right Shakespeare quote is there about roses.

Uh, you know, we put a new title on 'em and we pay 'em a little bit more money. The bottom line is tech engineering roles are in demand still. AI hasn't quite replaced them all just yet.

Uh, I think it, you know, you're gonna have ups and downs and waves depending what's in vogue. And certainly platform engineering's in vogue right now, but I I, I don't think there's a big gap there. I think the bigger issue is how many openings are there for these roles, and can people get those jobs?

I know a lot of my friends, not only in the DevOps and the platform engineering space, but especially in the cyber space, which has also been traditionally a full employed kind of vertical. A lot of my friends are having hard times finding jobs. They see, I, I know you've dealt with this, and a lot of your friends have as well, no matter what they're paying, are there enough jobs out there?

I mean, we're told there's all these openings, but yet who are, it's taking people eight, nine months to find a job. Yeah. Well, and some of that is, is what the companies are going through.

What's the hiring process? You know, this is it. What are the perks around it too?

This in-office work? Do they have to come into the office to work? That's, that's one that's, uh, strong, is finding the jobs that are a right match out there for people.

And there are jobs out there, but really it's like you have to know someone. You have to apply over and over again, and it becomes really disheartening for people. So there are the jobs out there, but it's, they're just really hard to find.

And then to make yourself stand out as a candidate and to make sure that you're getting your needs filled, whether it's being in office or working at home. I see the working from home piece that comes up more often than not with these, with companies being asked to bring people back in, whether they need to or not, because especially I think for developers and engineers, they can do most of the work from home. And so people will sometimes turn down opportunities.

The opportunities may be there, but if it's not fitting either their work life balance or the, the needs that they have in their day-to-day life, then they're not being considered or they're being considered, and then the wrong people get put into them and they come up again causing churn and causing issues internally. So the, the jobs are out there, it's just, it's hard finding fit and it's hard standing out. I think there's another component to this as well, e especially in the discussion we just had around, you know, platform engineering versus, you know, cis admins versus, uh, you know, developers.

And that is the whole notion of presence. Do I have to be there, um, to your point, Stacy, but to one more. I I think there's, again, another component to this, and that is that for DevOps, for instance, if I've got software to develop, I, I can, you know, farm that out to some group in, in India that'll work for quarter of the, the cost and, and still give me my software platform.

Engineering is a little different because you really do need to be onsite. You need to be present, you need to see what's going on from the hardware perspective, the networking perspective, the integrations perspective. And so, uh, a and that's, that's also reflected in the salaries, uh, because if there's more competition, salaries go down, right?

Uh, it's, it's, you know, it's just normal economics. So it's a, it's a pretty interesting discussion, um, about what does it really mean that there are jobs available, but people aren't getting hired? A lot of that I think is, is a discussion around, do I outsource it or do I actually hire somebody here in the states?

You know, the pol political ramifications that we're going through right now aside, I think that's still a major discussion in a lot of companies that I see having that, that, that that thought process going through. And it takes some time for them to, to get through it. So the, there's another human factor here.

I'd just like to mention both these types of jobs, DevOps engineers, and even more so for platform engineers, have a level of stress that can be not so fun. Because I've sat in these, both of these jobs and, you know, the whole term platform engineering, um, is the idea of bringing into the DevOps process or bringing DevOps more into creating these, um, self-service environments. So development teams can get onboarded quick more quickly and many other things.

How, how to better streamline your production environments. It's kind of an expansion of the site reliability engineer role in my opinion. But when you're talking about DevOps, this is an important role.

It's compiling and making sure it's, it is building and deploying very, it can be very stressful, it can be very hard. It can be, it can be frustrating. And platform engineering even goes farther into managing these live environments.

And our live environments are becoming more and more complex. So in terms of these two jobs, uh, it doesn't surprise me that you can, that they're, that people are hiring people and more remote, uh, situations. It doesn't surprise me that the salaries are sitting.

I think that it showed between 140 and 200,000. 'cause they're critical jobs and they are stressful different from a software developer. So if you're, if you're out there looking for work, there are jobs out there.

But one of the other problems that we're having right now as an industry is many dev, many companies and the Art Tillies community have just been doing this job seeker, uh, series. Many companies rely on LinkedIn now to do their hiring. They use LinkedIn as a primary platform and software developers, platform engineers and DevOp DevOps engineers.

Were not storytellers. And your LinkedIn profile has to have a story. You have to look interesting, you have to get noticed.

This is a real problem for, for, for engineering types. We're we don't do that. We're very dry.

We wanna put some five bullet points out there and leave it at that. And they're all acronyms. So if you are out looking for a job, look to services that help you create and build your LinkedIn profile.

It's not just about your resume anymore. Make sure you have references, because every single person that's listening here, that's been in this industry and is looking for work today have, have people they know they've worked with. Get you references up, make sure you have something that pops on your, on your banner.

It is a silly thing to ask for somebody super technical and has been solving problems for big companies for years and years and years. But it doesn't work to have a list of acronyms. So that is my big push for you today is spend some money on your LinkedIn profile if you're struggling with, uh, uh, finding work.

Yeah, no, I mean, you know, good advice, Tracy. I, I, I will tell you, I'm of an age where, you know, it was always the, uh, headhunters who would call you up or the exec, you know, search people would say, Hey, you're interested in, I've got this great position, blah, blah, blah, blah. Or if I was looking to hire a a for a position, I'd go out and find someone who's gonna go find and, and it's expensive proposition.

Well, that's been totally replaced, I think today with the LinkedIns of the world. But, but as you said, not everyone is that savvy to sell themselves on LinkedIn. And, and you know, there's probably, there should be a course, there probably is a course on that somewhere, Stacy.

Yeah, there's lots of people it that, um, that specialize. Even just helping people with their LinkedIn profiles. I, I struggle with it.

I'm not a big poster, you know, online or, you know, I don't post to my socials very much. My presence is difficult. And I was submitting for a, a publicly, well, how many followers do you have on, on Twitter?

Sorry, I can't call it X, but still X Twitter, whatever. But how many followers do you have? And I thought, well, what about the content of the proposal?

No, it's about my followers. It was all about social media presence. I said, that's not me though.

I'm not out there spewing information and posting everything. And so I had to kind of think about that. Like, how do I, you know, who am I?

And, and I am not somebody who's comfortable just posting every day and putting myself out there, but I watch the people who do, and, and they do, they get noticed. So you have to be strategic about it. And I, I love that advice.

It's like, how do you put yourself out there in a way that strategic stays noticed, but also you're posting something that's meaningful or you're designing your, uh, your own profile in a way that's meaningful, has the information, has the titles, it has the recommendations in a way that, uh, doesn't seem too cumbersome and represents who you are. And yeah, there are companies, there are people that can help you through that. And it is worthwhile to get it.

'cause there are, there's little tricks to it. There's little ways to put things in there that can be more effective. Uh, and it becomes back to that networking and to people know that you're available and what you're looking for and who you are, so that you stay in someone's mind when, when something comes up.

Oh, I know, you know, somebody that, that can, that does, does this, you know, DevOps or platform engineering. I know somebody or I've seen their posts. Yeah.

And just a tip on that. One of the recruiters said, or actually two of 'em agreed, don't put open to work on your, on your LinkedIn profile. That was one of those little tricks that I went, really?

I know, I know. And most of the people who were on this webinar that were looking for work had had that listed, had they were posted that they were open to work. So don't do that because then it, it, it, it turns off the, uh, the recruiter, they won't even, they'll just bypass you.

So student, I mean, who's to say who's, what is my question? It's like, so I'm a DevOps engineer and somebody posts a job for a platform engineer. I'm like, sure, I'm a platform engineer.

Who's to say I'm not? I think you gotta look at what the, the requirements for the job bar. And, and that brings up something else.

You know, I love when you, you know, looking for a platform engineer with 12 years experience. Uh, you know, platform engineering's been a thing for three. Um, so, and, and we had this with DevOps too early on.

DevOps engineers 10 years experience. Well, DevOps has only been around five. Uh oh, and by the way, you have to have a PhD and two master's would be next.

Um, is a bigger problem in our tech space. We see it in cybersecurity as well. Um, you know, does experience trump education?

Are we being realistic in what the prereqs for a particular job are? What happened to just strong oral and written language skills? Right?

I I still think that's communication skills is very, very important. Uh, if you're teachable and coachable, I, I can make you anything just about, so I, I then my 2 cents right there, A lot of companies are dealing with recruiters too, who may not be necessarily technical. And so coming your point out, it's like if you, there's a checklist of things that I'm looking for.

Okay, yes, there we go. That checklist, the, the degree that checklist. And sometimes that's what recruiters are looking at.

They don't necessarily understand the nuances of what the actual hiring manager may need, though. They work their best to do that. And a lot of the, if you're just applying for jobs off of websites, you get lost in the batch.

Like when I've hired for roles within three days, I had over 300 applicants, and there could have been the most amazing applicants in that pool. I didn't have the time to really look and go through them, and quite honestly needed the recruiters so much. So that's also an upfield battle with some of these highly technical roles that are also nuanced in ways that the front door doesn't recognize.

I, I would just say we're all great communicators. Now. We have chat GPT, right?

Exactly. Exactly. All right.

On that note, let's end this. Um, guys, great discussion today. Tracy, Stacy, Jack, thank you so much, Mike, as always, thank you.

I hope you've enjoyed, this is a great way to kick off your Monday. As usual, we have a full text on TV lineup immediately following today's show, so stay tuned for that. com.

Um, you can also download it on iOS, Google Play, Amazon Fire, apple tv, Roku, or what have you. Or on YouTube, maybe a text on tv. But no matter how you watch it, thank you for watching it.

We'll be back tomorrow with more gang. But for now, this is Alan Shimel, we're out. Hey, everyone.

Welcome back here to Tech Trunk tv. Our next guest is Rajeev Singh. Ra Rajiv is the Senior Vice President for NTT Data.

Joining us today, Rajiv, welcome to techstrong tv. Thank you, Alan, thanks for having me here. Looking forward to With you.

I have our pleasure. Thank you. So Rajiv, let's talk a little bit about you first, right?

You, as I mentioned, your SVP over at the NTT data. Is there a particular portfolio that you're in charge of, or, you know, and give us a sense of how you came to be here. You know, not, not here on text, drunk TV, obviously, but you know, in your role at N-T-N-T-T data.

What, what's kind of your journey been like? Yeah, so Ellen, thanks for asking that. I'm, I'm actually the group senior vice president for, uh, what we call as applications and BPS.

It's a global practice, uh, in entity. As you know, entity is a $30 billion company. Uh, we've got 200,000 employees and we've got a portfolio which spans the entire stack.

So that's why we call ourself as full stack company. And out of that full stack, I run the go to market solutions and offering for what we call as a global practice. And the global practice that I'm representing is application and BBPS business process services.

Sure. We are on the upper end of the stack if you, if you want to visualize that way. Um, my journey, I've been in this industry for, um, over 25 years now.

Uh, I'm an electrical engineer by, by, um, by education. Um, I did my electrical engineering and MBA back in India. Um, I have worked with, uh, many companies outside of IT systems as well.

I started my career in shop floor and other, other places, uh, have been in IT industry for well over 20 years now. Uh, been with NTT data since 2011. I came in as part of the Dell Services acquisition.

Um, and then, you know, have been with applications for, um, gosh, about 10 years now. Right? So, um, so that's my background and I'm really looking forward to this conversation because while I'm here talking as an application, BPS, uh, senior Vice president, I mean, we obviously, the entire ecosystem that we operate is full stack in nature.

And as you know, we are one of the largest, uh, backbone providers. We've got about 45% of internet traffic that goes through us. We are second largest IT managed services player in the market.

We are the third largest data center provider. I mean, I can go on and on. Uh, when we talk about a full stack, we've got 200,000 employee base.

Uh, we operate out of 50 plus countries. So massive scale that you're talking about. And we are present in most of the vertical industries that you can, that we can think of, right?

So that's our global scale. And with that comes a lot of opportunity to, uh, play in this new world of ai, which is actually, uh, having a profound impact. So that in short, is, is where we, where I'm placed in the organization.

Love it. I love it. So, Rajiv, I, I hope that gives people a sense of, you know, a lot of people, uh, look, I'm of a certain age, right?

NTTI grew up in the world where at and t was the Mabel, right? Was the US telecom provider. NTT was Japan's telecom provider.

Deutsche Telecom was West Germany back then, right? There was no just Germany, it was West Germany, east Germany. And now, you know, obviously the world's changed at and t is not what it was, but it's yet greater than it was perhaps NTT, you're still the national telecom provider for Japan.

But NTT data, as you said, operates in 50 company in 50 countries, right? It's a global powerhouse providing bandwidth, internet traffic, data centers, everything else. And now we come to maybe the next great era in it and in maybe, maybe as profound as the internet itself, right?

Having that kind of effect on, on humanity, on civilization. And that's this AI thing. Um, and the race is on, right?

It, this, you know, in the, in the internet era, Rajiv, we had the dot coms, right? com business. In the AI world, there's a lot of that.

There's a lot of people starting AI businesses. But when we look at where a lot of the action is in ai, it's distinctly with the big guys, with the NTT datas of the world, with the Googles and the Microsofts and, and so forth. Um, and the reason is, is it seems to have, well, I think one of the reasons is there's a high bar of entry.

You've gotta put a lot of money, a lot of resources into having a state of the art AI type of, of of infrastructure, though, you know, the people at Deep seek the, the Chinese have shown it, well, maybe you could do it. And, and they, you know, jury's out on that, on whether you could do that cheaper or not. So we look at NTT data and say, that's a company that has the, the horses to play here.

You guys recently announced a whole AI agent ecosystem, and you are not the only one. Salesforce has announced it. ServiceNow has announced a similar ecosystem of agents.

Uh, agent AI is some call it. Um, you know, a lot of the big guys are the big players. What makes yours unique?

What do you, what do you, what are you guys talking about with it? Yeah, no, Alan, that's a, that's a great point. I mean, um, AI I think is a seismic shift in the entire ecosystem.

com and the internet boom because of its profound impact across industries, right? Um, you know, we are playing AI quite deep inside first, obviously, like I said, we have 200,000 employee base and we had a lot of responsibility inside our company as well, right? In terms of retraining some of the folks and building a workforce, which is, uh, AI ready.

But let's talk about, you know, what is smart AI agent ecosystem that we have built and we have announced, uh, in this world of ai, you are right. I mean, a lot of other companies are doing a similar announcement. What we are targeting basically is that, look, AI is going to be a big leverage for productivity and efficiency and all of that that you hear, but Agent AI in itself has the potential to rewire workflows and thereby, you know, um, evolve a new way of doing things.

The potential is pretty big. And I think for us to make that real, we started obviously working, uh, inside our company a couple of years back. We started experimenting in different processes, for example, impacting our HR systems, impacting our marketing systems in a positive way.

We have slowly built an ecosystem of agents, some working internally with among our groups, some working with partners, partners like, you know, partners like Rafa, system partners like Core ai. Um, we, we've also entered into a deep alliance with OpenAI. We are forming a center of excellence.

We are building agents with them. We have relationships with Hyperscaler. We have a deep relationship with Salesforce too.

You mentioned the name, right? And we are building agent ecosystem over there. So, uh, end of the day, the idea is to create an agent ecosystem, which can help impact and make a positive impact to our customer base, our, our dear clients that we have, um, full stack.

I mean, some of these agents are going to impact their business workflows. At the same time we have an agent ecosystem, which is going to impact the operational side of the it, uh, you know, as mundane as helping a service desk agent as looking at contact center from a different point of view. So we are looking at all of that ecosystem for our clients because we believe that it is profoundly going to impact not just productivity, but even answers to global, uh, workforce shortage, for example, looking at reinventing business processes.

So, so our entire ecosystem is built on that foundation, Alan, we've got, we've got an inventory of agents and we are also looking at relationship ecosystem around that. 'cause obviously this requires deep relationship with different, different partners. And that's what we are trying to offer to our customers, that there is a, there is NTT data, which is a known name in this industry, which has an arsenal of agents, which has the potential alliance and relationship.

And we are working with our customers from advisory all the way down to building as well as management of these agents in a very responsible manner where security and compliance in ski and it's part of our philosophy, if that makes sense. It all makes perfect sense, Rajiv, perfect sense. You know, philosophically, I think one of the realizations that people need to come to is that not all your agents are going to come from one company.

I mean, you mentioned it, right? You call it an ecosystem. There's big companies, small companies, they're all producing agents.

And, and to me, the real prize is being the company that helps people manage all these disparate agents, right? You don't have to, no one company is gonna make the agents for everything. I think we're gonna live in a world where we all use agents at different companies, but you know, who, how do we manage them and who manages them?

I think that that's the big question, is that NTT data end game here That Is absolutely helps them manage. That's, thank you, Alan. That's absolutely, uh, one of our key focus areas because as you rightly said, most of the hyperscalers, as you know, as Azure, for example, they have their AI foundry, they have their OpenAI services.

We have a deep practice, which is just focused on building agent tick factory using these hyperscaler technologies. From a customer point of view, it is my personal belief that large enterprise customers are going to look at agents as loosely speaking. We used to talk about custom apps, right?

I mean, so you'll have a portfolio of an environment where there are a lot of agents, there will be specialized agents coming from different areas, and off late obviously you've got agent to agent protocols coming in for seamless communication, basically handling the interoperability portion and entity is a, is a measure player in that segment because we are going to operate this entire agent ecosystem for our enterprise customers trying to get them the right behavior. Now, when you look at agent to agent interaction, it has profound impact on even a term like observability. You know, that term itself is going to change because you Loaded Yeah, I mean, you're looking at an agent interaction, which is now you are looking at conversations and behaviors.

I mean, we are, we have, we are, we are working on, um, accelerators to actually manage those kind of environment. So when you look at, let's take a old school managed services deal in that deal is going to morph where agent ecosystem is going to become part of the deal. So your entire observability is going to change, and we are actually working in those kind of nuances, basically, because it is an important ingredient of, you know, handling a com complex, multi-agent environment, if that makes sense.

It may, it makes perfect sense. And this, that, that is exactly the point. Yeah.

You look at something like observability, what, what, you know, observability is gonna be turned on its head when, when AI is done with it, right? It takes a lot out that Rajiv, as you can imagine, we talk a lot about agentic AI and stuff here at Techstrong TV and on Techstrong, you know, on our sites. And one of the things we hear from our audience is how much of this is pie in the sky?

How much of it is real? Right? Where, where, like what, what's available now, you know, and, and obviously this changes day to day, week to week, but you know, when you look at the NTT data, uh, this recent announcement you guys had with smart ai, AI agent ecosystem, how much of it is real and available today?

Yeah, Alan, it's a very good question and I'm, I will probably answer this question with couple of client examples and, and talk about little maturity graph that we are going on. So, um, we've got some interesting engagements. I'll give a few examples.

You know, we are working with highest, uh, highest rail materials or you know, from advisory all the way to defining a multi-agent ecosystem, which is outcome oriented. But take an example. There is a global automotive giant that we are working with.

They were having issues like on their assembly line, they used to have probably like 20 issues in a week. And these issues were on the assembly line. We actually started building a, they, they called upon us because we have a relationship with them, and we brought in what we call is a manufacturing AI solution, where we initially started with consuming all their artifacts, their operational document, their assembly line documents, the repair guidance.

We built an ecosystem of a knowledge base, which was completely driven through vector databases and LLMs and all of that that you heard. We got that to an accuracy level, and then we started layering a very intelligent chat bot, which is not exactly a chat bot. You can have a conversation where the workers can have a conversation to get repair guidance.

Now you are talking about a mean time to repair, really, really coming down because the SME knowledge is becoming a bigger, wider spread, right? You're, you're sort of making that knowledge flow into the ecosystem. The next steps that we are working is on that repair guidance to trigger agent flows.

For instance, if you have to order something, can it go and order on its own? And those are becoming a reality. The second example I'll give you is, as part of our smart agent ecosystem announcements, I, I'm pretty sure you notice that we have built an accelerator.

What it does is it takes legacy bots and it converts them into agen applications. AgTech, right? Um, the challenge over there is, you know, legacy bots follow a simple workflow, a linear workflow, a true ag agent system is going to follow sort of undefined processes.

And that's the effort that we are doing to build those kind of ecosystems into our solution. Take an example of a Fortune 50, a large healthcare company. They actually cater to a lot of these managed care products and government services like Medicare, Medicare, and we are operating with them where we have built a complete clinical and claims intelligence platform, and we are building agents which drives things like pre-authorization drives, things like medication adherence.

So it is starting to become reality, Alan. We still have nuances where these agents have to be closely monitored and captured and monitored carefully, but there are simple workflows which are converting. We are trying to disrupt their contact center industry, working with a partner where the partner is an expert in conversational AI from an agent point of view.

So we, we, we, we've embedded ourself in insurance and health insurance and card payment industry. What we are seeing is that as we are building these agents, these agents are, are very flexible in nature and they're able to handle a lot of different question. One agent can handle a lot of other activities, which you would probably code using multiple legacy bots.

And we are starting to see a lot of success as this technology matures. So that's an example of how it is going to start to disrupt this industry. An entity is playing in the forefront of that.

So, Alan, it is coming. I, I, the rapid expansion is taking place and people, there are folks who are still apprehensive, and I'm playing this from inside. I actually went and took a degree personally three years back, you know, I, I live in Texas and I went there and did a degree.

It took me about nine months to really play from inside. And I can tell you that people are going to be really surprised if they're not on it. It is coming.

I love it. I love your passion. Yeah, that's great.

Rajiv. Hey, I promise I know you have commitments and we were, we're running up on the top of the hour. Thank you so much for coming on Techstrong TV and sharing with us.

I'm gonna hold your feet to the fire, invite you back on here in the next couple months as this more and more becomes real and keep us posted on this. Okay? Thank you, Alan.

Appreciate you having me and I look forward to our next conversation. Alright, thank you. Rajeev Singh, senior VP NTT data here on Techron tv.

We're gonna take a break. We'll be right back. Hey guys, thanks for throwing, we're here with Punkish Patel.

He's the CEO of Nile. And we're talking about infrastructure and how it's gonna evolve in the age of AI because, well, maybe it's just not enough to slap an AI agent on top of this stuff and call it done. Hey, ponage, welcome to the show.

Well, thank you. What is going on here? Because everybody you talk to is saying, well, I'm gonna put an AI agent in front of my existing infrastructure, whether it's networking servers or storage or whatever it may be.

But the question then becomes, well, do we actually need infrastructure that is designed to natively run those AI agents from the ground up? Or is it enough just to try to train something on a legacy platform and hope for the best? What's going on here?

Well, I I certainly believe that it needs to be, uh, done ground up, which is what we have done. And there are reasons why I believe in that, because, you know, this is, this is an industry I have been in for 33 years of my life, uh, of my very long career. And, uh, 22 of them were at, uh, Cisco, um, you know, being in charge of, uh, the largest engineering and marketing organization, uh, in the world.

And, um, you know, when you, when you built all the products and you delivered and you see, uh, the complexities that it created over time, uh, and, uh, what the customers had been asking about, which was to truly simplify the universe and to give them something which is not only insanely simple, but also very secure. And in order to do that, uh, uh, automation, uh, plays a very key role, and AI plays a very key role. And, uh, and, uh, today when you hear about, uh, every organization or vendor trying to integrate AI into, uh, the solutions that they present, it's dominantly aim from the operations point of view.

And, uh, the goal is to lower the cost and optimize the efficiency, you know, but they truly do not, uh, implement the AI in any effective way. And, um, uh, regardless of how they might claim to use the ai, um, uh, on the legacy stack, because, you know, you do run into limitations on the legacy stack as to what you can do. And the closest analogy I can give you is, uh, uh, somebody, uh, working on a new, uh, EV to try to take our old Toyota and, and, uh, put the battery and slap a bunch of sensors on it or jobs back in 2005 and six, uh, could have started with a Blackberry phone, which we all loved and admired back in those days.

Uh, if he could have just, uh, uh, slapped the user interface on top of it, would it have worked? I, I submit to you, we won't have the iPhone today, right? And it's the same thing.

I think we are seeing a repeat of exactly the same phenomena. Uh, you cannot, you cannot, uh, slap this on the patchwork of technologies which have been created over four decades and, uh, try to think about it, that this is going to give you that, uh, a vehicle. I actually give you a couple more data points, which is, you know, I mean, when you look at the legacy of each one of these companies and, uh, and you age when the software stacks and the entire vertical stacks were created, the products are created in, uh, patches in islands.

There is no cohesion between, uh, uh, the things coming together as a solution. And, uh, what what that would end up in, uh, offering is a patchwork of solution without any true, cohesive data model. Consistent, giving you holistic design to really bring the AI principles to really apply in a very effective way, right?

And, uh, the people who, and the vendors actually who talk about, uh, uh, the AI and, uh, the cloud native, uh, I would ask the simple question to say, if you are, software stack tends to be, uh, uh, 10 years or older. Um, at that time there was not a giant, uh, notion of, uh, cloud, cloud was coming of age. And, um, uh, so number one, uh, you could not have used the CI/CD model, which all the, uh, all, all the public cloud providers use today, right?

You couldn't have done it using that model number one and number two, back at the time, how many, let's face it, Nvidia was a video game company and a crypto company, right? So I mean, like, you know, nobody talked about AI back then. So, you know, that shows that, uh, the things coming together from these companies are pretty much, uh, um, bolted as, uh, siloed, uh, result of the siloed decisions.

Whereas in our case, we truly look at this as from the point of view of the customer experience and what we had to do as a completely different and a consistently holistic design in the age where we have done this in the chat GPT universe, this is not the predictive model universe, which is where most of the vendors will be in. And this is using the true generat AI models to do what we had to do. So, geek out for me with a for a minute, what exactly is different about a, a networking platform that's optimized for AI than what I might see in a traditional switch?

All the products that you see, they're all from the traditional switches and the other networking elements, all are designed in the embedded universe and not only embedded universe, they're designed in silos. They're designed by either the org structure or the legacy that these vendors created over decades. Okay?

So because, you know, look, I mean, I was head of the, I was the head cheff of the kitchen for so long, so I know exactly how the universe evolved there, right? This is the universe that has evolved to, to give you basically or a dozen major architectures in enterprise, uh, with another 15 to 16 protocols and each protocol having from couple of hundred to thousand knobs. When you multiply all of this along with all the different software revision levels that all this products support all the years, you essentially have a snowflake in every single deployment.

Everything from the time you run it, from the time, all the way from day one, all the way through day two and day plus end, also from the point of view of when a problem occurs, how you end up in debugging and solving the problem, right? When you run into the security, uh, bug and you have to create a piece of patch or any other patch to address the patches, it'll take customers weeks and months and quarters to do that, right? Because they all are developed using the embedded methodology and not in the cloud native universe that they all were designed, they were not thought about a, a simple holistic design.

We have, we have designed the entire Nile service block, by the way, which consists of, for the campus networking as the access point, very high end of access point, very high end of the access switch, very high end of the distribution switch, uh, sensors, a controller and of stack in the cloud. It's all designed very simply using the new cloud development principles and the most simplified and the most predictive design, Mike. So this is where the key differentiation comes in.

It's a very predictive design. So for example, in my universe, when we come across, which we come across very rarely, but when we come across a software bug, my ability to, number one, to identify the bug root cause it, and by the way, to propagate that bug fix throughout the entire embedded base, I can do that in literally no time. This is the advantage that it gives us.

And by the way, that advantage propagates in the world of ai, it's no different, um, in terms of what we end up in doing. So it almost sounds like to me, um, the legacy platforms to your analogy, are based on more of embedded stacks, and there's a lot of, um, code that runs in those stacks on those devices. It almost sounds like you've taken a cloud native approach where I'm using technologies or, uh, that are consistently applied across all those switches.

So I've kind of got a little more separation between hardware and software, but then I'm also able to pull telemetry data off of that more consistently and then expose that to AI agents to automate workflows. Is that fair? Absolutely.

Absolutely. And by the way, in our, in our university, I think this is where, uh, one of the example I can give you, Mike, is that, uh, you will hear from the traditional vendors that they have developed tools where they can identify, they can identify, for example, uh, oh, a missing vlan, and they can, their tool will identify that, and the the tool can help them fix it. Our approach is don't have the VLAN in the first place, okay?

We don't, okay, because we eliminated layer two entirely, completely. Same thing when it comes to security. I, I would submit to you that over 98% of the network today are nothing but a complex collection of the legacy technologies duct tape together, which actually creates a giant surface area for exposure for security attacks.

And we have seen that many places, I don't have to name the casino, but you know, remember a very last casino many months ago, uh, when somebody was able to get through the first thing, you know, this is where, uh, some of the VLANs, uh, when they're left unattended, uh, or where somebody provisioned something on that wheel and forgot about it, and later on somebody said, oh, somebody, somebody did something horrible to that, that happens, by the way, all the time, right? So this is where what we have done and what we have built, uh, from ground up, it gives you a very unique advantage. So in our universe, going back to the example of the missing million or whatever, right?

We have automated it completely. So not only we provide this as a service mic, where, uh, on one hand it is like public cloud provider, like, you know, AWS Azure and GCP for from the point of view of the consumption model flexibility and the agility. But on the other hand, it's also like Wemo, which is day two, after it is, uh, after it is erected and stacked and deployed.

By the way, all that process in a universe is insanely simple. I mean, like, our gear is as easy as pulling out the iPhone from the slick white box from Apple, like very simple to no configuration, easy to install, easy to bring it up. And once it is brought up, right, there is no human touch involved.

You don't need MSP or it to run it. This is where it's like Waymo, I mean, you tell it that you want to be dropped off at 1628 Market Street, then bingo, it will open the door for you, and here you get out, right? That is, and, and you and I, we both know that when you do not have the humans tinkering with it, uh, the gear, uh, because, you know, the, the old estimates from Gartner and others have been that over 60 to 70% of the attacks originated from the network surface because it's very porous, right?

So this is what we are able to prevent. So not only the value proposition of, uh, performance guarantee, incredible security, you know, uh, and, uh, saving people a ton of money and more. I mean, we are, we are able to provide the true zero trust security, uh, through, and by the way, and the kind of AI automation that we have done, it provides a very new age compelling value proposition.

It takes customers into truly the new, new age of ai, honestly. So now there are, I don't know, billions of network switches and access points that have already been deployed. What's the math for replacing all of that stuff versus, you know, just kind of moving along and trying to add a layer of AI capability on top of it and hoping for the best?

Yeah. Well, in our case, uh, Mike, uh, that's a great question. Uh, in our case, uh, um, uh, we require, uh, the entire stack to be deployed.

Uh, so, uh, by the way, having said that, I would say this, pardon me, that, uh, over 80% of our deployments tend to be Brownfield. And I'll explain to you where and how we succeed, uh, but we, we deploy the entire stack. You cannot take, uh, a piece of hardware from us and slip on a piece of, uh, software from us and have that work, mainly because we borrowed the principle from Steve Jobs, which was to have the entire vertical stack because we needed to access every single knob that we possibly can, so we can automate to the nth degree, uh, which is what we have done.

So, uh, in a Brownfield environment, uh, the way we'll deploy it is that we will, uh, start with a floor and they will do the entire building, or we'll start with a building, do the entire campus. We have done many of them, uh, like that Greenfield is obviously much easier because, you know, there's nothing there. But Brownfield, we have done a lot, and people do it for various reasons.

People do it because they, they, they are having, uh, challenges on performance guarantee. People don't want to deal with the lifecycle management. People want, uh, zero trust security.

We have done some giant deployment. We have done some one very large deployment in New York City recently. We told 1800 access points and several hundred switches.

It was mainly driven by two factors. One was absolutely zero, trust, security. And by the way, they removed everything that they had before, and they deployed everything new from us, right?

So security was front and center in their mind. And the second thing was, by the way, Mike, they gave us one of the largest contract, uh, a typical contract stand to be about five years. This one was a 10 year contract, and they gave it to us mainly because the CIO was able to tell his CFO that what if I give you a 10 year predictable network spend model?

Wouldn't you love it so that I don't come to you every couple of years and I say, I need a new switch. I don't need two new access points. Or, you know what, my maintenance contract is going to go up after two years?

He said, no questions asked one price for 10 years, and the CO said, I love it. So these are some of the things that come into play As I listen, it starts to occur to me that maybe the architectures that we've had from yesterday year are designed in such a way that, well, we wind up increasing the total cost of it because we have to layer on all these different capabilities and things. So maybe the time has come to redesign this stuff to reduce that cost from the get go.

I, I agree with you completely, right? And, uh, I, I will, I'll point out to, we haven't said this yet, but we should, uh, there is a large storage company and their ad, uh, uh, for last six months has been, even if we, we have simplified our stack so much, and we have created the operational efficiency that even if my competition gave away the software and hardware for free, you will still save money in three to five years. And that is the beauty of what we have done, right?

Which is from the point of view of not just saving the money, but, uh, saving the money, and more importantly, to create this unbelievable security so that, and you and I know what the cost of security breach is like. I mean, Palo Alto tracks this very closely, and they pub, uh, publish the data every year. 8 million.

And this keeps on going up and up and up all the time. So, you know, one of my board members, uh, esteemed, uh, George Kurtz, the founder and CEO of, uh, CrowdStrike always coaches me and advises me that, hey, this is, uh, security is front and center on people's minds. So that is what we have done, what we have created.

It is something instantly simple and instantly secure. And in order to create that simple and secure, we had to create it automated and autonomous, that there was really part and parcel of what we had to do. Right, folks?

Well, you heard it here. Hey, if it's a new era of software, maybe it's time for a new era of infrastructure. Makes sense.

Hey buddy, thanks for being on the show. Well, thank you so much, Mike. Thank you for your time today, and I appreciate, uh, you bringing me on the show.

All right, I'm back to you guys in the studio. Hey, everyone, welcome back here to Techstrong tv. Hey, I've got two people to introduce you to here, and we're going to hear how their companies are working together and also maybe find out a little about their companies.

They may be companies you haven't heard of. Let me first intro introduce you to Josh Shaul. Josh is the CEO of ALO security.

And Josh, welcome to Textron tv. How are you? Hey, folks.

Uh, doing great. Nice to meet you. Great, Josh, I'm gonna come back to you in a second, but let me introduce our second guest, and she's Abby Ross.

She's head of channel marketing for a company called Hydraulics. Hi, Abby, welcome to Text Drunk tv. Thank you.

Thanks For having me here. Thank you. Thank you both for coming on.

Josh, let's start with you. You are the CEO of Allure Security. How did that come about?

Tell, give us a little bit of your, uh, personal history. Gee, I'm one of those folks that spent my entire life working on cybersecurity. I was like the idiot teenage hacker in the late eighties and, and never did anything different.

So, uh, grew up as, as a developer early in my career, eventually made my way into, uh, some leadership roles, was running the web security business for Akamai for a few years, where I saw that the world had a really big account takeover problem. Uh, Akamai had great solutions for stopping bots that would take over accounts. And, uh, once we were able to do that, scammers popped up and they started stealing credentials for accounts directly from folks.

And that's how, uh, allure Security was born and, and how I ended up here. Okay. Were you there with Andy Ellis?

I was. I was Andy and I got to keynote RSA together, uh, seven or eight years ago. Sure.

Actually, I think I remember John. So I, I've also been in security lifelong pack of bango ahead on the wall too. Um, so I had started a company called Still Secure back in the day outta Boulder, Colorado, co-founded and been in security 25, 30 years.

But I, I do, I think I remember the year you keynote noted with Andy. Um, so give us that. That's it, you know, six degrees of separation and security.

It's a really small, even though it's because such a big industry, it's still a small community. John, talk to us about allure, though. What's the mission there?

You mentioned a little bit, but let's dive a little deeper. Yeah. So what Allure does is we provide comprehensive disinformation security solutions.

So we protect enterprises from digital impersonations, reputation attacks, and sophisticated mis misinformation campaigns. And there's been this enormous acceleration and scams and fraud and disinformation that's been driven by generative AI over the last couple of years. And what we've done is build AI systems that can fight that generative AI that can go out and actually figure out what's real and what's fake differentiate between the legitimate website, where you're actually should trust putting in your credentials and the fake site where your information's gonna get stolen, the fake profile on social media that you shouldn't trust versus the one that you should.

So that's, that's the battle that we've been fighting and using some pretty interesting tech to do it. You know, we're seeing more and more, and they're getting better and better. Let me just quickly ask, are you using AI to fight ai?

We are. It's the only way that you can keep up really, is to have your own models and, and, and train 'em properly and use that, uh, to scale out your capabilities so that you can keep up with what the attackers are gonna have gotten their own hands on and what they're gonna do next. Absolutely.

John, if you don't mind, I want to turn over to Abby for a quick second or more than a second. Abby, I mentioned your head of channel marketing at Hydraulics. Why don't we give us a little bit more background on you and then let's talk about hydraulics.

Sure. Well, I've also been in cybersecurity for a long time, since 2013, which is when I met Josh. It is a small community.

We both worked at Trustwave together, um, and I, I mainly worked in public relations and then moved into marketing. And since then I've worked for large companies like IBM, I've worked for smaller companies like Bay Dynamics, and then I also worked at Akamai, and now I'm at Hydraulics. And Hydraulics is a streaming data lake company.

Uh, a lot of people use us for observability. We have major partners like AWS and Akamai. And the, the reason why Hydraulics was formed was to solve the economic and big data challenges that are plaguing many companies today.

Uh, the fact that we have an explosive amount of data and it's data that has details about the data, and there's just data everywhere across all these distributed systems and services, and getting all that data together into one platform so you can use it, has been a challenge. And retention, as many people know out there, is extremely expensive, especially when it comes to long-term retention and large data sets. So our founders, uh, they created hydraulics to solve this problem.

Excellent. You know, and it is a problem. I, I think, well, you know, I always call it a little bit of the Splunk issue, right?

All of a sudden we came to the realization that we could, we could log everything and we could capture everything and we could store everything. And then we found out that we really want to afford or pay for everything. And then, okay, if we're not gonna pay for everything, what should we pay for?

And what shouldn't we pay for? And what is important and what's not important? Just because she can doesn't mean you should.

And, uh, and that that's a, that's a big issue here. Josh, I have to apologize. I think I referred to you as John earlier instead of Josh, and if I did, I apologize.

Um, so let, let's focus in on how Josh you are partnering with hydraulics. It, it seems on the AWS platform and getting, getting you more than your money's worth, right? Better application performance at a better price.

Sounds too good to be true. You know, you, you sort of hit it on the head when you described the problem, Alan. What, what, what we've been dealing with at a lower security is massive amounts of data that flow through our systems that we need to collect and store for AI training purposes.

Well, yeah, we process more than 50 petabyte of data every day through our system. So we're talking a lot of, Just think about that though. 50 petabytes.

That's nuts. It's impossible to really wrap your head around. Yeah.

It's like light hears We're, we're dealing with. And, and just to collect logs on that, just to know what we've actually touched, what we haven't touched, the important parts of the data, it becomes an enormous, enormous data storage and retrieval problem. We've been using the traditional log tools to do our work for, for years now, and we've run into limitation after limitation.

We can't spend unlimited money to store the logs. We just run into, Hey, we've got economic issues here that we gotta run a business that's led us to have to make tough calls about which logs flow into the systems that we give visibility and where we don't. And then when you don't have visibility because, oh, I couldn't afford to put my logs here, the frustration's enormous.

We've also run into trouble over and over again where we can't look back far enough, can only keep a 30 days-ish of logs in, uh, in the system because that's all the real data volume that we could afford given the massive amount of data that we're processing. So, hey, oh, that we, we learned about it 32 days after the, the thing we needed to go back and look for. Data's gone, frustration after frustration there.

And then I think for us, the real, the the, the real fine point that I'll put on it is we had occasions where we rolled out new capabilities into our software, and we are, we are a go fast and break stuff kind of, kind of company. We move really, really quickly rolling out new capabilities, especially in our non-production environments where, you know, we're, it's okay to move and break stuff, but we're still of course, logging and collecting data in those environments. There have been times where because of those new functionality rolling out, we blew through our logging limits with our provider, and that provider just didn't give us enough time to deal with it and ended up shutting us down and having to go back for, Well, because your impact, look, as someone who was doing web hosting in the late nineties, when when you blow past your limits like that, it's not just a question of do you have the money to pay for what you're going to use, but you're, you're affecting my other customer's performance.

And should everyone suffer because you're a log cog, you know, for lack of a better word. And, and so I, I get where they're coming from. I'm Not sure if I'm on that page with you when you're talking about a big time company who's running a cloud-based logging service, who's happy to sell you as much cloud-based indexing as you're willing to buy, but you uhhuh, hey, the real pain is, Hey, we can't, we, we, we need to stop indexing this data.

We can't. We need a new license. We can't get in here anymore.

And for us, that meant outages of certain capabilities and like those outages were just unbearable. So we ended up meeting hydraulics. Uh, we were connected actually through Akamai, who, who told me, Hey, if you're frustrated with what you're doing today with your logs and the way you're doing it should check out hydraulics.

They're, they're saving folks money and, and giving them great performance. So, uh, so that, that's just what we did. We took a look at hydraulics and we found that it, it, it met our use case perfectly.

Didn't really change our workflows in any way, but allowed us to expand our data storage to a much longer term. Allowed us to, to throw away all of the, Hey, can we keep this log? Can we not keep that kind of data and bring it all into one, one store that we can query all at once And, uh, replace all of the workflows and, and functionality that we had built up in, in our or legacy log log management and alerting system.

And we did that at a ridiculous pace with, uh, with hydraulics. In fact, today was cut over a day this morning, we woke up and our legacy log system was shut down. Today, we're sitting here, it's two o'clock in the afternoon.

Every one of our systems has been updated to use the hydraulics endpoints. Hydraulics helped us through the, the, the transition. And, you know, I'm able to sit here, calm and do the interview together with you, rather than be in the middle of, oh my God, hair on fire incident, we shut off.

Oh, yeah, That, that, that says something. Sure does. Unless you're really a good poker player.

I, you know, maybe, uh, Abby, so is there black magic here? How are you doing this? Well, I mean, that's music to my ears, um, what Josh just said.

Uh, yeah, I mean, we have a lot of differentiators in the market. Uh, you know, our decoupled architecture makes it really easy to scale up in real time. Um, ingest endless amounts of data.

I mean, we ran the Super Bowl. We were the sole multi CDN observability provider for Fox Corporation during the Super Bowl. We were ingesting at peak time, 55 billion.

No, we, we ingested 55 billion records throughout the whole event. And at peak time it was more than a petabyte a day. So our platform is used to ingesting massive amounts of data, um, no matter the amount of that data without log limits.

Uh, and we do it all in real time. 'cause data, data also remains always hot in hydraulics. So querying data takes subsecond, um, and then just even time to glass.

So the time that you ingest all that data to the time where you see the analytics on a dashboard, less than 10 seconds. Wow. Um, yeah.

So, you know, we, we've built something that solves the problem of ingesting massive amounts of data and correlating it from all these different sources into one dashboard, but also the retention component of it. We have a policy that's at least 15 months, and it's 75% less the cost of other observability providers on the market. So Companies, so let, let me just wrap my head around this.

You are ingesting and retaining that data for at least 15 months. Yep. That's our, that's our standard policy.

So companies can retain it for years at 75% less the cost. So, and I think I've mentioned this before, but we're also known as headless observability. So it's, you know, if, if, if folks wanna keep their front end dashboards that they love, they can, uh, and use us for the ingestion, analytics, retention and save money and keep data always hot, or you could do a full rip and replace whatever works best for people.

But we are very flexible in that sense. I'll be honest with you. You blew my mind here.

So how, how, how are you doing this? It obviously, if you could tell, I mean Yeah, well, I mean, I, I can't tell everything that's, uh, that's under the hood, but the, uh, the architecture has a lot to do with it and how, and how we built it, um, with that decoupled architecture and in indexing too, we do right at the point of ingest. Um, so is it A lossless kind of algorithm though?

Or What do you mean by lost list? I mean is are you really retaining a hundred percent of that data Yes. Or are you somehow indexing, summarizing, using some AI something We Have, we have summary tables yes.

To summarize the data, but we are ingesting all of it and retaining all of it and using all of it. I love it. What a great story.

So Josh, you switched over totally today as you sit here calmly with not a beat of sweat on your forehead. Um, how long did this process actually take though, for you guys? So we've been, uh, we, we, we performed a free thorough evaluation with hydraulics where we split our data stream and, and, and had that data flow to both our legacy provider and to hydraulics.

And one of the really cool things hydraulics was able to do was use our legacy observability providers UI as the UI for their data. So we could compare side by side in the same software using the same graphs, charts, dashboards that we've been using for a long time to run the business. How are we doing?

Are we actually getting all the data into hydraulics? Is the data there, can we query it the way we want to query it? Does it populate the way we need it to populate?

That allowed us to build tremendous confidence. We knew everything was gonna work, and at first our plan was to leave the, uh, the legacy provider's UI in place and just use that as the front end of hydraulics. But as we sort of moved through the path and we got to migration time and hydraulics said, Hey, here's the, here's the API documents you need, here's the info you need to query us directly.

My team took a look at that and said, it's just as easy for us to go direct to the hydraulic endpoints, start getting the data directly from them, cut out the, the piece in the middle and, and just move forward. You know, some, sometimes you end up with, uh, an unexpected, uh, uh, boost to your, uh, to your systems. Like, uh, you take a pat, you take, you take that.

Ah, it's not that important to work on this now. Oh, I guess we're gonna work on this now, and boy, am I glad we worked on this now. Hey, we just just went through one of those, uh, the experiences with, with, with hydraulics.

And honestly, that the toughest thing that that happened was us not understanding our own schema and having to ask questions back to hydraulics about, Hey, how come we don't see the data where we expect to see it? Oh, wait, uh, we need to do a better job reading the manual. Yeah.

Or we come back and Kill myself. Yeah, that's crazy. So, and if you're not allowed to say this, just tell me.

But your currently, your, your hydraulics instance is running on AWS um, Abby, do you run on the other major clouds as well? We can run on any cloud. Um, so we are agnostic in that sense, but our major partners are AWS and Akamai.

But yes, we can run on any cloud. I got it. So I I, I would assume maybe you're running on top of Akamai, CDN, kind of, We ingest Akamai CDN data, so for AWS and Akamai in those, with those partnerships, we ingest data from those specific sources.

So for AWS we ingest edge services data, so that's CloudFront, waf, elemental, those are the main ones. And we're always adding in, in new integrations. Um, with Akamai, it's their CDN of course, and their security services.

So we can, we can, we're constantly adding in more integrations on that front too. So it just depends on the cloud we wanna run in. But we're also a multi CDN, so, you know, like for the Super Bowl, we were ingesting, I think from five CDNs all at once, all that data.

Hmm. So we can do that too. It's crazy good stuff.

Hydraulics, uh, US based, Oh, yes. We, uh, our, our main headquarters are in Portland, Oregon, but we are global. We are everywhere around the world.

Well, you'd have to be with, with that kind of footprint. Um, I didn't even ask websites guys. Abby, what's the hydraulics website?

io. And that's H-Y-D-R-O-L-I-X, just like it is under your name in the bottom third there. Um, Josh, how about allure?

com. All right. Hey guys, I appreciate you both coming on here and, and, and actually opening the kimono a little bit and giving us a look behind the black curtain and what's going on.

What a tremendous story. Now, Josh, we may have to have you back on here and see if you're still a happy customer in three months, you know, or, or if we see you sweat. Uh, but it sounds like a beautiful marriage, so congratulations and good luck to both of you.

Um, hydraulics all lost security here on Tech Drunk tv. We'll take a break. We'll be back with more, As enterprises roll out production applications using AI Model Inferencing, they're finding that they are limited by the amount of memory that can be addressed by A GPU and a lot of the other architectural considerations.

This episode of utilizing tech features Steen Graham, founder of Metro ai, discussing Modern Rag and Agentic AI applications with Ace Stryker and myself. Welcome to Utilizing Tech, the podcast about emerging technology from Tech Field Day, part of the Futurum Group. This season is presented by soy and focuses on AI at the edge and other advanced enterprise IT topics.

I'm your host, Stephen Foskett, organizer of the Tech Field Day event series. And joining me today as my co-host from Soy is Ace Stryker, somebody you may recognize from our last season of utilizing Tech. Welcome to the show, ACE.

Thank you, Steven. I'm very excited to be back and, uh, we're here in beautiful Sunnyvale, California today. Thanks for having me.

Yeah, It's, uh, pretty cool that we were able to get together in person to record this episode. We're actually here for our AI infrastructure Field Day event. And, uh, Heim is gonna be presenting, uh, this afternoon.

And so we thought, thought that it would be fun to record an episode of utilizing Tech right here on the show. Uh, talk to us a little bit about, uh, what we're gonna be thinking about today. Sure thing.

Yeah. Well, the, the topic de jour is AI, as it has been for, uh, our last, uh, several conversations. Uh, what we're getting into today is really around, uh, the inference side of the AI equation.

So we've been spending a lot of, uh, calories there lately, uh, talking with partners, customers, understanding emerging use cases. Uh, as we've said, uh, there is no AI without data. There is no data without infrastructure, right?

And that's where soy comes in. Um, what we're learning is that the, the sheer magnitude of data, uh, on the inference side of things, uh, is just blowing up. There's, um, uh, uh, that's not just our point of view.

I mean, there's, there's analyst reports from McKinsey and Tech Insights and others that'll kind of reinforce that notion. But what we're seeing is people use these models so much and they, they, there's so much, uh, data involved in going in and outta these models during inference, uh, that it's really putting a strain on infrastructure and it's driving requirements higher and higher at a very fast rate. And so I'm, I'm looking forward to getting into that topic a little bit today with our guest.

Yeah. And, and, and importantly, uh, memory constraints pay a huge or, uh, play a huge part. So one of the things you're gonna be talking about today on both the podcast and as well as at the field day event, is how that can be, uh, reduced, uh, through some clever engineering.

And speaking of clever engineering, that's why we've got Steen Graham here. Uh, Steen, welcome to the show. It's nice to have you.

Well, thanks for having me. And Steen Graham, CEO of Metro ai, and we do a lot of work around building AI agents and, and also model and hardware performance evaluation as well. So talk to us a little bit about, uh, well, I guess let's, let's just go right into it.

Um, many companies are trying to deploy AI based applications. Many companies are trying to build applications that incorporate, uh, enterprise data, uh, retrieval, augmented generation or RAG has been a huge topic, but what we're finding is that that can place some pretty extreme, uh, stresses on infrastructure and require quite a, a lot of memory in order to implement in the real world, right? Absolutely.

And I think, you know, the, the modern kind of transition from your historical CPU first data center to A GPU First Data Center has caught a lot of enterprises in this chasm of that making that transition. Meanwhile, the pace of change in AI is presenting all these opportunities for them to move to these modern software stacks, but their existing infrastructure doesn't quite work for it yet today. And, you know, I think what we've been working on with the team at soddy is how do we kind of look at those scenarios and pave a pathway instead of going, you know, full GPU centric data center with all your infrastructure, how do we kind of give them a pathway and an affordable TCO optimized pathway into deploying those, those latest AI agents and agent rag stacks for their their use cases?

Hey, Steve, can we, just to start, uh, no doubt some of our audience, uh, is familiar with retrieval, augmented generation, what that is, but can you give us a quick summary, you know, how is that different from just selecting a foundation model off the shelf and, and plugging it in and starting to feed inputs into it? What, what does RAG buy you and why are folks so interested in it? Yeah, so I think the, the, a simplified view of things is, you know, several years ago with the advent of transformer based large language models, we, you know, you would just use a model, serve it in a chat bot, and then the model would hallucinate.

And so obviously for, you know, enterprise quality needs, that doesn't work. And so the, the next kind of iteration of, of innovation was how do we actually put your data close to that large language model? Um, and notably what sometimes we use a, a vector database or, or a graph database in some scenarios as well.

And graph graph could be super useful for, for different different use cases. Um, but you just basically take all your existing data and usually it would be like, you know, for a particular use case and the, the associated tribal knowledge associated with how that use case is solved, and you're pairing that, that vector database with the language model. So now the language model before it generates an answer is querying, um, you know, high fidelity, accurate domain specific information.

And that's really kind of the simplified view, you know, of, of the rag, you know, environment today. Now, what you'll hear in the year 2025 is everybody will say, this is the year of AI agents. Um, and, you know, I have, I've definitely been hearing that.

Yes. Yeah, yeah. And I think AI agents are kind of, you know, an extension of that evolution where we're actually now allowing the AI to query like, and use tools from the company API calls, you know, to the company's internal CRM system or their HR system, or your supply chain system, or your Jira tickets.

And so now we can, we can do things like take that domain specific information that RAG already has, add an AGENTIC framework on top of it, and then extensively do that where you can actually create a digital worker that can get the job done while a human's not in the loop. And that's where, you know, people are really trying to look for, where's my 10 x, uh, ROI with ai? Turns out it doesn't happen when a person's in, in a chat bot, co chatty, and even a rag based environment, traditional rag environments, have had that kind of chat bot type interface with your own data.

So you're chatting with your own company's data for higher fidelity outcomes, no hallucinations, but you're still not getting the scalability of a digital worker that the AI agents will provide. Yeah. And, and that's, I mean, I guess if you wanna talk metaphorically, I mean RAG is a great idea, but essentially, I mean, it's, it's metaphorically, it's a librarian with a really great card catalog who can look up things and make sure that things are contained within the dataset and that they're the right things and so on.

You can validate data. Uh, there's a lot of things to love about it, but the problem is it needs to have that really big card catalog or vector database, and it needs to have this huge data set, and that can take up a lot of space. And that's been, uh, something that's, I think, held this back, even though it sounds great.

How do you have that encompass, you know, your, your company's entire corpus of data? How do you Yeah, Well, I think, I mean, most companies, you know, sit on, you know, terabytes or, or petabytes of data. So like step one is, is just basically organizing that data and the high fidelity data.

And there's the, like the last 10 years we've went on a data journey. So for the companies that have transitioned to leadership data lakes, you know, they're in a good position to be able to, you know, start vectorizing that data. And most of the database companies and data lake platforms are now offering the opportunity to kind of vectorize data as well.

So there's a lot of opportunity and pre-work that's already been done to put the AI models in a position to succeed, but we're still kind of probably in the position where you do want to nail a particular use case. So, you know, curating that data set for the pro the domain specific problem you're trying to solve, um, you know, who's the business unit owner that wants to solve that particular problem is still incredibly important. So you don't want to just dump everything, um, you know, into a vector DB and start querying right away.

You're probably not gonna get the highest fidelity results. And a lot of corporate data is, is greatly outdated, um, as well. So there's, there's obviously some curation you wanna do to set your up self up for success, but there's a lot of pre-work already already being done today to, to make that possible.

But that is the most important part of the journey, having your data organized and ready to go. There's no question there's a lot of data involved in the, in the stuff we're talking about. Um, I'm curious, when you look at the actual, uh, architecture that, that these models are running on, um, and that this rag data is sitting on, can you give us a sense of, um, you know, is this stuff typically done, uh, in, in memory?

Is there a lot of, uh, storage involvement in, in real time as, for example, in enterprises running, uh, you know, an inter uh, an inference workload and consulting, some kind of rag connected external data source? Yeah, I think, I mean maybe like just looking at kind of the, the models themselves like, and where, where people are battling right now with GPUs, because we're centering the data center and all our applications around GPUs 'cause they're the bottleneck, you know, so any rational kind of throughput analysis always says the most expensive component is where you want to have the bottleneck. And so there's a lot of pressure on the GPUs right now.

The trade-offs that the, you know, the companies making GPUs have is it's really, really costly to put a bunch of memory in the GPUs simultaneously. The larger the model roughly, the better the performance, the more state-of-the-art model, um, that occurs. So there's this really big challenge, you know, and in the GPU memory around fitting big, big large models in the GPU memory.

Um, and so that's kind of the number one bottleneck that, that we all, we all face, um, in the market today. Now when you start pairing that with, you know, a rag based architecture, which usually we're running the Vector DB on the CPU and then, you know, we're, we're bringing in all that, that vectorized data, the memory hierarchy kind of levels out a little bit more, you know, more like a traditional, uh, memory hierarchy that you would an anticipate as well. Um, but yeah, the GPU constraints are happening and then you're pushing the workload now into more of a full application systematic software workload that's, that's driving more of the traditional, you know, storage memory as well.

When you shift to a rag based architecture. And AI agents is more, extens is similarly extensible to that where, you know, you're, you're running a lot of application logic for that domain specific use case. API calls that are all happening in more like traditional compute infrastructure that doesn't need to happen on the G-P-U-G-P-U is just focusing on serving that model performantly.

And when you say GPU, I'll just point out too that, uh, when it comes to edge especially, but even, um, increasingly in data center AI and cloud ai, uh, it's other types of accelerators too. I mean, you know, there are, there are definitely acceleration engines out there that, uh, in, in many cases can provide, uh, better service than just a standard GPU, but they have the same constraints that you're talking about. In fact, in many cases, those accelerators have even greater memory constraints.

Yeah, you, you absolutely nailed it. If you look at the companies, um, that are doing a lot of innovation in AI accelerators mm-hmm. And I use JPO as kind of to, to cover the AI accelerator world almost interchangeably.

Um, but the, those companies in many cases made probably decisions to, to save on bomb costs. Yeah. And not have memory, um, you know, in their systems that have, they've driven them to make decisions on how they serve models, and then they have to paralyze model to serving these large models mm-hmm.

As well. And some of great wrote great systematic software to do just that. But this is like a big challenge in the world today, especially as you, you apply larger models, but also you apply a chain of thought reasoning.

We start to like, massively increase the amount of inference calls we're doing, um, by giving the model the ability to kind of think through things more. Um, you know, now you're, now you're really driving a, a significant workload and ultimately a high memory footprint too. Not to mention people are announcing like nearly unlimited context windows, like 1 million token context windows at all.

And, and wanting to kind of like, make sure we sustain that over time. Which, you know, the context window versus rag scenario is, is an, is another, another trade off to ACEs because with these massive context windows, you're almost getting rag in that mm-hmm. MLM application as well.

Um, but the kind of the fidelity of an enterprise application, I think still, still likes the separation of a rag environment. A lot of those context windows are mm-hmm. Teed up more for consumer based applications at this point in time.

Right. But definitely muddies the waters quite a bit. Yeah.

One of the things we've been hearing about a little more often, and call it the last nine months, is, um, approaches for, for grappling with the increasing amount of data involved in inference. We've seen, uh, storage vendors come to market and talk about approaches, for example, for offloading some of your rag data and, and accessing that directly from storage. Uh, NVIDIA is just a GTC talking about the key value cash and, and approaches for, um, you know, placing that in storage as opposed to in memory, especially as you, uh, involve more complex models or longer interactions between models and that just grows and grows and grows.

Is that a, uh, is that a feasible approach? Is that, is that something that folks should be thinking about as a, as a realistic solution to the problem of memory constraints as more and more data gets pulled into the pipeline? Yeah, absolutely.

I think, um, you know, kind of one of the, one of the gifts that we have that I think is under underutilized today in the AI world, probably 'cause we're all focused on like this GP memory constraint is, um, you know, disc a NN. And what that allows us to do with the, the disc a NN base optimizations we're allowed to offload workloads onto solid state drives that traditionally would be run in memory. And, um, you know, while the indexing time it takes a bit longer to index it, the net results is your queries per second and performance improved dramatically.

So that's, I think, a little hidden, you know, hack that you can use to lower the, the memory footprint. Um, That's a little, I, I wanna, I wanna dig into that one. 'cause that seems a little counterintuitive, right?

When you tell someone they can read some data from, from storage as opposed to from memory and you're actually seeing higher queries per second when you take that approach. Is that right? Yeah.

And that's, Yeah. That sounds wild. Yeah.

Yeah. Well, I mean that's, I mean the, I think the work that's been done, um, within the dis a n working group, and obviously we're spending a little time on the indexing side doing some pre-processing and some optimizations there. But once you've kind of made that, that trade off, when you index that kind of one time ish trade off when you're indexing, then you know, you've got a little bit of the algorithm based optimization that will give you that, uh, queries per second performance.

And it's not like a two x type differentiator, but it's like same level, uh, you know, level performance, you know, plus or minus. Um, it can even in some data sets be dramatically more. But, um, and then you're looking at same, same level recall accuracy.

So just at a trade off of indexing time. Wow. And so it's indexing time, not even capacity.

'cause I was thinking that it would be a trade off with capacity as well. 'cause the nice thing about storage is that you can have a lot more capacity than you can have with memory. Yeah, that's, that's a fair point.

You're definitely using more capacity with that implementation as well. Uh, but you know, that's capacity that, you know, as long as you're using like the, the state of the art, you know, like PCI based drives, like that's capacity usually have, you know, in the system, uh, in initially as well that you're using for other applications. Yeah.

And that, and that's where I want to go to too. So, um, the side effect of making things, and again, even if it wasn't faster, even if it was just not slower, that's still groundbreaking. Yeah.

Right. And the side effect is that you have much more capacity, and so you can deploy applications with much, much more data to support them than you could in memory, even if it was not the same level of performance. Right.

I mean, even if memory blew it away, you'd still run outta memory pretty quickly. And I mean, so, you know, we've talked about soy, uh, you know, you guys have, you know, very big drives, you know? Mm-hmm.

I mean, I remember the announcement of the 60 terabyte drives and of 120 terabyte drives. Uh, I don't think we're talking about having 120 terabytes available to a rag application right now, but are we? Well, I think it's definitely, you know, in the scope, it really, I think it really depends on how much high value data an enterprise has.

And if they've got, you know, a hundred terabytes of high value data that's for a domain specific application that improves the quality of the output, you know, it's definitely, you know, in the scope of deployability today. That's wild. Um, and of course it doesn't just have to be OneDrive, I'm a storage nerd.

I mean, absolutely. I mean, most, most storage systems use multiple drives, but just the fact that we have that kind of capacity that could be made available to these applications is really, really shattering because there's just no situation in which you could have that kind of ram at an affordable price point if you really wanted to deploy an enterprise application with many terabytes of, of data, you just couldn't affordably. Yeah.

Yeah. So, speaking of affordability, one other cool thing we've been, um, having fun with recently is because that kind of core problem that we, that we've always seen about GPU memory footprint, we thought it would be really interesting to see if we can offload the actual large language model onto the SSD. So this is actually very unique and, um, you know, what, what We've done done, and I've heard about people investigating that that's a really cool idea.

Yeah. And there's some, there's some tools and technologies. Um, in this case, we're using a, a feature in deep speed, which, you know, in many cases we use for training applications, but deep, deep speed has some capabilities around model offloading.

And so, uh, what we've done recently is we've taken a 70 billion parameter model. Mm-hmm. Um, which doesn't fit in like a L 40 S based Nvidia, GPU.

Um, and we've actually offloaded the model into solid state drives. And while you don't get the same performance, you know, you couldn't deploy that model at all. You know, and so it gives you kind of model capability based on offloading.

So for people that haven't refreshed all their infrastructure mm-hmm. Or they're waiting to get the latest and greatest GPUs, you can actually use this technique to use a bigger model, um, on a lower cost GPU by SSD offloading. So that's kind of a, a cool innovation.

And I think just like disc a and n has evolved over time and performance has improved over time, I think we'll see a level of innovation and performance improvement in SSD offloading as well, that it's gonna be, it's gonna warrant paying attention to, especially as we're, we're increasing the number of, you know, chain of thought reasoning and applications. And all these inference calls are exploding right now. So at some point you have to look at affordability.

Um, and that's, that's a great way to hit a different, totally different level entry point on pricing. Hmm. The, um, the model off way offload thing is, is really compelling to me.

It's a really interesting, uh, idea. And I wonder, like, my, my, my gut sense is that that may be interesting to folks particularly, uh, who have interest or, or needs to deploy, uh, AI solutions at the edge, because in a lot of cases we have, uh, more severe power constraints, space constraints. You may not be able to put the latest and greatest GPUs in your Edge servers.

Right. Um, do you see that as a, as a potential play for this where, hey, you can now run a 70 billion parameter model on A GPU running at potentially much less power than the GP would've otherwise needed, and now we can, now we can take that AI to new edge environments? Yeah, it, it definitely meets that, that criteria that you look, when you look at the edge, you think about, okay, we're, we're constrained, you know, from power footprint.

Um, usually there's a big latency requirement at the edge, um, but the existing infrastructure at the edge that that's lit legacy, typically Edge has a little bit more legacy infrastructure, so it kind of checks all those boxes as far as trade offs you'd wanna make at the edge. Now, it might not be a 70 billion parameter. That might be a technique you use on a 7 billion parameter, or if you're deploying on some really legacy infrastructure at the edge, it might be a 700 million parameter model.

So I think that it scales down to kind of the right footprint, uh, for the edge as well. I wouldn't, um, ignore kind of the enterprise, uh, cloud applications here as well, because what's happening with, um, with the transition from chatbots to rag to AI agents over time is AI agents are running autonomous of human intervention. Um, now you can always, you know, human in the loop it, but what we want our AI agents to do is they want to, we want them to be our digital workers that are working for us while we're asleep, we're hanging out with their family and enjoying life.

And then we wanna come back in the morning the next day and see the output, all the reports and documents that, you know, the, a agent conducted for us while we were enjoying some great sleep and some great family time. And that can be done on a batch based processing node. So we don't need to like, you know, get the most high performance, uh, GPU in that scenario.

Um, we can kind of use what MacGyver, whatever we have available today, and, and leverage that and then, and deploy it as well for batch based workloads. So I think AI agents offer us a great opportunity to do some trade-offs in, in latency. So like real time tokens per second, little less important for AI agents, um, depending on the particular workload.

Yeah, we've been hearing that as well. And, uh, with RUM and some of the research that we're doing, in fact, we're starting to see people talk about using, um, CPUs for, especially for AG agentic ai, for the same reason, because it's sort of an asynchronous workload. Um, also because there's a proliferation of CPU cores.

The CPU cores have a lot of specialized functions. In many cases, they're actually getting specialized AI instructions, and because they have greater addressable, um, memory in many cases than GPUs or accelerators do. So CPUs can look increasingly attractive for this.

And especially, and, and, and with many of the things that you're talking about, I could see a lot of that going hand in hand with this CPU trend as well, uh, wanting to use more storage instead of memory to reduce the overall bill of materials to deploy some of these agentic applications. Because essentially, um, you know, you kind of take this to its logical conclusion. We could see sys systems running, um, agentic applications on conventional servers with, you know, a reasonable amount of memory and a reasonable CPU and a reasonable amount of storage, thanks to the fact that we now have capability to use that.

Is this a vision that you would share? I think yeah, absolutely. I think there's a lot of opportunity to take that historical data center architecture and run AI agents on it, whether it's CPU, and we actually have a, a number of AI agents that run a hundred percent on CPU, new GP required.

Now that being said, I think some of the, you know, older GPUs, fabulous performance still, you know, for those type of workloads as well. So I wouldn't, I wouldn't try start transitioning a hundred percent to CPU in all case scenarios. But for those batch based workloads where you're, where you're fine with a little bit more latency, um, it definitely works.

So those existing data centers, I don't think need to be totally retrofitted today in all scenarios, you know, for a GPU centric architecture, um, we can make use of them for, for deploying AI and AI agents. Uh, one, one more question from Mete. Um, I'm curious, since you're our expert on agents here, and we haven't had one before on the podcast, I want to pick your brain on this.

Um, let's say you've got, you've got a model, you've connected it to some rag data, right? And, and what it can do without identifying is that the right term without giving IT agency, is it can, it can give you insights and advice, right? And then, and then when you give IT agency, you're now connecting it to other tools and systems and allowing it to, to take actions on your behalf.

Does the, does the act of giving that model agency have significant, um, uh, repercussions in terms of the amount of data generated or used? Like you, there's a lot of data clearly involved in training a model. Mm-hmm.

There's a lot of data in RAG potentially, and then whatever systems you're connecting the model to have their own dataset, which presumably existed before the connection. But are, is there a, is there a big impact to incremental data simply by virtue of making a model agentic? Yeah.

Is that something you guys have looked at? Yeah, I mean, it absolutely. I mean, we see this in, you know, our AI agents that have, you know, chain of thought reasoning.

Um, and the more autonomy you give these agents, I mean, the, the human is the bottleneck in the scenario. So if you, the, the better you design the workflow, the more API calls that that agent can do, the more tools they can do, the more, you know, autonomy you can give it and problems it can solve, it just massively explodes the level of data, uh, being created. I don't wanna like characterize that data as synthetic data, but I would say, you know, kind of non-human generated data footprint mm-hmm.

You know, is, is massive. And I think, you know, obviously I think we're, we're kind of at, at the era where the non-human data generated footprint is, is gonna be, you know, much greater than the human generated data footprint, uh, based on giving the AI autonomy as well. And also kind of the advent of modern robotic tools are very data intensive.

So I think, I think that that transition point's gonna be a very interesting transition point. It's great for the storage business, so as you should be happy, um, and it's also valuable synthetic data, right? And, and this is kind of what we're struggling with with AI right now, is, uh, you know, there's, you know, the open kind of web data we've certainly ran out of as far as, you know, training models.

So then there's a path for synthetic data. There's more human labeling, more reinforcement learning, and now we've got this whole category of, you know, AI agents doing chain of thought reasoning with their data footprint. And so, you know, all, all of that kind of non-human generated data is gonna be really important for the fidelity in the future of AI models.

Because if that's high quality data, if if we're generating high quality workflows, then that'll be super helpful. Obviously, if the workflows don't work or the have models failing, then maybe that will, you know, devalue that mm-hmm. That type of data relative to human generated data at this time.

Right. So it's kind of a really exciting to watch that play out. Cool.

Yeah. Yeah, it really is pretty interesting. What's, what's happening here.

And I, and I'm actually excited because, you know, generally people, uh, there's sort of a, a thought that, uh, AI applications require absolutely cutting edge, high-end hardware that's really expensive, that consumes tons and tons of power that just, you know, basically there's a lot of negatives around ai and, and many of these negatives are true, but the industry is absolutely working to address those challenges and those criticisms. And in many cases, we're going to see applications being deployed on much more, um, restricted or, uh, modest hardware with, at, at a much lower price point. You know, and, and I think that all of this means that this, this technology can have a bigger impact than we might've assumed, simply because it doesn't necessarily require the, the, the biggest, baddest, hottest, uh, hardware to run on.

You can run it, um, more approachable on, on more modest hardware. So all of these things, I think, go in that direction, and I think that that's a, that's a positive for all of us. So thank you so much for this conversation.

I guess, um, what last thing would you wanna leave our audience with? What's your summary of, of this message? I think, uh, for me, I think there's, there's a lot of ways to deploy ai.

And I think the, the innovations we've seen in the last year are, you know, on affordable deployment of AI are probably 10 x, um, what we saw in the last 10 years. I mean, it's just an incredible pace of change on driving affordable models. And I think, um, what's most important is, you know, designing the right business workflow, you know, for these autonomous workers or AI agents.

And then, you know, figuring out the deployment methodology. There's so much innovation happening on, on affordable off the shelf hardware, or even affordably deploying a state-of-the-art hardware, um, that I wouldn't let that get in the way of, you know, your company's innovation. Excellent.

Well, thank you so much for joining us. It's been great having you. Um, before we go, where can people continue the conversation with you?

ai. Excellent. And, uh, ACE, it's been nice seeing you again.

As I said, you were one of the co-hosts last season. Um, so check out utilizing Tech season seven. Where else can people catch up with you?

Where have you presented recently? Or, or where are you gonna be? Uh, Boy, oh boy, there's, there's, uh, a lot going on at Solid I these days.

Certainly. com/ai, where, uh, we hope to be featuring some of Metro AI's excellent work in the near future. Um, we'll be at, at conferences all summer long, so, so keep an eye out at, at all the big ones.

Uh, for now my head is spinning, uh, with all the implications of what scene is talking about. And so I need to go have a, have a lay down and kind of chew on some of this stuff. But really, really appreciate you being here, Ste.

Uh, I've learned a lot and, and thank you. Yeah. And, uh, me as well.

And, uh, I will point out that, uh, by the time you watch this episode, the solid IME and, uh, Metro AI presentation will be published on YouTube. Just go to YouTube and, and search for solid IME and Tech Field Day. And you'll find that, uh, that was part of our AI infrastructure Field Day event, which happened in April.

Uh, we're also gonna be doing an AI Field Day event, a cloud Field Day event, and we just announced another AI infrastructure event later in the year as well. So check out the Tech Field Day website for more information about that. So thank you very much for listening to this episode of Utilizing Tech.

Uh, you can find this podcast in your favorite podcast application, just search for utilizing tech, or you can find us on YouTube. If you enjoyed the discussion, please do give us a rating or a view or a comment we'd love to hear from you. Uh, this podcast was brought to you by Solid, IM as well as Tech Field Day, which is part of the Futurum Group.

com. And we are present on the socials. You'll find us on x Twitter, blue sky, and Mastodon.

Uh, just search for utilizing tech. Thanks for listening, and we will see you next week. Hey everyone, it's Alan Shimel for techron, and welcome to the first episode in a series we're doing that we call Shift Left Shift, right Shift Everywhere.

I am really happy to be here. I'm really happy to have these two guests I'm gonna introduce you to in a moment. You know, we're doing this series with our good friends at Adobe, and I know everyone out here has heard of Adobe, and many of you use Adobe products, but I don't know how many of you know how influential Adobe has been in the world of security over the years.

When you, when people are trusting you with, with the, their files and their work, like millions around the world do with Adobe, they don't have a choice. But to take security seriously. And as we were talking with my guest offhand, off camera, you know, a lot of security innovation has come out of Adobe.

Um, Adobe of course, is all about you, the technical people out there who are working in all of their products for graphics and documents and applications and everything else. And they have for a long time. This whole series is gonna be focusing on sort of what's Adobe's view of security, about what's some of the frontiers, some of the, you know, areas of security that we, we wanna shine a light on.

And, and specifically, as I said right in the title shift, left Shift, right Shift everywhere. Where do we put our focus on security? Look, I've got two great folks from Adobe to introduce you to who are gonna be talking about this with me.

Let me introduce them to you now. First I want to introduce you to Pelli. Yuli.

I hope I got it right. I've got, I'm doing the best I can on names, but Pella's name is actually not that hard. Pelli is the lead security strategist at Adobe, and we're thrilled to have him on Pelli.

Welcome. Welcome to our podcast series here. Share with our audience a maybe a little bit about your journey.

Um, sure. So I've been in the security industry for 25 years. Um, I started out working for a company called Anonymizer, which was sort of a commercial version of tour way back when.

Mm-hmm. Uh, I worked Insecurity Consulting for a while with had Stake and Symantec, and I've been at Adobe for 17 years now, working in all sorts of areas of security. And, uh, when we brought, uh, Florian into the team, I decided to go and focus, uh, mostly on shift right type projects.

So I'll be representing the shift right aspect of it. So you're the right hand. Yes.

I hope it's still right on your, this is my right hand. I sometimes it mirrors. I know, but that's funny.

You know, at Stake, of course is legendary, right? Chris w Ball and, and the folks there, they went to semantics. So it sounds like you were involved in, in all of that, you know, in the, in, I've also been in the security business 25, 30 years, legendary, legendary folks.

There, it's folks still doing great things. Um, but thank you for joining us. Sure.

Next, let me introduce you to, uh, Florian nut Netting. No noting, I know you gotta curl your tongue. And in New York, we just don't curl it so well.

But Florian, nerding, Florian, pronounce it correctly. Tell a little bit about yourself. Difficult help me or rescue me, Difficult name, my name's, uh, Florian Noeding, or if you want to use a German pronounced Asian, because I'm originally from Germany and it's Floridian, which is even more difficult.

But let me also talk a little bit about, about my background. I started my professional career in, uh, 2000 and and 10 at a small startup, which built, um, network firewall de devices with a focus on being very, very user friendly so that anyone without networking or security ex expertise could actually set some up and have a secure net network for their, their office or their, their home. Even after a couple years working as a software engineer there, I joined at, at Adobe, and I've been with Adobe by now for 11 years.

And, and, um, most of the, well, a bit more than half of of the time I, I spend in software engineering. I am, and it's still what I, I'm at heart. I'm a software engineer.

I want to make the lives of, of developers better and really focus on pragmatic security solutions. Roundabout six years, I go, I joined the security org, started working to, together with, with Palace, and I'm taking care of all things Shift left. And so the cutoff point is basically when software gets deployed to the cloud or otherwise released to our customers.

So in, in my scope is there's a lot of stuff from security training, security, awareness, code analyzers, and various aspects around secure by, by design, and especially memory safety. Excellent. So you're the left hand?

Yes. Got the left and the right. Okay.

I feel like the Pope, um, anyway, He's home from the hospital, so that's good. Anyway, um, let, let us, let us talk a little bit about history. com in 20 November of 2013, published March of 2014.

A big reason that I personally felt compelled to do this was because I thought that DevOps offered us the best hope of, of getting security, right, of, of correcting a lot of wrongs, right? I I, I grew up, or my career in security, probably much like you, Pellis was on the right side, right? AF post-deployment, I helped found a company, intrusion prevention network, access control, vulnerability management, you know, all the traditional network security stuff.

And the problem was we were, we were always the caboose on the engine, right? The end of the train, the engine got pulled by the developer or, or someone else, right? It was too late.

By the time we got involved, it was too late to often to fix a lot of the wrongs that were there. And I always felt if we move further up the food chain further left, if you will, we would be able to fix these things. And what a perfect opportunity DevOps was, right?

Ops and Dev working together, let's get security in there and we're going to move security to the left. And, you know, the, at the time, the notion was, and I don't know if you believe it, I'll ask you both, that it was a fraction of the cost to fix a vulnerability or a defect, far left. Then it was to try to fix it in product production in the right hand, right?

So it was cheaper, it was more efficient. It was, it was just everything was better doing it to the left. And why start just left of deployment?

Let's push it all the way left. Now, like both of you, we, we have friends who are developers, but the average security person said those developers, they don't care about security. They just wanna push out code, right?

They get paid to about how many lines of code they publish. But an interesting thing I learned when I got into this DevOps thing, a lot of the developers, and not only the developers, all the people on that left side really felt that the security people were like an anchor that was dragging them down. They were slowing us, we were slowing them down.

We were the people who say, no, no, no, nope. Go back, go back, go back. No.

And I found it incredibly difficult to bring together what I used to call the, the, the cybersecurity, or we didn't even call it cyber back then, but the security tribe with the DevOps community, it was sort of oil and water. I was trying to make chocolate and peanut butter. Pelli, you've been around if you were at at stake.

You've been around a while. I know. What, give us your take on that.

What do you, you know, was, was it an impossible mission to begin with? Uh, I I don't think it's an impossible vision. I mean, part of, even as a shift, right person, right?

Like my job isn't just to find as many bugs as I can. My, I'm a feedback loop into, into Florian, right? So, you know, we go and we try to look at patterns of, in within the vulnerabilities and say like, okay, are the developers having this consistent class of problems?

If they're having this consistent class of problems, you know, what can, you know, Florian and I coordinate on, and what can, uh, Florian help build to address that class of problems? Like how can we shift the company to using a framework that's maybe a little bit, um, more secure by default, so they don't have to think about security as much. Uh, maybe it's a pipeline problem.

Maybe, you know, it's they're, they're having trouble keeping their amis up to date in, in the cloud. So, uh, it's, I I found that developers tend to want to do security. Well, they, they, some, a lot of times they do find it sort of an interesting topic, but they're, they're just constrained by the realities of, of their situation, right?

They, they have so much time and, um, to get things done. So, uh, from my perspective, you know, I'm not just looking to find as many bugs as I can to get as many points on the board as I can. You know, everything's a feedback loop.

Even if you're doing red teaming, the, the goal of a red team isn't to go N or Nina, or we got in the goal of the red team is to then talk to the blue team and say, look, this is how we got in this, this is where you have gaps. Um, if you wanna catch this, the next time we do this, here's how you can improve. And so there's always a feedback mechanism in, in from shift, right?

To, to make the shift left team, uh, more knowledgeable and enable them to make better plans, to make things just smoother for the developers overall. Absolutely. com, Florian did all this DevSecOps, to tell you the truth.

Give us your, you know, what, what's been your experience at Adobe primarily? 'cause that's where you've been to all these years, but is what I describe, was it true then? Is it true now?

What, what's changed? What's gotten better? So there are multiple perspectives on, on that.

Certainly DevOps, the, the ideas is fantastic. We have a group of people who really focus on, on the engineering aspects of building working software and operations people who then run it in production and take care of all, all the problems that happening in productions there. We have a feedback loop too.

And if we now add security to to, to that mix, both sides need to, to do some of the work. But the challenge with shifting too far left is we, security people should not move all security work to the en engineers operate as of systems, because they are not experts. We as the experts.

So we need to make it as simple as possible for them to find these issues. And there are many different approaches of shifting left. For example, you might shift left and say, well, let's do threat modeling at design time, because obviously it's cheaper to change the design that hasn't been implemented yet.

Then all you have, uh, architectural complete, um, system on, on stage, ready to be deployed to production now and architecture changes very hard. It's, it's too, too late. So shifting left in that sense is very, IM important, giving all kinds of feedback in an IDE on, on the other hand.

Well, now you need to balance different aspects. Do you want to send all the findings to, to the developers only the sets that you care most about? What is this the set, what, what security aspects really matter?

And with my background as a software ENG engineer, I, I wanted to always help other software engineers make pragmatic security decisions and Italy reduce security decisions. So the recent trend in shifting left is secure by design solutions that's, for example, started for cross scripting issues, um, with libraries such as React, where it's really hard to accidentally have, um, injection vulnerabilities because the framework by design prevents it. And that is a very, very powerful concept that I want to see much more.

Yeah, the, the, the secure by design, that whole concept of secure by design does not get enough light, right? I mean, we, we all, for instance, palace, I'm sure on the right side of things, right? Uh, zero trust networks, zero.

The, the idea of zero charge security, right? Everybody kind of wraps their head around that talks about it. It's, it's very, you know, very, uh, everyone, you know, buys into it, so to speak, the secure by design.

I think people shake their head, but they don't necessarily drink the Kool-Aid, if you will, right? In that. 'cause at the end of the day, they're not quite sure what secure by design means, right?

Yes, of course we want to design secure software and we want to try to put in frameworks that take out your buffer overflow SQL injection, you know, the OO watch top 20 or whatever, right? That hasn't changed in 17 years, but, you know, but actually implementing that is hard. It's hard.

And without, again, some ground rules, we, let's not let out state secrets and get us all in trouble. But how does Adobe do secure by design? Yeah.

Let me talk a little bit about memory safety in, in this context because it, uh, showcases the fundamental challenges that we have have to deal with many of Adobe's products, like any company that that is more than 10 years old, probably has lots of c and c plus plus code. You know, operating systems are written and c and mostly c maybe some in CC plus plus desktop apps. See, foundational libraries are all CNC plus plus desktop apps themselves.

CNC plus plus look at any network d device at code running on other than the apps on on your mobile, mobile phone, whether iOS or Android doesn't matter. The foundations is all c and c plus plus it's all memory unsafe. And unfortunately we have learned that humans are not capable of reliably writing memory safe code just too hard.

So we need a, a system so solution, and that is memory safe programming languages where a smaller group of of people is just focused on, on designing a system where it is very, very hard to have accidents like, like that. If you use Java, Python, well, these are not systems programming language. You don't deal with memory safety issues.

If you need to write highly performant code, well then you have rust or may maybe swift. Uh, the two most common choice there are certainly more than these two programming languages. But if you now look at, um, the ecosystem where you have memory safety issues, it's c and c plus plus.

You can't just re widen an entire application in a memory safe programming language. There's no business case to ever make that happen. Even if we had a way to automatically transform, uh, tens of millions lines of code base into to rust wouldn't be interesting because the teams that maintains the c plus plus code base could maintain the rust code code base.

They wouldn't understand the structure if we used AI to transform it, if that would be possible. So we need a much, much smarter approach to memory safety. And the first step is, again, feedback loops.

We need to identify which parts of, of, um, the system are most vulnerable to this kind of vulnerability and does this vulnerability matter at, at all? And that is where the shift dry testing comes in. And I'll hand it over in a moment to palace to speak about fuzzing and what we do there.

And once we've identified these spot that are safety critical, we'll recognize a recurring pattern that, especially areas that do, um, pausing and decoding of file formats are risky. And it doesn't matter if it's an image file format, an audio and, and, and video or a complex document or even an archive, it doesn't really matter. That is the key functionality that we need to protect because an adversary is that sends you a file via email phishing via phishing, which is very targeted phishing.

And with one click, you open the attachment and then open it with an application, and then the adversary achieves remote code execution. That is really the thing we want to, so figuring out which code is executed during this one click attack that is most, most important, and it's file pausing, decoding, and maybe a little bit of running logic, then you can take different mitigations strategies instead of rewriting everything in a memory safe programming language, maybe rewrite one safety critical component in a memory safe programming language. Palace.

Can you talk a bit about f Yeah, sure. So, so this is one of the areas where like you, the goal isn't necessarily always just to find as many bucks as you can. It's to do things strategically.

And this is where shift left and shift, right? Collaborate. So yeah, when we're trying to decide what to fuzz, we could do like just generic fuzzing and try to go after the entire application all at once.

Um, but to do a more strategic approach, you would look at your adversary intelligence, right? Like in, in the wild, what file format types are attackers currently using to go and exploit things? You can look at bug bounties and you know, the people that you have in your, your bug bounty community who are contributing crashing bugs and looking at the techniques that they're using because they're often also emulating what they are seeing, uh, in the adversary intel community.

And then you can go work with the product teams and go, okay, who are the teams that actually are responsible for this code? We can go and you send a specific team into there, we can work to set up fuzzing around that specific section of code and it can actually make the developer experience a little, uh, more predictable. 'cause you're, you're directly working with the team, you're working with one team at a time or two, maybe two or three teams at a time, uh, to do this type of work.

They understand what, they understand the bugs, they're not context switching. Um, like if you're just fuzzing the overall application, you're hitting different teams all the time and they're concept switching versus, you know, working with a team directly where they're like, okay, we're gonna focus on this problem for, for this quarter and we'll, we'll work with you. We'll set up the fers we'll, we'll give you insights.

And then, uh, they can start to see the patterns in the bugs. And if they see the patterns in the bugs, they can say like, okay, well you, you can quite rank the fuzz. We, we know this paradigm that exists in the code, so we're just gonna go tackle that overall and then we'll come back to the fuzz once we've, we've addressed that.

So, uh, you know, with with Shift Wright, you know, I'm always looking for ways not only just to, to find the bugs, but also ways, uh, to do it effectively and ways to empower the teams to move faster. You know, I remember the first time I was exposed to fuzzing, so I think it was black hat around 2006, maybe, something like that. And, and what a, what a fantastic development that was for, at the time, I don't even know if we called it AppSec, Pelli, I don't know if you remember, but did we call it AppSec then?

No, not really. It was still, I guess, vulnerability management. I don't know.

But I mean, what a, you know, the whole idea of fuzzing the code and looking for, you know, the, the zero days before the bad guys found them, if you will, was, was just, you know, what a concept I like, duh, why didn't I think of that? Right? And I wouldn't be working here today.

But, um, it, it, it, it really did help us a lot and it helped the developers fix code, right? Not in real time, but much earlier in, in, in the, uh, in the process. But, you know, I I also, I feel almost like duty bound to say we have made a lot of pro progress on memory overflows and, and, you know, memory vulnerabilities in, in our code at Adobe as well as, you know, all applications we're, we're better at finding those kinds of, of, uh, of defects of vulnerabilities now than we were 10 or 12 years ago.

We, we, we have, and we also have new, you know, you mentioned, yes, the world's full of Brownfield, not greenfields, unfortunately, we have a lot of legacy code written in c and c plus and even C Sharp, but you know, we're seeing this at the Linux Foundation now, right? Lioness, lioness says we should be using rust. Yeah, there's, there's definitely been a shift, and you've seen it across the industry that there has been progress, right?

Like Microsoft's done a lot of work to introduce secure compiler flags. Yeah. That can help secure code at scale.

Um, Microsoft themselves have been playing with rust in, uh, in their code and they've been putting rust into the kernel. They've written a, a couple blogs about that. So things are getting better, but, um, at, at the same time, it's always a race, right?

So, you know, you're, you're always, they're always gonna find one more way or one more tactic. So it, it's always gonna be a bit of a progression, You know, it's good. I'm sorry, Finding in, you know, security is always constrained by the EE economies of building software and selling it.

So if you can't make money with it, well, even if it's perfectly secure and turning something off is usually more secure than running it. So we need to find an acceptable risk threshold, and for example, for our products a and and reader, the addition of sandboxing to really isolate the memory, unsafe parts. And yes, we have active content and, and, and there too from the rest of this system allowed us even before we had secure by design solutions like memory safe programming languages for systems use to reduce zero days and vulnerabilities in, in, in this area by large decre.

So there are many, many different techniques. And, and the key thing to always figure out is what is the best way, the most cost efficient way to mitigate risks at scale? And as security professionals, we always have a pretty large toolbox available, and we need to help the software engineers understand what are the options and tells them about the different pros and, and, and cons, both short term and, and long, long term.

A sandbox doesn't fundamentally remove the vulnerabilities in libraries that it protects. So we still have to, to fix any B bug we might find. Whereas in a memory safe programming language, you have eliminated or reasonably eliminated a class of, of vulnerabilities.

Yes, the rust you can use unsafe, but how then you better know what you're doing. Yeah. And we're, we're sort of, yeah, you talked about the industry changing.

We're, uh, at a place where, you know, like when I first started, like finding a bug was super cool kind of thing, right? And now, uh, you know, and a large enough company, you, you have, you have tons of bugs, right? So like RS a coming up and there'll be a ton of vendors on the floor who are gonna be marketing, application security, posture management tools.

Sure. Which are, you know, taking into account that you've got vulnerability feeds from all sorts of places. You've got your internal pen test, external pen tests, bug bounties, dast SaaS, Kev list, um, cloud security, posture management tools, et cetera, right?

So you have vulnerability. You, you now have a wealth of vulnerability information available to you. And, uh, part of working together with shift left and shift right, is being able to look at that data and look at that information, say, how can developers most effectively spend their time to, to knock down as many vulnerabilities, uh, with as little effort as possible?

Is it updating their baseline images? Is it, as Florian mentioned earlier, switching language to like react or rust? Is there some sort of tool in the pipeline that we can build that makes, you know, keeping these up to date more, uh, easier, uh, for the developers?

Um, you know, managing third party libraries, you know, since right now we're like at almost at the other end of the spectrum where it's, we, we have a wealth of information. Now the question is, is how did, how do we use that information effectively? Well, we're almost a half hour in and we haven't mentioned ai.

It's time. You know, is AI the answer to that question, fellas? Uh, AI definitely helps.

Like AI is, is another tool in the toolbox, right? Uh, so, you know, you can use on the shift right side, there, there are places to use it. And I'll let Ian talk about, uh, places in shift left.

Um, in, in the shift right side, like, because you have all these different tools, you'll have the same bug finding for multiple tools. And the a common, uh, AI function is document similarity search. So you can do, you can do deduplication, make sure that you're not double filing bugs against teams.

Uh, there are tools, uh, to make reproduce, uh, the reproduction of tool, uh, the reproduction of a vulnerability, uh, easier. So they can take a bug report and translate into a nuclei template, which, uh, utilize an open source tool for, uh, doing scanning. Yes.

That, that helps the development team in terms of reproducibility, uh, when they get a bug report. So there's definitely places where, where it can help. And we've seen, uh, places where it helps and also places where it expands, you know, the attack surface that I have to monitor as well up to, yeah, Expanding the attack surface is a good, good keyword.

We are living in a world where more and more code will be authored, or at least co-authored by AI systems. And these large language models, which writes this code for us, have been trained on publicly available source code, which of course has been written by humans and has sometimes a lot of security issues. So you might find that AI generated code is not substantially better and maybe not substantially worse either than human written code.

But since much more code will be generated than humans can produce in the same amount of time, we should probably think about, uh, addressing these concerns at the root cause. So can we get into the space where, um, AI generate code for us to directly influence how the code is generated and take care of security recommendations at code generation time? That is as far left as we can, can go in, in, in the process.

Um, at least for, for code, we can could also use AI to, or to generate code fixes. So if you understand a, a pattern well enough have AI after it was somehow detected, have AI rewrite the code and so that it's, um, vulnerability free, for example, from using string conation to create SQL statements to parametrized queries. That is, especially Im important when queries need to be dynamically con constructed because in that's the edge case that humans often get to get one.

We are also running other AI experiments, for example, on our block. You can, can find a post about how we think of AI for use and, and threat modeling. That is an, an experiments that, that we are still con continuing to, to this day, to, to see can we augment something where humans truly excel at with AI use to scale it across the entire company.

Because, oh, economics, again, you can't start model every tiny feature by a security specialist, but AI could, is it good enough? And the answer is still, still open, but let's, let's see how, how this space e evolves. I don't think it's good enough today, but it's getting better every day.

Certainly. And an interest thing we hear from security companies and developers is that today anyway, AI might be better at fixing bugs than it is writing code. So in other words, if you give a code that a human wrote, it could find and fix vulnerabilities, bugs, whatever you want to call it, and it does a better job than that.

And then if you just ask it to write code for an application, then it, of course, a human or someone else has, you know, something else has to look at that code. Um, but certainly we're not at the point where, where I think we can trust it to just write the code for us. And, and, and security is, is, is at the top of that list.

Very much so. Um, but you know, you mentioned something before about third party components, and this has really been a bane of shift left and shift right of shift everywhere. 'cause we have to be in the repos.

I mean, today software is assembled on an assembly line, like cars are, I assume it's the same at Adobe, you're not a right. Most of that code inside of these applications represents components that come, they're open source perhaps, or they, you know, they come from repos, container repos or, or or whatever. And, and a lot of the security incidents that we read about or hear about are the result of third party vulnerabilities that made their way into code, not from the developer actually writing that code at the company, but from the third party component that was assembled into that code.

This the software supply chain, this whole issue of SBOs software biller materials, right? And that's a left and right issue because you know what, when you're assembling the code, integrating it prior to deploying, yes, you wanna make sure your s om is, is up to speed. But that SBO has to almost be a dynamic document that, you know, as things change, it changes and then pelli you on the right side of the house have to be able to reference that SO to say, Hey, does this thing need an update?

Or is is a component here out of, out of, uh, you know, they found a vulnerability, we need to upgrade that component. Are you already starting to rely on SBOs to help fix or to help secure the software supply chain? Yes.

We, we do that is one of the projects i I lead. Okay. So yeah, the basic idea is first you need to figure out where do is your visibility into the software composition limited, especially with cloud native applications, things that are developed in modern programming languages.

Any one of these Python, Java, ruby, JavaScript doesn't really matter. Usually has a good package manager. So it's relatively easy to introspect a GIT repository or a repository for the packages that, um, software depends on and figures that out even at deployment time.

Uh, cloud native security tooling can figure that out too. But there are gaps in older systems, especially CC plus plus, again, just like memory safety is a, a problem there. The software composition is hard to determine automatically.

So we are working on, uh, on improving the ability ability, especially in these areas to understand which dependencies to have our, uh, CNC plus plus based products, how do they relate to internally and to external components. And that of course, this visibility then enables us to, to have a more standardized approach to vulnerability management. And Palace mentioned earlier things such as the catalyst that is CSARs list, obviously known exported vulnerabilities, things that have been exported in the world so we can prioritize the remediation of these issues and a whole lot more palace.

Yeah, and this is also a place where, you know, secure coding often gets talked about separately from just standard coding practices. And this is like an area too where, uh, you know, teams that have good development practices, they have the ability to do automated, uh, testing their environment to confirm, confirm patches, uh, the work that they invest into that actually benefits security. Uh, it's a mutual win for both teams because the more, uh, testing they have that's automated and can confirm something and, and get you closer to a continuous deployment model, the easier it is for them to test these third party libraries.

Like one of the things that allowed developers, uh, have a challenge with, with testing these things is that occasionally there's, you know, a breaking change where you have to go and re-architecture code to, to deal with the new version and they're always scared of that. And the longer that goes on, the higher the probability that occurs. And so, uh, a lot of times, you know, when we're partnering with developers, we'll look for opportunities where the thing that they want is also something that we want.

And you know, so if we see them like, hey, we wanna do initiative to improve testing, just normal testing, like unit testing within the organization, you know, we'll go and we'll back that and say, yeah, the security team believes that would be a good investment as well. So there's opportunities to look for, uh, partnering with, with organizations on that. And then from a shift right perspective, yeah, we have to keep track of all the feeds and one CVEs and, um, which ones are relevant.

You know, are they on the KEB list, making 'em a higher priority, uh, those types of things. So it is definitely something that we would monitor on the shift right side. Great guys, I've got one more topic area I wanna jump in on and that actually brings us full circle back to the beginning.

I said the name of our episode here is Shift Left Shift, right Shift Everywhere. It's not enough to have one hand shifting right? And one hand shifting left.

Those are two hands, they act independently and they're not necessarily coordinated, right? Video directors say don't stick your hands out too far. You go out of camera, so I gotta keep 'em here.

But so your hands are not necessarily coordinated. The idea behind Shift everywhere is coordination left and right working together, right? Not in.

Absolutely. Yeah. Talk to me about how Adobe, other than having you both on the show with me, how Adobe is, is putting left and right together to truly shift everywhere.

Uh, sure. I I can start that one. Um, so one of the things you have to keep in mind too is we talk about shift left and shift, right?

And that's important to the security team. But when you're working with the product team, they, they just know the security org, right? So, you know, the reason why we wanna collaborate and work together and, and come up, you know, make sure that we're coming up with like unified solutions and looking for patterns and looking for higher ROI activities for 'em is they wanna hear from a security team from a single with a single voice, right?

They, they just need to know what they need to get done, um, and what needs to, to happen, uh, to get there. And so, you know, with Florian and I, we we're in constant communication with each other every day, every day of the week, um, about some topic or another where they're, we're trying to collaborate so that when we go to development teams, there is a unified voice and I can say like, Hey, these group of bugs don't deal with them individually. It'd be better for you to do this thing.

And Florian can help you. Florian and his team can help guide you through that. And that makes, you know, just a better relationship between the security team and, and the development teams to, to know that we're not just coming up with work for them to, you know, busy work for them to do to, you know, prove we can find bugs, but that we're trying to actively work with them to, uh, get the most security from, from the limited time that they have.

Um, and, and Florian, do you have anything you wanna add to that? Yeah, um, we have so many different tools and as PET said, speaking with one voice is, is most important. So telling the engineering and operations teams what exactly is the most efficient and effective way to reduce their security burden, that is really important.

And this problem feel might feel simple if you only deal with one product. But at Adobe I'm dealing with many different products. So I need to rely on multiple teams that help both PE and me send this message and amplify it at scale to many, many en en engineering teams that use different tech stacks, have different products, have different business cases, are facing different kinds of threats.

So in really identifying what are the key things from a risk perspective to protect our crown jewels, and this might vary by, by product certainly is very I important. And then PE and I work closely together to figure out what are these risks. We work with our security partners to amplify our message and we work with the security partners to pull in the specialized functions of our security organization to affect positive change.

And the part of that is certainly also evangelizing for security to create awareness because, um, not all business leaders might be aware of the security threats a product is, is facing. So really having a holistic perspective is super important. And there is a model that I use, how, how to think about the, kind of the maturity of, um, the security that that we have.

And I found it on, uh, Colin Green's block. The basic, I I am, when you classically think about shifting left, you start with the development process. So design, write code, build test, deep deploy and and, and so on.

And then left this at the beginning of that process. But instead you can, can have a different model that calling green called sees the six buckets of security risk and see rightmost one where I start is exploited. That is the thing we want to avoid.

Then we have the bucket of unfound. And most risks probably stay there. If you now add investments, you can shift things further left to found externally, for example, via a bounty program, further left, found internally, but manually manual testing, pen testing thing, rat teaming.

Oh, you can decide if that's external or internal, doesn't matter too much. Even further left, found automatically with an automated code analyzer. So solution and even further left to prevent it.

And then ask your safety question, what is your maturity? Where do you prevent risks? Where have you only capabilities to find some automatically or man manually?

And that is much more, more expensive. Then the economical question is not, can I do this at this time and moment, find a security issue, but how can I address the root causes of issues instead of only fixing symptoms? So it's a whole different way of thinking about a vulnerability management program and using all the tools you have at hand to make it better.

That was excellent. Thank you Florian. Guys, as I promised you when we started, I was gonna try to keep this under 45 minutes.

We're, we're hitting right up against it. I feel like we've barely scratched the surface though we have a lot more to go over and I look forward to continuing our discussions in, in subsequent episodes of, of this series. But I think we've laid a great, a great foundation here and, and defined a lot of these things.

And what's nice is sometimes we talk about this in such an abstract way because we don't have a real live company who's actually living and breathing this every day. Adobe is living and breathing this every day. And, and that brings a, a, a reality show, if you will, aspect of things where, hey, this is, this is what we're doing and this is what works for us.

So thank you both for coming on. Thank you to Adobe for participating in this series. Thank you for watching this.

I hope you found it interesting. Um, if you're watching a summary of this, click through, go watch the full, the full 45 minute version. It's great.

Until next time, this is Alan Shimel for Techstrong. Thanks for what, what being with us today.