Techstrong TV August 27, 2025

Hey everyone, do I have a semiconductor for you? You're watching Textron gang. Happy Wednesday everyone.

Welcome to Wednesday's Textron Gang show. I'm Alan Hummel, and it's great to have you here. As I mentioned, uh, we're gonna be talking a lot about the semiconductor market, new report out of fu and we're gonna dive into, and we've got some other great things to talk about.

Clean energy, uh, a report of a a, a report from a class trip to share by our own Stephen FST and more. Let me introduce you to our gang for today, though that we're gonna discuss it with. I mentioned already the man from Ohio, Stephen fst.

We also have, we didn't know, she's also from Ohio, a Cleveland gal, Kate Scarcella, uh, Chris Blas, Dan o Dan O'Brien's joining us. So he looks like he's back home in the Boston area. And Mike Ard from a Dark Cave somewhere in Las Vegas is what the word is.

But gang, welcome. Thanks for joining us today. Let's jump right into it.

Mike, I mentioned a new report came out from Ray Wang over at fu. Uh, it was kind of an update to his Q1 snapshot of the semiconductor data center business. And it, it's gone, uh, I think in 2024, the number was 230 some odd million billion dollars.

And by 2029 forecast to go to $500 billion. So almost double in just a couple years here, four years. Um, I wrote an article, actually, I wrote two articles, one on Techstrong IT and one on Techstrong ai.

The, the summary of the report I think is available on FU research, but what do you think? Well, when I was small, 500 billion sounded like a lot of money, but these days I'm not entirely sure. So Dan, kind of walk us through this report a little bit, if you would.

And is it your sense that this is a huge number or is this kind of like just steady as she goes? This is the course run. Uh, I think it's a good number.

Um, you know, encourage folks to head over to the new futurum website and sign up for a kind of early peak at this, uh, for all of our subscribers long into the future intelligence platform, you can see the report right there. Uh, listen to me, I, you know, I think it's pretty well, you know, regarded the consensuses that, you know, you get a trillion dollars semiconductor market coming up, um, in just a couple of years. So, you know, half of that spending coming from the data center, you know, certainly passes the sniff test for me.

Uh, that's about a 22% cop on annual growth rate over the next few years. So, you know, relative to the results that we're putting up, and obviously we've got Nvidia earnings tonight, so we'll check in there and kinda see how things are tracking. But, um, I think it's a really reasonable number.

Um, you know, really not just coming from the GPU side though, you know, Ray really kind of highlights the growth coming from the XPU side as well. You see these custom AI accelerators taking, uh, you know, taking some share, particularly on the inference side. Companies like Broadcom, all Chip media, tech, Marvell, um, you know, a lot of growth happening there.

You know, it's, it's the number one driver in the market right now, right? You know, it seems seemingly feels like the entire economy is kind of, you know, resting on the AI infrastructure build out. We've got, you know, trillion and a half dollars in ai CapEx committed, you know, across kind of the large hyperscaler set over the next few years.

Um, feels like a very reasonable forecast to me. You know, there's a question in there I wanted to add to that, though. Does they, do these numbers take into account what we're seeing on the political front?

And are we moving towards a world where, you know, each semiconductor company is gonna have a version of the same chip that they're making in different markets to comply with different political and economic rules and regs? And is this number low? I, I, it could prove conservative.

I mean, I think, you know, anybody who's forecasted this market, you know, the last several years has probably, you know, ended up having a conservative view relative to reality, right? Obviously, you, you see that in the market and, you know, just the growth trajectory of, you know, companies like Nvidia, um, and their supply chain. But, you know, I think you're kind of alluding to Intel, the government support behind Foundry.

Listen, dual sourcing is nothing new in the semiconductor industry, right? Um, we've gotten away from it in more recent years where, you know, TSMC has really kind of owned a lot of the designs, but, you know, think back to the big apple boom when, you know, the last big, you know, driver of the semiconductor market and the smartphone was happening. You had Apple's dual sourcing across Samsung and TS and C with their Aeries chip sets.

So, you know, I think the design companies would love to see, you know, those variations where you've got, you know, multiple different skews kind of happening outta different foundries to comply with this sort of stuff. I think you're seeing that trend happen anyway. You know, look at the number of chips within the TPU family generation to generation.

You know, the broad trend in the market is more heterogeneous compute. It's more custom built for the workload. And, you know, certainly some of these new requirements around where things are made and, you know, the export restrictions around it, you know, that's only gonna accelerate that trend, I think.

So Mike, I've a few thoughts on this. So, and I, I, you know, I had a chance to slack with Ray, actually didn't actually talk. We slacked and, you know, read the report.

Couple of things. First of all, it very much is an all in, right? That $500 billion number could just as easily be six, 700 billion if AI goes hog wild, right?

I think it's a conservative number in that scenario. By the same token, you know, we're seeing a lot of people starting to, you know, reign on AI parade a little bit. It's become fashionable to become a doubter all of a sudden.

And, and if that market slows down, you might see only $300 billion or, or some, you know, something less than 500 billion. So I, I think we're at a, you know, everybody's put their chips in the middle of the table here and, and we'll see, you know, what the inside drawer is or what, I forget what they call it, and you're not, the cards in your hand, the cards that are on the table. I'm not a poker player, as you could tell.

But anyway, it, it, it's, it's, it's in there. Um, I do think the whole chip sovereignty, if we could call it that thing, is going to be an issue. Um, I wonder how many of these companies the US will have a stake in, you know, as the government.

I don't know if that's a good or a bad thing, but here, here's what I, what I do know, I think we're going to see the world bifurcate in terms, not, not nation states. I mean the market. I think we're gonna see the nation or the, the market bifurcate into companies that design chips, which is what we have now a lot, and companies that make chips, and quite frankly, that's T SMCs, uh, you know, yeah, that's their business.

You know, how long I, I, I'll, I'll give Pat Gelsinger credit where credit's due. He recognized that, that that is a market, a huge market. How long are we gonna let them have it to themselves?

We've got to develop the capability. When I say we, I, I mean Intel, but I mean other foundries. We need a strong, healthy, competitive worldwide foundry marketplace where people do have choices to go to two or three.

You know, once I design my chip, I could have a built in two or three different foundries, if you will. And if we could get there, then you've got a real market. And you know what, I'm not a big fan of government ownership of means of production.

I'm a, I'm a free market guy, i's to see your market's. What, right? I mean, TSMC is government backed.

You've got the Japanese separate with rapides, government backed intel, now backed by, uh, Well, there, there's a difference between government backed and government owned, but in, but look in Taiwan there that no one ever claimed Taiwan is, you know, Singapore's a great country too, but people there give up their freedom for economic stability. That's not our way. But I'm not here to debate that.

Yes, you're going to need governments involved in this, right? You need government blessing, you need government help. And it, and it's more than that.

It's a critical infrastructure imperative, a strategic imperative for governments to make sure we have that. But the best medicine we could have is a healthy market of multiple players, so that the people who are designing the chips, the apples and so forth, they have choices of where to go. And, and you may get some who, who are both, they both design and, and fab like Intel always did, and more power to them as well.

I think for me, when I think about this situation, it's that, you know, why we see billions being spent in the, in the data center. My concern is the, isn't the lack of demand, but the overbuilding. So I think that we haven't seen AI unleashed and what it can actually do.

And so I fear that we're gonna have all these huge data centers and, you know, and we're not gonna need them because AI will improve everything that we do. Yeah, that's, that's a real good point, Kate, because you know, one of the things that I've seen mentioned, people love to compare the current AI build out to the web, uh, build out, or the fiber build out from previous booms. And yet, as you point out, uh, one of the challenges here is number one, that we haven't seen a really profitable application of AI yet.

And that is, despite the fact that the true cost of the infrastructure and power is not yet reflected in the product, but also one of the things that gets pointed out by analysts is that a lot of the spending on capital equipment is on capital equipment that may in fact be obsolete in a few years by the time the, um, the market matures. So if we assume that AI will have productive and profitable applications, a lot of these things that are being built out, um, may not actually be all that useful. And the, you know, investment may not come to fruition.

Somebody pointed out the other day that you can currently buy a, um, a 48 gig, uh, a 100 Nvidia processor on eBay. Uh, guess what, $2,500 as you and a 100 on eBay, uh, right now. And, um, that's because it's effectively obsolete by the next generation that's already being deployed.

Well, what happens when the next generation can the next generation come out? Uh, you know, unlike, for example, the fiber boom where all that dark fiber ended up being incredibly valuable, powering the internet as we know it, unlike even the data center boom where things go obsolete, but they don't go obsolete next year and not quite so radically. I mean, Google and Microsoft have announced that they're gonna have 3, 4, 5, maybe even six years of lifespan out of some of the servers that they've bought.

Uh, will we get six years of lifespan out of the ai, uh, processors that are being bought? I'm not sure. Nah, I don't think so.

I think Have to worry about that, the financial engineering of this, all right? I mean, you know, companies, to your point, Steven, are, you know, depreciating these assets over 4, 5, 6 year timeframes and, you know, realistically you could argue it could be a year or two is more realistic. So Really, or most, yeah.

Hey, Chris, certainly Chris, if your mic's back on, you wanna weigh in? Yeah, I'll come out on the Buller side of this one for structural reasons. I mean, all the reasons you guys are, uh, laying out are, are, are accurate.

But look at what we've been talking about in this form in this show over the last several months, right? We've been building up massive stacks. We use AI on massive compute, and really, there's all sorts of horizontal space that we, and we've talked about the per proliferation of other chips.

And, you know, I, this little thing, this little ai, you know, I'm afraid to say it's named because it, it'll be stupid needs so much more power. So when I see, you know, I, I, all the factors, uh, you're bringing up geopolitical things are just driving it further, but we're, we're maxing out on the vertical verticality, you know, perhaps in the processor space and usage, which is leaving all sorts of, all sorts of room horizontally. So I think the, I think generally speaking, this is right.

I'll leave the, you know, I'll, uh, in this particular topic, I'll leave the market analytics and timing to others, but it just feels right. You know, this, this is, there's gonna be a lot more smart stuff in a lot more things, and that means a lot more chips and they're not all the same. Yeah.

And you Know, to Steven's point, though, when I talk to AI developers, I mean, it's amazing when there's a new chip out, that's all they want, and they don't want to hear from the old chips, and they're just like, as far as they're concerned, they're like, scrappy junk already. So, but, But why is that any different than it's ever been? It hasn't, no, I, I mean, right.

We always want the, the greatest and Everybody wants the latest and greatest I, the penem, I, the, the core, the I seven, no, I want that I nine, when's the I 12 coming, right? But as Dan points out, these things are heavily financially engineered as well. And so it could very well be that we could both be right, that effect effectively today's, you know, CapEx build out may not end up being profitable, but the profitability of future applications may wipe out any, you know, even the tremendous CapEx that we've made now.

And to Chris's point, I, I totally agree with you. So much of this market is unaddressed. You know, if you look at this report, all of these companies that are developing, I mean, the thing that makes me excited is to think about sort of what is, you know, what we're gonna see from companies like Broadcom and Marvell and Media Tech and so on.

Those are the companies that are gonna be building just incredible stuff on all of the, in all of these new fabs, there's an appetite for increasing compute everywhere, not just in ai. Agree. And, and I wanna, if I could just drive that home a second.

It's not just about NVIDIA's GPUs this report and, and do go check out the exact summary, and if you like it, get the whole report, be, you know, subscribe. But this report makes clear that while NVIDIA's going to get their fair share, and then some right. Take no shed, no tears for Jensen, but it's the Broadcoms, it's the up and Karmas like Marvell.

It's a lot of these other chip makers or chip designers, players in the market who are gonna reap the benefits here too. And you've gotta be, if you're gonna be smart about this market, you gotta be familiar with those folks on what they're doing too. So I highly, highly encourage you.

Go check it out. Someone else wanted to say something? I just wanted to clarify one point about the separation of design and foundries.

'cause I believe credit for that belongs to Arm. I think they started that whole trend. They did, They did pretty good with it.

I heard. I think a MD might also deserve a little bit of credit there. Yeah, listen, I mean, the, the fabulous trend, you know, has, has been playing out for, call it 20 years, right?

The market's been moving that way. Where, where you see the IDM trend really still in play is not the leading edge, you know, it's all the trailing edge. It's the Texas instruments of the world that Are still, don't, don't forget how many chips.

They, I still does a lot of calculators. On that note, let's take a break here, and we're going to come back and talk a little bit about clean energy to power all these chips for $500 billion. You're watching Textron Gang Discover Textron Group, the epicenter of tech innovation.

We are your go-to for reaching IT leaders and practitioners worldwide. Our secret impactful content that sparks awareness, engagement, and top quality leads with us. You'll access editorial websites, streaming videos, virtual events, custom content analyst research, and more.

Join our satisfied clients. Let's revolutionize your tech journey. Contact us today and tell your story to the world in the most powerful way with Textron Group.

Hey, folks, we're back in, is Alan alluded. We're gonna have a little chat about clean energy and big tech. The big tech companies sent a little memo back to Washington saying, please don't kill those clean energy credits.

We need them to drive data center investments, especially around things like nuclear energy, which everybody and his brother seems to be now saying that they want to be a part of. Alan, I know you kind of looked at the space and, um, you know, I know you have opinions about what happens in Washington, but yeah, do you think we can, you know, save this clean energy credit thing? 'cause maybe it is one of the few bipartisan things that's good for everybody.

We, what's bipartisan mean? Even today, some stop, just stop. There is no more bipartisan, it's ruled by fiat.

But here's the thing. I happen to agree, I think, look, when 92% of the energy that came on last year, new energy that came on in Texas last year was solar and wind. It's a little bit crazy to try to just kick that to the curb.

That being said, we don't have bipartisanship in our government, nor do we have rule by a a committees like that. It, it's whatever they decide, and they've decided that clean energy and, and I don't even think they like to use the word clean energy. You know, they don't like solar and wind and we're not using it.

God darn it. And that's the end of it. Um, the good news is, the good news is that it's funny how nuclear energy has been brought in from the woodshed and, and is now everybody's little darling.

And, you know, I, I, including mine, I'll be honest with you, I'm looking forward to see container sized nuclear plants, powering data centers, powering medium sized cities. You know, I, I think we've come a long way since, uh, three Mile Island. And, you know, in New York where I grew up, where I grew up, what was it, Indian Point, Mike was the, the nuclear power plant.

Never had a problem there. It supplied New York City with tons of power. Um, but those were, you know, those were designed in the fifties and sixties.

It's been a minute. We've made improvements. We need, we need all good energy that we could get.

If you, if you buy into the 500 billion trillion dollars semi market, we need all the energy production we can get. And, and nuclear might be our, our best bet. And in, given the current political climate, it is, given the current political climate, it, it might be the fastest road to getting there.

Yeah, I'm with you with, uh, nuclear energy. When I did my thesis for, um, for information security and securing the electrical grid, I actually went to nuclear plants to look at what we were talking about from a cybersecurity point of view. I love nuclear energy.

I think it's this untapped, um, it's really something that we have been scared of, uh, I think growing up because of Three Mile Island, uh, the movie China syndrome, which some of you may remember. But, um, but it, it's good. I mean, it, it's nuclear energy.

And at the same time, I don't think we should stop the wind projects that we have in, you know, the New England area. Um, I think like all technology, the more that we use it, the more that we understand it, the more that we understand, the more that we're able to, you know, understand, you know, storage capacity. And, you know, this almost leads me to, you know, um, I was gonna say IBM and storage, but the whole other, that's the next section.

But Chris, what do you say? Yeah, I, I'm a nuclear hippie, right? You know, I, I, I agree with everybody else, and, and frankly, I've never been, been worried about a nuclear power.

In fact, you know, I, I, as a young person, I thought I might, uh, spend some time with Greenpeace, and I talk to an actual Greenpeace person. It's like, save the whales again, anti-nuke. I had no nuclear war.

It's like, no, we mean no nuclear power. It's like, wait, that's actually better. But to, you know, to weave this into the point we, you know, the reason that I'm bullish about the future writ large is that we have a million reasons to need the same thing.

And that's visibility, right? It's not about supply chain security and knowing where your software bill of materials is necessarily, maybe. And it's not necessarily about knowing where you're green, your impacting the environment because your usage and production of, of power, maybe.

But it's all of those things sometimes. And the same systems do the same thing. The the lake Great.

Julie Goodman, in 2019, when Medi and Za and I first, you know, had that angry conversation and ended up putting together the deb bomb attestation ecosystem, we meant supply chain security. We had mainframe securities. We talk about the next segment.

But, uh, but it turned out that surely wanted to, uh, um, had a system for carbon credits. How do we actually track them? Is it better?

Is it not? Look, you know, recycling, you know, consumer recycling in my generation was mostly a scam. A field good scam that caused more pollution that it saved would've been better landfilling at all.

You know, I don't care what the answers are, I don't think any of us do. But we would like to know what works and what doesn't. And in each of these things, we are, we're developing the ability to know, and then we can decide.

Yeah, the challenge, you know, I've said for a while that, um, one of the, the biggest challenge for nuclear is that it's frankly just not financially competitive with alternative energy sources. Uh, you know, as of today, uh, the projected cost for these nuclear, these small modular reactors, is actually three times the cost of installing solar and batteries. And, um, and easily three or four times the cost of, uh, wind turbines or solar panels alone.

And that is a big challenge. I said, you can't put that genie back in the bottle because you know, money, you know, essentially, uh, solar and wind are so financially competitive. It's actually cheaper to build out a new solar plant with batteries than it is to continue to power your old gas, natural gas turbine today.

But the challenge is apparently you can put that genie back into the bottle by denying permits and shutting down projects. So that's kind of where we're at. Um, given that I actually am gonna come over to the Allen side of the table and say, let's go small modular reactors, you know, I mean, you know, if, if we need something, and these things are, you know, they, they are very promising.

These are not your dad's nuclear reactors. These things that are, are cooled with molten salt. They have, uh, you uranium in, uh, pellets that is, uh, much, much safer.

E every aspect of these things is much, much more attractive. No, they're not financially competitive with alternative energy sources or even with natural gas and coal, but they are competitive in other ways. And if the externalities of especially fossil fuels were weighed in these things would be more than up to their, uh, competition.

And I think we need 'em. Agree. Well, I that's the word, right?

Sorry, the word right there weighed in. Right? We all of all ideologists all right, human beings, we just wanna know, and I have shared a lot of nuclear power, uh, uh, conferences and so forth, sell fields, like you say, Steven, the cost, no, they're completely outta line.

That does not mean they will be forever aligned. We, I think we made mis mistakes in choosing fuel cycles in the 1950s, whatever, right? But whatever the answers are, we, I, it's not safe to be locked into anything.

Our mainframe is right or wrong. Do we need more chips to where the power come from? None of us know.

We do know, though, if we can see things clearly enough, not that much, but just enough to be able to see we can make reasonable choices on all these things. We don't have to take sides, our sides. I don't, I don't think people really care whether it's clean energy, green energy, nuclear.

I think that there's an imperative that we need more power and energy in the country. Like we have created a new application that is incredibly power hungry. And I think the name of the game is just supply here.

I mean, much like, uh, we've talked about in the previous segment, you know, the ability to build leading edge chips as a kind of national strategic, you know, imperative and, you know, strategic advantage. I think having, you know, a huge amount of excess power that we can generate to drive this AI thing forward, I think is also just, it's the end game, right? I mean, you know, I think we're, we're looking at a world where having, you know, the most advanced AI clusters and quantum computing is, you know, a strategic military advantage even.

Um, so I, I tend to think the name of the game is power at all costs. And you know, I think if you look at some of the math on, you know, if you triple the power costs, what that means to, you know, JUULs per token, you know, it's negligible in the grand scheme of things, given the scale of the investment and all the other areas that we're pouring money in here. You said something though, Dan, excess power.

I think one of the problems we have is we have nowhere to store excess power. We need battery. I mean, this, this is a tech, this, this whole thing is a giant tech puzzle, right?

If we, well, on that note, uh, as of right now, global battery supply is actually running three times demand. So we have, uh, actually excess supply of batteries right now, too. Amazing.

But, but our batteries aren't as efficient as they need to be Because there's so much that, you know, and, and Dan, we actually can care about multiple things at the same time. But I'm, I'm totally with you, Dan. You know, look, everybody who wants to go live green in the woods and eat granola, we are gonna use 10 times as much power get over it, right?

Everybody wants to burn coal. No, we can't do, you know, do that long time either get over it, right? We just need to weigh these things out, right?

And exact, there's so much room in the middle, we'll use a lot more power and we'll use it much more, more wisely. But, but we using power now, Let me just 10 my tree hugging hand here for a second. And Steven, I, I hope you'll join me in this, okay?

You're a hundred percent right, Dan. You're a hundred percent right too. That's why it's moronic to throw away solar and wind just because somebody doesn't like it when it's probably dollar for dollar, the most efficient one out there.

Now you wanna get rid of clean. You wanna say, look, let's level the playing field. Let's not play favorites.

Energy is energy. I don't care whether it comes from fossil fuels, uh, uh, sun, uh, solar, uh, nuclear wind, liquid gas, fossil fuel, whatever, dollar for dollar for dollar. Where do I get the best bang for my buck?

And that's where you go. Now I'm not, that's do with's Coal's. It's not one size fits all right?

Different geographies, different use cases, different solutions, right? Exactly. Coastal city versus the planes versus, you know, the middle of the desert.

You know, let, let's, let's, you know, like we talked about in the previous segment around finding the right solution for the right workload. You know, I think we're talking about the same thing on power. Yeah.

And, and I, another aspect that was in our notes here is, you know, Microsoft is piloting a water free data center that would be great for Phoenix, Las Vegas, you know, places where water is at a premium. I, I don't know enough about it, Stephen. I know you are my, you're my green man.

What's that about? Oh, that is such a great idea right now. Uh, it would honestly, it's very analogous to sort of the current state of nuclear versus the future of nuclear.

We've been pa we've been cooling our data centers using evaporative cooling because it's cheap, because we can pour water into it. And you pour a, what is it, a liter and a half of water equals, you know, a kilowatt of cooling or something like that. Essentially, you, you evaporate some water, it gen it, it, it cools and you're good instead.

But that, the problem is we're running outta water, especially in places where these data centers and fabs are being built. You know, you look at Utah, Nevada, you know, Arizona, Texas, and even in places like Ohio and Virginia that have lots of water, we're still running out of fresh water in order to do, to cool cooling because it's just not in the right place at the right time. If, you know, and again, it's one of those trade trade-offs.

You look at it and you say, wait a second, just like, uh, power generation is inconsistent from, um, renewable sources, and at night the sun doesn't shine well, then you gotta put batteries, and batteries are add a layer of inefficiency and you end up wasting some power. Same thing with cooling. If you look at the cooling problem, you say, wait a second, let's stop pouring water in here, and let's do air to air cooling.

Yes, it requires more electricity, but it doesn't require that water. Okay, good. Solved.

Let's do it. Right. I love this.

Excellent. All right. I would just point out one thing to Steven's point about China.

'cause when I talk to AI developers, one of the, and they go visit, one of the things that they come home and remark on is like, Hey, nobody over there is freaking out about where they're gonna find the power for their AI project. Oh, it's a, it's global. It's not just a US issue, but hey, we're, we're over time on this segment.

I want to come back. We're gonna get our little field report from Cher. com is the leading resource for news analysis and education on challenges facing the cybersecurity industry.

com covers all aspects of cybersecurity, including data security, DevSecOps, cloud security, application security, network security, security threats, and more. com has the largest selection of security content featuring breaking news, blog posts, podcasts, and more. com to learn more.

com. Home of security bloggers network. Hey, folks, we're back and we're gonna have one of our famous little field reports from when one of us went to visit somewhere.

Steven was at the chair conference, which if you're not familiar with, is kind of a, I guess I'll describe it as an emotional support group for main framers. But, um, oh, Okay. But Steven, what's going on with these folks and what, what are you hearing on the latest and greatest on mainframes?

Sorry, my head just exploded. No, uh, share. First off, Cher is actually a remarkable thing.

It is the longest running tech conference. I, I said another in the known universe. Um, we don't know, there may be one running longer on Alpha Century or something like that.

But here in the, uh, on our planet, Cher is the longest running TE conference. It was, it was started in 1955. Um, you know, so Alan, you know, you're the, I think the only one that remembers the first Oh, oh, anyway, uh, no, it was started in 1955 before any of us were born.

And it is still going strong. The other thing that I love about share is that it is deeply, deeply technical and to the name very much about sharing. So if you love that sort of, um, hacker vibe of CubeCon where people are sharing their code and doing code reviews and stuff like that, share is kind of like that.

Except as you point out, most of the people, in fact, pretty much all of the content is mainframe related. And you might say mainframe, isn't that just a term they misuse in movies? But no, uh, actually most of the world's transactional data still runs through mainframes.

It's still a massive, massive business. There's big companies, I mean, IBM obviously, but Broadcom derives a huge amount of revenue. Uh, companies like Rocket Software are huge in this space, BMC, and those companies are still powering much of the, what the world is doing.

Now, I went to share last year in Kansas City, um, I missed one, and then I came back this year. It was actually here in Cleveland, which was tremendous. I think everybody loved Cleveland.

Um, it actually does rock. And, um, and the conference, you know, the things that I wanna zoom in on here from a, from a content perspective, are the many ways that share is showing sort of the evolution of the mainframe. IBM, first off has a brand new mainframe, um, that's out and being deployed right now.

It has AI capabilities, it's got accelerators. It looks very much like if you were to describe the hardware and forget to mention what operating system it's running, you'd think, man, this is a pretty cool new modern system. It is.

It's running mainframe stuff. You know, the other thing though that, that really strikes me about Cher is that there's a generational change happening right now. I heard from some of the insiders at the conference, some, some of the sessions that are kind of closed door sessions, that essentially the old school mainframe folks are finally starting to say, you know what?

We gotta turn it over to this, this young generation, because the young generation is there. If you walked around, I looked old compared to a lot of the people in that conference. Well, You are Steven, Thank you very much, Alan, are you, I owed you that, by the way.

So we're even, There are a lot of young people. It's a very diverse crowd. It's not at all who you might guess would be at a mainframe conference.

And they're doing cool stuff. They're applying DevOps technologies. Uh, we heard, uh, tech Field Day presentation, a company called Popup Mainframe that is on the leading edge of doing essentially, you know, it's kind of like containerization, what containerization did to modern application development.

They're doing that on the mainframe, right? Um, we've got observability, we've got ai, uh, we had Broadcom talking about how they're doing AI ops on the mainframe platform. And it is legit.

It is like what you would expect from AI ops in the cloud, or an open systems, except with a mainframe slant. And there's all sorts of new people. I mean, one of the big things that I focused on on Wednesday was a lot of sessions on running Linux on the mainframe.

There's a few different ways to do that. And it's not about basically having better hardware to run Linux on. It's about connecting all of those mainframe applications to the modern world, you know, the open systems world through, you know, Linux basically as the bridge.

That's just incredible. Now, I know that y'all have a lot of thoughts on that. I just wanted to kinda share those are my takeaways from share.

Uh, what do y'all think about the modern mainframe? So I I, I've been covering, look, I've always been like fascinated with DevOps and mainframes ever since. com 12, 13 years ago.

And I, I'll tell you something, uh, we don't give IBM enough credit for what a fantastic platform that Z is, right? So I have a good friend at IBM, Roslyn Radcliffe, Dan, I don't know if you ever met Roslyn over there. She, brilliant, brilliant woman.

All of their mainframe people I met over the years are, you know, and let's face it, everybody in the mainframe industry worked at IBM at some point or another, right? Because they all, they all gen. That's the genesis of it.

But you wanna talk about taking a platform that is, I wasn't there in 55, Steven, but it is, you know, that old, but refreshing it, making it modern so that it does it. Look, it was running containers before we ever heard a docker. Let's be, uh, clear, right?

It Docker didn't invent containers. Mainframe's been running them forever. But it runs Linux, it runs Java, it runs it.

There isn't much, it doesn't run. You want to systems of record, systems of engagement, bifurcate your stuff, put some stuff in the cloud and make it, you know, hum back here on the mainframe. It does it.

IBM consistently refreshes this platform. Is that every two or three years, I think is the cycle. It's three years, right?

Dan, two year cycles. And every cycle is just rocking rock, solid rocking. We sit here today and, and there are a lot of young people who maybe want to make fun of it.

But Penny, for Penny mip for mip, dollar for Dollar, it's still your most effective platform for compute, for, for mission critical stuff. If security's important to you, if uptime is important to you, there still isn't anything that touches it. And just one last point about young people, Steven, I'm glad to hear that you saw all these young people.

I want to give a shout out to within the Linux Foundation, uh, a project called the, uh, open Mainframe Project. Yep. OMPI, I saw them, I talked to them there, Great people, my friend John there.

And, and, uh, the lady who runs their marketing, I, I did the Open Mainframe Project podcast for a year and a half, two years with them. And they do an amazing job of education, of, of bringing up like scholarships to people. You know, it's a funny thing in India that turning out COBOL programmers like mad because they know they could get a job.

There's, there's a shortage of them. But the Open Mainframe Project, and, and I'll give them a shout out and also say they don't get enough support from IBM and BMC and Broadcom and Rocket, the big four in the mainframe space. We should have a better mainframe circuit events, right?

Share is, share is share, right? It is a very unique conference and everything like that. But you look at what they do with CubeCon, right?

Why, why can't we have that kind of buzz and craziness on, on Mainframe And, and the Open mainframe project's a great place to, to do it. But look, DevOps runs great on mainframes, right? So, so Dan, let me ask you this question though.

'cause part of the issue with mainframes isn't how robust the platform is. I think everybody knows that it's awesome, but the issue has always been the price of entry into that environment. And it used to be kind of a million dollars.

Now I think it's probably a half a million, or where do you think the point of entry is? And that's why everybody winds up buying a bunch of X 86 boxes. I, I think that's changing, Mike.

I mean, I think, you know, Steven alluded to it a bit earlier. You know, there's a flavor of the mainframe called Linux One, um, that takes all the Z hardware and, you know, really creates an open system can run r you can run OpenShift containers. Um, and I think there's a lot of those X 86 workloads are transitioning.

Certainly ARM is picking up the majority of them, but there's a lot of use cases where the total cost of ownership, you know, really argues quite strongly for, you know, for the Linux one and the mainframe system. Um, you can check out some of the work that, uh, signal six five did on that, um, you know, a, a lab project that, uh, the company did in, in, um, in concert with IBM on that to kind of prove that point. Um, listen, the only thing not modern about the mainframe is that it also runs really old quote, right?

Like it's got backwards compatibility like nothing else. Um, everything else about it is completely modern. It's got the highest clock speed, almost any chip out there.

You're doing, you know, uh, quantum safe encryption. You've got, you know, on-chip accelerators that are, you know, doing incredibly low latency compute inside the credit card transaction window now to detect fraud detection. Um, I mean, there's, there's an incredible amount of in innovation that they're pouring into that system.

Um, and I think, you know, in a world in which, you know, we've all kind of recognized it's a hybrid world, um, we talked about this concept of heterogeneous compute really finding the most efficient way, um, for every workload. You know, the mainframe, you know, has really a, a, a lot of new life into it. Um, you know, with all these strengths kind of converging out there at the market, It, it takes about three generations, you know, for innovation to turn into infrastructure, right?

And I think this is a fascinating example. 'cause the 70 years out, you know, from the first to the, the share conferences, right? The human generation is 25 years, right?

So there's about 200 generations of human written history, say 500 since we started doing infrastructure. And as you look at these transitions, when, you know, somebody gets a crazy idea to start directing the water towards the plants, right? And now it works, there's a lot more food and it booms and it's a, it's a great time.

And then there's a drought and you know, if you weren't, you know, the man water management person and, uh, the survivors will kill you, right? So you end up with standards and so forth. We see infrastructure maturing over and over and everything in this, in this show.

All, all the episodes we're doing that sort of thing. You know, Hey, we've got AI that's gonna use these crazy chips, so we're gonna put a whole bunch of chips in one spot. But that's not a mature infrastructure that the ot, I mean, we are using, we quiet wire today are using 15-year-old chips to run AI because, you know, a small manufacturer does not need Illumina class, you know, a emerging ai, they need Bob who sits there and watches the, the files that works perfectly fine.

So in ot, historically, we've shown how this is done in OT today, we're demonstrating it, and in it we still think, you know, this is, uh, a new shiny thing when it's infrastructure and now it's responsibility. That means writing it all down. I find that the, the biggest problem, um, that we had with the adoption of mainframes was actually just a lack of talent out there.

A lack of people being able to actually run the systems. You know, there was, that the field was, was very small, so Mm-hmm. And that's actually one of the coolest things too.

Uh, you, you mentioned in, in China, well, here in the US as well, there are a few universities that have basically mainframe bootcamps. And, and in many cases they're smaller colleges, you know, community colleges, that sort thing. Some historically black colleges too.

But let me tell you, people that graduate from those programs can write themselves a check. You know, and again, shout Out to the mainframe, open Mainframe Project. They're the ones behind this.

I, I forgot the professor who runs it at two or three different universities. I've interviewed him a few times through OPM. They, I, IBM helped a lot here too.

I mean, they've created, uh, two AI tools, you know, one to help on the coding skillset, you know, cobal, Java, you know, specialized AI models to help with, uh, you know, the lack of COBOL skills and mainframe modernization. But they've also got AI tools for the operator side. Um, you know, really, you know, kind of democratizing that knowledge out there so that you can interact, you know, in a natural language way and get the help you need on how to stay up and running.

So, Kate, lemme ask you this. 'cause when you talk to IBMers and main framers, there's a little bitterness in the language because, or the feeling, because they essentially say, no one's hearing us. We just talked about all this great stuff that the IBM m mainframe does, and yet, you know, the market's dominated by all these other boxes out there.

I mean, what can that IBM community do besides grouse amongst themselves about the fact that the rest of the world seems tone deaf about this? Yeah, and for that, I would say, you know, let's go back to Alan, because I, I do think, you know, IBM and other mainframe companies really need to reach out to the Open Linux Foundation. They need to reach out to schools.

It is so important for, for schools to start to train, um, people, you know, with the skills needed. So when that, when they get out, they can write their own checks, you know, so I think it's a, it's a matter of, of a community coming together and helping say, you know, this is really good technology and just getting it out there. I think it's important.

Yeah, and just as quick call out, uh, one of our folks around the tech field day table, Jeffrey Decker is a professor at Northern Illinois University, and he runs one of these boot camps. So if you're interested or you know, somebody who might be interested, Northern Illinois University in DeKalb, Illinois has just the program. Hey guys, I gotta call it we're outta time.

But just quickly, Steven, the Tech Field Day, uh, uh, videos from that you did at Share, they're available up on the Tech Field Day YouTube site, I assume, as well as on Tech Stunk TV Going up today. We just, uh, putting 'em through some review process and, uh, we'll get them posted just as soon as we can. Good man.

He's pretty good for a young guy. Um, all righty, on that note, we're gonna end our, uh, chauffeur today. Okay, Chris, Steven Dano.

Mike, thank you. Enjoy your day, everyone, as usual. We've got Tech on TV following this.

Uh, enjoy that. You know, I've been looking at the Google Analytics on that. It's crazy how many people are watching or viewing whatever I impressions.

It's all kinds of things there. But we hope you've enjoyed this and enjoy Techstrong tv. We'll be back tomorrow with yet more great coverage.

Until then, this Alan Humma, we're out. Hey everyone, welcome back here to Techstrong tv. You know, I'm really glad, I'm really glad to welcome my next guest back.

It's been a, it's been a minute since he's been on, but welcome back. Colton Andres Colton, of course is the CEO and founder of Gremlin, actually past and present, CEO founder of Gremlin. He was always the founder, but took a little hiatus from the CEO role.

We're going to hear about it. But Colton, welcome back to Text Drug tv. We've missed you.

Thank you very much. I appreciate the kind words, and, uh, it's always, I was, I was excited to talk to you today. I'm always excited to talk to you, Alan, so thanks for taking the time.

Uh, our pleasure, man. So let, let's get right to some nitty gritty. You are, you found gremlin's been your baby, right?

You were founded, you're the c longtime CEO, you took it up to kind of the top of the chaos engineering marketplace, and then you ditched me. I didn't, I didn't hear from you all what happened to you, Colton? Yeah, well, we, yeah, as you said, you know, we had, we came out, we pioneered the, the category.

We did a lot of hard work. We, we grew the business and we hit us point where, uh, I needed to go focus on the product and engineering side. And so I brought in a CEO to help run the business.

Uh, he did a great job. He was around for a few years. He helped us, uh, run the go-to-market side.

And I really, I went to town on the product. I think what we learned in 2020 was chaos engineering, great idea, hard to execute, hard to really get the value out of it. And that hard part was really the people, the process, the accountability, not really the tooling.

And so I had to go back into the lab as, as, uh, the team has said, and I've gone and, you know, had to go focus on really making the product amazing. You know, not just a chaos engineering tool, but a reliability platform. And we did that and we got it in a good spot.

We got engineering humming along, and it was time for me to, to take back the reins of CEO because people like you were missing me. And I need to get back out there and make sure, you know, we were, we're spreading the good word. Absolutely.

And, you know, all all joking aside, Colton, you, you as a founder, you realize it, right? I'm a multiple multi-time founder. I realize it sometimes, Sometimes it, you've gotta be the person who carries the flag, right?

One of the most important, and I don't get it 'cause I'm not a military guy, but one of the most important positions in the military is the guy carrying the colors, right? Because that's what the troops rally behind, and that's what the market kind of focuses on, and that's where people look for signal. And if you don't have someone waving that flag up high, who people look up to or, or are looking for it, it kind of, you know, the morale of the troops goes down, things disperse.

It's just not, it's just not the same. And, and, and it, and it's not a knock on anyone. I'm not knocking right what the CEO over the last couple years has done.

But not everyone carries the flag. Some people are administrators, some people are, you know, very efficient managers, but they're not necessarily a thought leader. Maybe plain and simple, I don't know, a better less sugar.

I'm not sugarcoating it, right. Uh, I don't know a better way, easier way to say it. Yeah.

Well, and I think, you know, SRE, uh, reliability, you know, dealing with outages, it's not the most glamorous work. It's not always front page news unless things have gone horribly bad. And so I'm honored that, you know, you look to me as a flag carrier and I can help carry that flag because I think there's a lot of good work that doesn't get recognized.

And I think there's a lot of good work that could be done that would make everybody's life better. And if we don't stand up and, you know, advocate for that, if we don't justify it, if we're not there making the case and helping the business understand, hey, this is what's best for you and what's best for us, then it just doesn't happen, as you've said. And it's, it's a bit of a shame sometimes to just see, see things lull or, you know, regress when you would hope we're always, you know, making forward progress.

Absolutely. And Colton, I don't wanna make an note. Colton wasn't here.

So the whole chaos engineering and S-R-S-R-E space went to hell in a hand basket that, that's not necessarily what happened here. But, you know, even when we look at the broader SRE market, I, I think, you know, we've seen an interesting confluence of, of factors that, that come into play here. First and foremost is ai, right?

And we're gonna talk about this, right? How ai, and whether you're talking about generative ai, agent, ai, ml ops, ai, you know, what, AI in all its many flavors and forms has, has really made a, a, a profound impact on the SRE and, and chaos space. But then also another thing called this, I think the rise of platform engineering, understanding that, right?

We, we have to have a platform that this factory that where we build software and maintain and run software is running on and, and the SRE space and, and chaos engineering. But you know, as part of it, they have to also play on this platform, if you will. And so that's the rise of platform engineering, I think has had a, a, a role in this, the continuing evolution of DevOps, uh, all, all the above and more, but that's my take.

What's your take? Yeah, I agree with you on platform engineering. I think, you know, I love the factory analogy.

Uh, the Phoenix project is what really helped me. Mm-hmm. Think about software like a factory, and by, by the way, a little peek into what I've been doing the last few years.

I was fixing my software factory and making it run efficiently so that we were not just building cool software, but we were building cool software on time, on budget, all of those things. So yeah, that's, That people want to use. Yeah.

Yeah. So the factory analogy, the, the rise of the platform, I think SE has always been in a tough spot because, you know, we, we wanted to embrace DevOps and we wanted to ask the engineers to do a lot of this work. And then what we ended up with is kind of like a rebranded CIS ops team in some regards.

And I think that, you know, it makes sense from an efficiency and expertise point of view, but I, I see a lot of companies struggle because they can't have, you know, a handful of SREs fix every problem in the company or look over every engineer's shoulder. And, and by the way, with ai, you know, churning out more code and, and, you know, helping velocity go faster, that problem gets worse, not better. Yeah, Absolutely.

Well, I think it forces you to say, look, if AI's gonna drive twice as much code as we did before, we need AI to help us deal with twice as much code Yeah. As we did before. Which kind of brings us to this whole AI driven reliability intelligence that, that gremlin is, is talking about now.

What do we mean by that, Colton? Yeah. So I think one of our goals at Gremlin has always been make it easy to do the right thing.

And, you know, we started with build a great platform that has all the bells and whistles people need that, that's doing, you know, enables them to do what they want. And I think what we learned five years ago is that that chaos engineering platform is great, but you also need to go into the organization and influence the right behaviors. Leadership needs visibility into the baseline of where your reliability's at, the improvements that have been made.

If you can't measure it, it, it doesn't happen. And one of the things that always tears me up is there's a lot of teams that do a lot of good work, but if they can't quantify the work, they can't quantify the value for the business. They don't get credit.

And sometimes that means budgets get cut and teams get cut. And sadly I've seen that amongst my customers. You know, otherwise it means people get passed over from promotions and they're not really recognized, oh, the system, no outages last year, guess we don't need our, you know, reliability team instead of really rewarding those folks.

So we spend a lot of time scoring risks tracking. It's a lot like security, let's really measure it, and let's give leadership visibility. So I think that addresses some of the organizational problems, but it doesn't really address that.

You know, I take this tool to an engineer and I say, great, go run some tests and fix your system. Well, the first question they have is, what tests should I run? And so we did a bunch of work.

Let us tell you that the recommended set of tests, we've got a stock set, it's the same 10 things that go wrong on computers. Let us guide you through that. So, okay, an engineer runs the test and they're looking at their screen and the test failed, and they don't know what to do.

And so that's where we decided reliability and intelligence really comes into play. Hey, you ran a test and it failed. Well, you know what?

At Gremlin, we've run a million experiments over the last decade. We got a pretty good idea why your test failed. And not only that, we got a pretty good idea how to go fix it.

So that's, that's the, that's it in a nutshell is we're, we're gonna analyze your data. We know a lot about your system, we know a lot about the experiments you're running. We're tied into your observability, so we've got a sense for the health of your system and how it's operating.

And then we are going to analyze that. We're going to say, based on what happened, here's the analysis, here's what happened, and then here's what to go do about it. And part of that is being really focused on credible, actionable outcomes.

You know, not just, oh, hey, the system broke. You should go figure that out. But very concise.

Hey, this dependency failed. You don't know. It's a critical dependency.

It either it is a critical dependency or you need to go, you know, make it a non-critical dependency. So that, that's really our, our goal is just take the expertise we've learned and build it into the product. Now, to me, that's a bit of a slippery slope, right?

We talked a little bit before we went live because you, you, there's a line between tools like Gremlin helping the SRE, the engineer, whatever the DevOps, do their job better, do their job faster, do their job, higher quality. Versus in today's world where we have this vision of some autonomous, you know, agent or AI empowered agent actually taking the place of the human, whether it's an SREA DevOps or what have you. Where does, where's your thinking and gremlin's thinking on that?

Yeah, so my, my personal opinion, there's this great quote from IBM I've been throwing around for the last year, and it's, it's, uh, here in, wait, I have it up. 'cause it comes up. A computer can never be held accountable, therefore, a computer must never make a management decision.

So I think this is an accountability problem. If you want to fully automate SRE, the core of what an SRE does is make hard judgment calls. com on the side of I five on my motorcycle.

I had to make judgment calls in the fly that if you get right, everything gets better. And if you get wrong, everything gets worse. So I think, uh, companies and teams willingness to trust AI to make those decisions, I think we're years off of that.

I think we need a lot more comfort there. And so that's, that's kind of my, my split, I think, look, if AI is augmenting you to do something that you're not great at, to do better at it. So if you, if you never have done reliability, if we can help make you 50% better at reliability, that's a net game.

But could we just do it for you? And we've talked about this, the idea, the original idea of Gremlin was we're gonna release autonomous agents into your system to go break stuff and find the failures. And let me tell you what, that didn't market well in the first couple years of the company, if I bring that up to somebody, they, they panic the blood drains from their face.

They're like, oh my gosh, please do not. So, you know, and I, and I agree, you know, it's like, look, if it's safe, if it's, if it's, you know, safety safety's key here. You know, we, we're here, we're about preventing outages.

We never want to go unleash something that might accidentally cause an outage. I, I agree with you a hundred percent, man. Hey Colton, let me turn back to, so obviously in your role as CCTO, you have, I don't wanna say re-engineered, but let's say refreshed the whole tech stack of gremlin here.

And in doing so, I don't wanna say it fundamentally changes sort of gremlin's mission, but let's say it brings Gremlin's mission up to today. Not when, you know, just the thought of chaos engineering, yet people saying, oh, I want to be like Netflix. Right?

And, and so yeah, if it, if it works for Netflix, it works for Mikey, you know, and, and so they were doing it just for that reason. But we need more than that today, right? We, we need to show ROI, we need to show, like, like you mentioned before, insecurity, when nothing happens, you do your, you're doing your job, but it's very hard to go get more budget because nothing happened.

It's, it's very, it's a very similar argument here. Tell us 10 a's the ground lit. Yeah.

I think it all, what Is it about, It's, it's all about that que you know, it's the, it's kind of the classic. Like if the tree falls in the woods and no one hears it, does anything happen? Like, if you prevented an outage and it didn't occur, did anything happen?

And, and again, I've struggled with that. I've, I've seen my champions and, and my peers struggle with that at their companies. They're doing great work.

Mm-hmm. They're, you know, I think there's two categories. Let's be clear.

There's, I see a lot of people that wanted to do chaos engineering. They wanted to be like Netflix. And they went out and they kind of ha you know, they didn't put the full effort into it.

They had one team do it. They tried it on a couple applications. They didn't build a process.

They didn't build any structure. They didn't build any repeatability. And then when it was time, when it was the business said, great, how are you solving reliability for the company?

They didn't have an answer. 'cause they were solving reliability for a pocket of it. I think my best customers, this is one of the things we've learned.

This has to be part of how you build software. And, uh, and it really, it's hard for it to be optional. I think that's the other analogy from security is like, you know what, at the end of the day, security isn't optional and neither is reliability.

And so we need to, I I, I like this term field of dreams. DevOps is what we've had for the last 10 years. If you build it, they will come.

If we build cool mm-hmm. Tooling, the engineers will arrive. Well, I'm here to tell you, I wish that were the case.

And it's not. The engineers show up where their bosses tell 'em to, and they work on what their project managers tell 'em is important. So if it's not important to the company, then the engineer isn't gonna be given the time to work on it.

And so we need to make sure there's time. We need to make sure, and then there needs to be follow up. Hey, the s this is the SREs team problem.

We'll just let them figure it out. No, who's the vp? Who's the C-level that gets called when the, when the service is down, when the website's down, who has to go to the board and report?

We had an eight hour outage that ended up on the front time and the front page of, of the newspaper. Those are the people that need to care. They need to be bought in and they need to see the progress.

And I think in general, for lack of visibility, those people, I'm sure they wanted that visibility, but they didn't have it. And so it was a lot of trust and a lot of flying blind. So a lot of what we built was that visibility.

Let's have a reliability score. Let's define your services in gremlin. Let's track that score over time.

Hey, you started at a 50 per, you know, a score of 50. That's okay, everyone starts somewhere, but where are you a month later, three months later? Are you making progress or are you standing still?

The other part is build it into the company pro the company culture. This is what I loved about Netflix. This is what I loved about Amazon.

You know what, when you told people at Netflix we're gonna do some failure testing, nobody balked. They were like, yep, that's a thing we do here. Sounds good.

We understand why it's a good idea. So getting that cultural buy-in that yes, this is a good use of time and yes, we're gonna invest in this, but, but also educating the business. We're investing in this to save the business time and money outages are expensive.

Yeah. They cost engineering time. They lose revenue.

They impact our brand. If we can just prevent those outages, we've saved the company a whole bunch of time and money. Well, how do we do that?

Do we need to drop everything and spend all of our time on reliability? Nope. We could spend an hour a month.

We could spend an hour a week and we could make substantial gains over the course. So, so a lot of what we built scoring, uh, a set of detected risks, there's a whole set of things we can detect our problems before we ever run a test. So let's give you a low risk way to calculate that.

Um, we built something called dependency discovery. So originally we thought, oh, we really need to know how this, these pieces fit together to test them correctly. And originally we thought, ah, tracing solves this problem.

We'll just integrate with tracing. We don't even have to think about this. Well, as you probably know, tracing is not ubiquitous.

It's, it's, it's a hodgepodge and it's different services. So we couldn't rely on that. So we went and built our own, you know, network, uh, traffic analyzer to understand how these services were communicating.

And that was one of those like, hidden values of, uh, hidden, hidden gems of value. We didn't, you know, we didn't think we, that was a means to an end for us, but a bunch of customers were like, oh my gosh, I didn't know I depended on this thing. That's a huge, that's a huge learning for me.

So again, giving people visibility into the system, helping, Are you shining a light? What's happening into what would, what was dark? Right?

Shining a light on what were kind of dark, just you don't realize they're there until, until something hits the fan. You know? And, and, uh, it's important, you know, Colton, no.

One of the things that we see here, and I, as I mentioned, you know, we're part of future. So their analysts do a lot of research on, like, spending on, in, on the software industry and how people are allocating dollars. You know, they, they, they just came out with a report, uh, uh, software, uh, spending would I think hit 300, almost $350 billion last year.

It might go up to as much as 600 billion in two years. It's crazy. A lot of money.

Yeah. But it's not necessarily net net new dollars that are going towards building, this is the enterprise market, by the way. Mm-hmm.

Building enterprise software. It's dollars that are being reallocated from other things, from people, right. From people spend from other areas within IT and other areas outside of it.

Um, so I, I think what you're saying resonates that, hey, if you're gonna ask people to put their hard earned budget dollars, you know that people lost jobs over into building this software and running this software. This software better work. Yeah.

It better it, it better do what you're saying it does or should do. And we need reliability and we need, we need to be able to put our finger on that reliability right. And point to it and prove reliability.

It it can't be that reverse. Well, nothing happens, so it must be good. Right?

I don't think that works in today's, it's, it's a tighter budget world than we've lived in before. And it's hard to get lucky that long is the truth. No.

Right. Dislike, stick your head in the sand. It did.

That's the truth, is like those, those failures exist. Would you rather know about 'em or not? Mm-hmm.

And that's the key. And so, yeah, but like to your point about, you know, being good stewards of, of our customers resources, you know, I think that's the other thing we saw a lot of people struggle with the chaos engineering tools to really get that value. And yeah, that's a problem for them.

But that's a problem for me. I want them to get value. I want them to see the same success I saw at Netflix that we see at Gremlin.

By the way, we do this at Gremlin. I'll, I'll tell you a fun quick story. 2019, we were chaos engineering experts.

We had the best tooling in the world. We weren't using our own tooling consistently. We were doing it spottily kind of like everyone else.

And one of the things I did when I took over as CTO is I made it part of my on-call handoff. Every engineer gremlin is on call. Every on-call runs all the reliability tests that week, or the ones that are scheduled, they follow up on 'em and they fix them every on-call handoff.

We look at our scores and if the score goes down, the team knows I'm gonna ask 'em what happened. So they're already on top of it. They're investigating it, they're paying attention to it.

The side effect of that is we very, very rarely have outages. We're in the five nines plus territory in Gremlin over our lifetime. And it's been super solid for the last three years because we've gotten so much better at this.

And we feel comfortable. Our all, you know, our, our team sleeps well at night because they know if something bad, you know, most of the dumb stuff isn't gonna bring us down. And if something really bad happens, we're gonna know about it.

We're gonna be able to deal with it. 'cause everyone's had some, some reps at Bath, some practice. So of course the question is, are you drinking your own champagne or eating your own dog food?

I love that. Jeff Bezos square. I was at that Amazon all hands where Jeff gave us a, who are you, uh, uh, champagne bottle.

Uh, I keep my magic cards in that champagne bottle actu in that champagne bottle case. But yeah. Good for you.

No, I think we're drinking our own champagne and, and I feel good about it. And it's like the new reliability intelligence stuff. We just launched my principal engineer for a month has been coming to me.

Colton, guess what I found? Hey, guess, you know, this dumb thing that was happening. We found the problem because reliability intelligence zeroed in on it and told us what was wrong.

That's what gives me, you know, a lot of confidence to go out to market. When my, when my team of really sharp engineers is getting, getting educated, finding value, then I know, you know, the vast majority of people are gonna find value. I love it.

Hey Colton, we're over time. I gotta I gotta wrap up. I apologize we didn't mention.

People wanna find out more about this reliability intelligence from Gremlin. What's the website? com That hasn't changed?

Hey, Colton, welcome back. It's a pleasure to have you back here. I expect to see you on here regularly now, right?

I'd love to. And, uh, we'll keep this conversation flowing. It's a pleasure.

Thanks for having me, Alan. Alright. Colton Andrews, CEO founder, CEO again, and founder of Gremlin here on Techstrong tv.

We're gonna take a break. We'll be back. Well, software development, chaos.

I think it's not much of a secret. We know it exists. And question is, is it gonna get any better or maybe worse?

Shannon, welcome to the show. Thanks Mike. Glad to be here.

And, uh, glad to talk on one of my favorite, not so favorite subjects. I think that might be true for everybody involved in software development. So let's just jump into it.

'cause it's one of those dirty little secrets about how we build software. It's complex, it's chaos, and sometimes it's more art and luck than it is science. But anyway, let's jump in here and say in your mind, what is the current state of software development and what kind of analysis?

Um, wow, you, you, you made me think of. Uh, it's better to be, uh, good than, uh, and lucky than, uh, than great, right? Um, but, uh, one of the things that I think we're really faced with right now, I, I don't think we can get through.

Mike. You probably haven't had any conversations without talking about ai. Um, so for me it's this combination of, um, a lot of really wonderful great engineers trying to figure out how do I incorporate this into my daily life?

These massive pushes from, um, executive leadership, which oftentimes we might talk a lot about execs today because a lot of chaos usually starts to originate up, up in that area in those ways. If it's not the market, it's usually some sort of internal chaos coming in and saying, Hey, we need to start investing in this area. We need to start, you know, improving productivity, improving speed.

We need to start doing these things. And we're starting to see, I think some of the results or some of the data now that there's been enough information, um, enough utilization, and it's showing us that, uh, it's actually increasing a little bit of chaos if you're not thinking about it smartly. So I think a lot of engineers right now are, um, both struggling with the continuous requirement to innovate, develop, maintain, and also the speed of which new technologies are coming in, which has been ever faster.

And then you have this added component of ai. So, um, the landscape, at least as I've seen in the last, especially the last year, has changed dramatically. Um, over at Tempo.

We're trying to look at even studies and how those studies relate to our own in internal transformation efforts. So like Cornell did a long-term study for this year. I think long-term is probably a little bit too, um, too gracious.

But they've been looking at and published a study just looking at whether or not the productivity speeds and gains that are expected from incorporating AI into your engineering organization. Are there. Um, short, long story short, uh, not so much when it's a complex code base and the more, uh, mature your engineers are, it actually detracts from their productivity.

It takes time away. Um, which I don't think if you spoke with any dev, they'd say it's anything differently. Um, so new code bases works great.

Um, younger engineers works wonderful or new, and career engineers works wonderful. Um, but the more mature and the larger and more complex or code bases, the more potentially detrimental it can be, especially from a security standpoint. And then yesterday there was even the report coming out, MIT from Nanda around, um, 95% of transformation efforts that are associated to AI and engineering are not seeing the success rates that they're looking for right now.

So we've got this like big, you know, push that's coming into organizations, but still software engineers need to build beautiful products, wonderful products. And so there's this kind of clash that's occurring, um, that's really I think a new, a new horizon for us, uh, in development. I also hear that folks are saying that the code generated by the AI platforms is, shall we say, verbose.

And when they go in and try to fix it and debug it, well they don't know how it was constructed in the first place. So they're kind of scratching their heads and saying, I might have been better off if I just wrote this thing from scratch myself. Anyway.

Yeah, and I think that that kind of connects back to what we're seeing in terms of studies. I mean, I love that there are people out there who spend their days researching this, you know, not just us using it. Um, it's one of the things that I love, you know, having a background in psychology, that there are people out there who spend all their days looking at organizational psychology and how orgs do planning and things like that.

But you're exactly right, right? Um, uh, there was a great, uh, webcast a few months back, uh, from Anthropic, you know, it's Claude. And, um, and they said, you know, you gotta think about this as having like the world's smartest intern sitting next to you.

But they need to be bounded and they need to be guided. And I think a lot of us are still in the place where we think of, uh, you know, adding maybe agentic development or even age agentic networks, and that those things are just going to be continuously self-learning, continuously evolving, but they're not quite there yet. Um, and they can run a little bit rampant because they're just smart enough.

And so they'll go through and make changes. So I think to your earlier question, right, like that, that can add a lot of work. 'cause then you're double checking.

You wanna make sure that you're not potentially inserting code that exposes you. Um, I know that GI had, GitHub had a little bit of a, an issue around that. Um, and those sorts of things I think are really changing the dynamics for engineers where, you know, 10 years ago we were just thinking about how fast can we deploy to production and how fast can we make sure that we're running through all the checks and balances predeep production.

And now it's like we've got all this other additional added complexity. And then of course, the inherent marketplace speed factor that's coming in from your leadership, from, you know, other product companies saying like, you gotta move fast, you gotta move fast. Oh my gosh, everything's gonna go move so fast.

So, uh, it's definitely a wild ride right now. So under the heading of, I'm not sure whether I should laugh or cry, but, um, we are talking about citizen developers giving them vibe coding tools, and they're gonna have AI and they're gonna create software, and it's gonna be rapidly iterated in the best of all possible worlds. Some professional developer might help them vet that thing and craft it.

And we will build more software than ever. Now in my low-code, no-code experience. Most of the applications that are built are well kind of ugly.

They're not very secure, and they don't ever scale. So how are we gonna do this with, you know, when we change the definition of what an application developer is, and all this stuff is supposed to come down into the DevOps pipelines and engineers who are already overwhelmed and maybe hopefully have a bunch of AI agents to help them manage this. Or has this in your mind, can this work?

Yeah, I, I absolutely think so. I think, um, what we're seeing right now is obviously a lot of early adoption and, and we'll go through the normal hype cycle with that, right? There'll be the trough of disillusionment.

I think it will happen more, more quickly than we're we're used to, just because the speed at which these things can change. Like, uh, that Cornell study that I mentioned that was from the beginning of the year, those models have already improved since then and you get different outputs. Um, but really for me, the benefit here and, um, you know, being so deeply involved in software development, being married to an engineer, these sorts of things, they, uh, they do start to show productivity gains if you're training your team on how to leverage them effectively, right?

So, um, for example, none of us is going to go use lovable to build an enterprise scale application, but it's a great tool to figure out whether or not something is or is not feasible to kind of play around with ideas to get really early feedback around potentially a net new concept with a customer. Uh, but you're not gonna take that code and then immediately transform it into a publicly facing, you know, site and application and like launch monetization around it. You're gonna use that as kind of like that early analysis and that early validation, which I think is amazing because what that means is that we can save ourselves a lot of those additional steps that can then happen further down the line.

'cause a lot of the rework tends to be, oh, hey, we didn't confirm with the customer whether or not they liked this, or we tried this particular path and it turned out that it was a little bit wonky to go through this interaction. Or the math looks weird, or how we can convey this sort of visualization is a little bit strange. So for me, the benefit is like if you are treating your agents like an intern, a very smart intern that's sitting alongside you, that you can kind of offload tedium to, that's great.

Um, and I think we should even start, you know, even more simplistically, which is like going through and pulling in information about how progress against all of the work looks like, right? We don't need to necessarily use this to do big giant development. We can offload the tasks that come in that are around like status updates and sharing information about progression with the folks that wanna know that information so that we can allow our teams to spend more time in the flow state and actually doing work.

So I think like, you know, there's so many little easy things that we can knock off that actually increase the joy for an engineer by leveraging, you know, models, large language language models, a lot of the intelligence that's coming out. But at the end of the day, um, we still kind of go back and I still kind of go back to the mythical man month, right? There's only so many hours in the day and we need to eat, we need to take time.

We need to find a place where we can actually focus our brains. Um, and unless we're using and thinking about all of the tooling from that standpoint, and that's AI and everything, it becomes a distraction. And every distraction, uh, takes us away from the core goal, which is to craft something amazing.

To your point about that there's so many hours in the day and our brains have some limitations. Theoretically, we're gonna exponentially increase the number of application development projects we can simultaneously manage. Well, that sounds great, but in practice, I'm not sure that teams can do that and they can manage that many projects simultaneously because there's only so much room in their brains.

Yep. Yep. And this hits on my background, of course, which is in psychology and neuroscience and the cost of context switching is already really well known in our current landscape.

And we've seen that really increase just based off of the speed at which we can deliver. I mean, I remember, uh, shipping software on disks way back in the day, right? To our customers, uh, not at Tempo in a former live.

Um, and then, you know, and going through deployments that, you know, were once a month and then all of a sudden it just became faster and faster and faster, and our code bases became increasingly more complex, but also had a lot of checks and balances in them to make sure that, you know, we're delivering things that are of quality and are secure. Um, and you start adding to that, and you're right, eventually our ability to kind of just keep track of all of those things, uh, reaches a limit, reaches a cognitive limit. And the speed and pace at which we've seen, you know, over the 18, 18 hundreds, 19 hundreds and now the two thousands, um, the speed at which technological change, there's some really interesting studies about the psychological ramifications of that, right?

There's only so much intake that we can do around change before our brains start to be like, there's too much happening, and then we actually default to being a little bit lazy, right? It's just occurring. We just kind of go with it because there's so much we can't keep up.

Um, and I think we see that in like a little bit of a microcosm in our organizations where, you know, you'll just start to see almost, uh, just, just exhaustion and exhaustion for folks that are actually, and I think of, um, developers and, and, and all of our folks that are actually building the magic as hand crafters, they end up getting exhausted. You know, there's just brain fatigue that occurs. And when we're fatigued, one of the things that happens is then we start to ear towards bad decisions.

So like, I've been reading a lot about, you know, like the, you know, 14 hour work cycles, you know, 20 hour work cycles, like this big push for like the, you know, seven day work week, or six, six hour nine hour work weeks, or six days, nine hours or something like that. Something crazy I was reading about a couple weeks ago, and I was like, that's a really great recipe for very bad decisions to occur as the brain becomes to more and more tired. So I think that's where you also see, not to tie it into our earlier discussion about AI agents, is that then people start to get lazy and they start accepting things because they're exhausted.

There's too many things to keep track of. Um, and then I think we'll probably feel a little bit of that just overabundance of applications because no code, low codes, citizen engineers, you know, I can come up with any sort of idea and I'll produce it. Um, but I also wanna look at the flip side of that, which is, it's an amazing democratization of ability.

So folks that, you know, might not have had access to the best schools, the best opportunities perhaps, but have really great ideas who might have been locked into jobs because they need healthcare and, you know, maybe they don't come from generational wealth and their parents are gonna give them seed money to start doing something. They have that ability now to actually build out things and, and concepts and test them. So I think there's, uh, there's two sides of this.

Like we'll see a lot, um, you know, we hope that the markets balance themselves. Um, but I think, you know, what we'll see is, is more opportunity for folks that, um, might not have all the access that they're, that they're, um, historically we've been used to. But yeah, I do worry about just like so much information coming in.

Um, and I also worry about just the general idea of learning. Like learning is fun. Um, and sometimes we can, I think, be a little bit lazier.

We're overwhelmed. So you have a background in psychology. Are all these things related?

I mean, people, again, they get frustrated and then they get angry and then they head off to the pub and they make all kinds of series of bad decisions that have nothing to do with it. But it all is somewhat related. I I hope it's not, it's not that dire.

Um, but it does happen, right? Um, and then, you know, the younger we are, the less fully developed our prefrontal cortex, uh, is in general. And, um, and so the worse the decisions that we make tend to be.

Um, but I, I, you know, it's, there's a lot of input and there's a limit to how much input we can manage as humans. Um, and then you add on top of that just, you know, living in cultures that require a lot of time outside of work. You have kids, you have family, you have a life, you have all these other things that are going on.

Maybe, you know, things are going on in your life that require your attention. Um, and so I'll, I'll kind of come back to thinking about, um, even in our organizations, if we're dealing with chaos or it feels like chaos, where can we offload the TDM to something that is intelligent and then taking a step back and adding focus, right? There's only so many things that a team can manage.

I think the, you know, like the Poppen D***s showed a few years back, and this data continues to, to kind of maintain its stability over time. Like the, regardless of the organization size, like any, any team, any team of size two can only handle about two things or two projects, um, two ongoing activities before they start to get challenged. Um, so then we start to get this incremental decrease in productivity and this really big existential cost associated to it that can be tracked realistically.

Um, which is, I think actually a great opportunity to bring your CFOs and your CEOs into the conversation is that like you do pay an opportunity cost by overburdening your teams with tedium by overburdening 'em with too much work, with overburdening them, with too much context shifting associated to too many projects. And so the adage is true, like, do one thing do really well, move on to the next thing is usually faster than doing five things simultaneously. Go figure.

So there is a pet peeve that everybody has that you might be able to help with a little bit. But one of the things that occurs is we do have too many projects and we're have all these balls in the air and we're trying to juggle all that stuff. And somebody sends over an email, says, you know, Hey buddy, can you update that project management spreadsheet?

And everybody's like, oh, I gotta go enter data. I gotta look all this stuff up. Is there some way to automate that whole thing?

'cause if you talk to a software engineer, they kinda look at those project management people and they say, all the data's already in the tools I have. Can't you just see that yourself? Right?

And that's, um, I mean, it's near and dear to my heart too, at Tempo, we got our start in time tracking, uh, which tends to be the bane of an engineer's existence, which is why we spent a lot of time figuring out ways to both automate. Last year we actually launched an intelligent way to track, um, and add in like just the, the knowing where people are spending time and effort based off of their source code management systems based off of how they're leveraging their ides. All, like, all this stuff can kind of be observed and shared without having to ping somebody.

Um, although, you know, with Slack, with all with Microsoft Teams, people love Ping and people. Um, so I think, you know, to your point, those are the sorts of things that we should definitely look at automating. Um, now the hard part then becomes all of the hidden work that people might be hiding, right?

Like, um, you mentioned you get pings, you get an email that says like, Hey, can you, can you do this? A lot of time that stuff doesn't get tracked. So for me, one of the biggest things that we can do, um, as an organization, especially as an engineering organization, product development organization, is make work visible.

Even if it's just a quick, like, Hey, this is happening. A lot of the systems that are out there, it's pretty easy to generate a little bit of a record. That effort is going somewhere.

'cause if we have that data, then we can start looking at the economic costs associated to adding in all of this additional work or all of that hidden work that we're may not be accounting for. If you're working in organization, we're making work visible is something that's frowned upon, it's usually a pretty good sign that the culture's not so great. Uh, and you're probably gonna get burnt out eventually anyways in that particular scenario.

So, um, but those are the sorts of things that, that kind of have stuck with me over my time. Um, in tech is just like, we have to make things visible. It's really hard to shift things if we don't have data.

There's great systems out there. I mean, tempo included, right? That will track these things, that will understand these things with as little friction as possible for the, the folks that are building the beautifulness.

Um, and that's on purpose, right? Because we wanna stay in that flow state. We wanna stay focused.

We wanna allow people to stay focused. However, if we don't know that these things are happening becomes increasingly difficult to, um, to make sure that we can change them. Mm-hmm.

What's your best advice to people? And I'm gonna ask this one little caveat in the middle of that, because, you know, sometimes, you know, I'll talk to people and I'll say, well, what we're gonna do is, you know, require everybody to have some downtime and they get more aggravated. 'cause now they're forced to have downtime.

Yeah. Right? Um, you know what, one of the, one of the, I think the, the nicest things that we can do is, um, actually declare a no meeting day.

Simple, simple technique, simple activity. Just a day that is about focus. It's a full focus day, whatever you wanna call it.

Um, build some kind of pomp and circumstance. This is something that also needs to be followed by your leadership too, right? It's one of those things where it's just like your leadership says like, Hey, we love vacations here, but then they're always responding to emails when they're on vacation.

Then we're setting the example that we don't really love vacations here. Um, so the other thing that I think an organization can take as like a really easy step is just a clearing one day, the day where we don't do meetings, we limit meetings. Now of course, stuff happens.

Um, and certain functions like support are always going to have maybe conversations that are occurring inside that might require them to hop on a quick customer call. But these are work related things that actually move the mission forward. So one of the things that we wanna wanna really focus on is whether or not we can create those really focused days.

Um, if you're interested in kind of pulling this into your organization, there's tons of, uh, like ar articles out there about creating a no meeting day. And then I think the other beautiful thing is if you have somebody in your organization who looks at mihi sheets, MI high's, work on flow state, um, because that's really what we're trying to create here is a day where people get to focus, um, where they get to research, uh, where they get to look into problems and challenges. I think the other things that are still kind of part and parcel of eng life right now, right?

Are also creating hackathon space as well too, right? Space for innovation space to think. Um, and then I'm also a big proponent of putting Slack naturally into any sort of plan or any sort of understanding of our organizational velocity that we have.

Like no one should be planning to a hundred percent of what they know they can do. Um, it's always better. I have never worked at an organization or worked with a company when I was doing consulting where if we under committed and then overdelivered, someone was upset.

But I've worked in plenty of places where if we overcommitted and underdelivered people got upset. So I think, you know, to yourself a favor plan to at least 80%, maybe less if you're doing interesting or new things, or you have new people coming into your organization. But Mike, to your point, that's where some of the tools can come in handy because they can tell us data around when we have a new person join, maybe an established team.

Our team velocity is expected to go down. And this is just facts. We have to train, we have to help, we have to assist.

Um, if we have new work that's coming in that's new for the team, those are the sorts of things where we have to assume that our velocity would decrease. And so it's just, I think taking a humanistic view. Um, and these are the things too that I hope that, uh, you know, intelligence inside of our organizations via, you know, bots, AI agents, agent networks, will help really surface.

It's one of the areas that we're looking at, at Tempo is just kind of looking at the overarching plan and how those things change based off of how the effort is changing on our individual teams. All right, Shannon, thanks for the insights, but I want to clarify one point real quick. Um, no meeting days.

Is that like once a week or every other year? How often? Once a week, Mike, if you could believe it.

And I think the, uh, the gains from that would, would definitely pay for themselves. So, uh, definitely worth the worth. The, uh, time worth, the effort, worth the investment.

And, uh, a quick, easy step that I recommend folks take. All right, folks, you heard it here. One thing to think about maybe is find that list of 10 things that annoys you and find a way to automate it, and life will be a lot better.

Shannon, thanks for being on the show. Thanks, Mike. Have a good one.

All right. And back to you guys in the studio. Hey everyone, we're back here.

We're on the floor of Black Hat, though you really can't tell. 'cause we, we went with a black screen here, but I couldn't think of two better people I'd like to introduce you to, if you read Security Boulevard, if you've been in the security world for more than a minute, you probably know both of these guys, though you may not know them, you know them. Let me introduce you to two friends.

I know them both 20, 25 years to my immediate right, Robert Hansen. A lot of you may know him as our snake though. Look, as we get more gray or less hair, we, we go by real name.

So it's Robert Hansen to my far right is a really good friend. I, I've known him for a really long 25 years too. He could tell you about his history.

It's my friend Jeremiah Grossman. Jeremiah, Robert, thanks for joining us on Techstrong tv. Always a pleasure.

So guys, you know, you're like cyber royalty to me, cyber security royalty to me. But you guys made a big announcement about a week before, a couple days before the, uh, event this year. Why don't we dive right into that and then we could come back and talk about what's up in your lives and everything else?

Sure. Um, Robert and I, we've been running companies together for a long, long time. What we care about most of the industry is keeping people safe, keeping people from getting hacked.

And the way we do that is we try to find the world's most important cybersecurity problem, the one that has to be solved. We learn everything we can about it. Once we find a solution, then we start a company and we go after it.

So we don't start with a technology looking for a problem, we find the problem and we go after it for as long as hard as it takes. Absolutely. And I mean, just for people who aren't familiar, companies you've done together.

So look, the first, I think I met you, you were still at Yahoo. Yeah. You had, or maybe just leaving Yahoo, you did White hat.

Yeah. I, I started my career, uh, 25 years ago at Yahoo. I was one of those kids that hacked Yahoo Mail and they gave me a job instead of calling the FBI.

Yeah, Lucky For you. I took what I learned there and I learned that web security was a problem. Mm-hmm.

And I wanted to solve it. So we created Wide Hat Security to, uh, scale and mechanized application security and vulnerability assessments. And of course, you had to get the other best in the world AppSec guy out there.

And that was Robert. Absolutely. Right.

And, and look, white hat kind of invented, uh, you know, uh, pen test or AppSec testing as a service almost, if you will. Um, but then that was one company. What came next.

Uh, the next one was, uh, when, when, uh, we left a, let's say a white hat. I, I did a short time at Sentel one in the early days to focus on ransomware, which wasn't a thing yet, and, uh, go after, uh, endpoint. But, uh, that was two years.

In the meantime, while we were working on something for attack Surface Management, what we were learning was a significant number of the breaches were having to do it a previously unknown asset, that they would've secured it if they know it, they owned it. And so I'll have to have to pass it to Robert here, but I said, Robert, the we only where we're gonna solve the attack surface management problem is to download a copy of the internet first. And, uh, so Robert, true to what he is, he goes, okay.

Yeah, Yeah, I think, I think, uh, people when they hear that, they're like, that can't be possible. But we were processing by the end, like multi petabytes a month. And I remember, and when people think of the word internet, they're thinking Google, that's just the web, that's a searchable web.

What we really needed was a copy of every piece of metadata, every Ip, everything, Everything, every ip, telephone and printer and whatever. So that was a pretty big challenge, but it enabled us to answer the question, what do people own? And uh, so that kind of took us down this path.

I, I remember you working on that. You know, one of the before I did still secure my friend Raj, who's one of the co-founders with me, is still secure. He started a company Kova, which was IP Geolocation, very similar thing.

You had a geolocate every IP address. So figure out, you know, which IP address went where. It's a huge undertaking.

It was simpler then, 'cause it was smaller then it was bigger by the, a lot bigger by the time you did it. And of course, that led to another company, and I'm blanking on the name right now. Oh, that are most recent one.

Uh, so, uh, so Bit Discovery, the attack source management company. So we raised money and, uh, during the pandemic, and uh, the company lasted a whopping three years. So it was a very fast moving company.

It was very successful, very fast. And it was acquired by Tenable, right? So, uh, you know, so Robert and I were, we have a background in application vulnerability management, but not, uh, so network or CDE vulnerability management was, uh, familiar to us.

And so what we learned there was that this is a very big 25-year-old problem. Everybody has a vulnerability management problem. They were sick of it.

Everybody was buried in v didn't know what to fix first, and everybody was still getting hacked. So we're like, okay, we gotta set out and solve this problem. So Robert and I have TA taken hundreds of meetings to learn everything we can from everybody we could about this problem, to figure out what the problem was before we could solve it.

So two years later, we were ready to launch the company 'cause we think we had some answers. Mm-hmm. And that company is, That's rude evidence.

Uh, also known as just evidence. We go by that as well. But I think one of the cool things is, um, we pulled in tons and tons and tons of data.

We, instead of like, most people are just like anecdotal evidence only, but we pulled in information from every source we could possibly find just to see if there's anybody who agreed. And it turns out no one agrees. And that was sort of the premise that got us thinking down this path is like, well, if everyone is disagreeing, that probably means that everyone is, at least everybody, but one is wrong, but probably everybody's wrong.

And so we gotta think of an entirely new tack. Like how do we, how do we tackle that? Fortunately we've been talking to the insurance industry for years and years and years.

So we had really, really good relationships with them and they started sharing with us some, some details about claims. And it turns out you don't need to look for 300,000 CVEs. It's a much, much more finite number.

Much easier to do. Absolutely. So I have a little experience in this, right?

I started a vulnerability management product that's still secure. In 2003, a lesson I learned the business, if there was a really good solution, that'd be one, maybe three. There's a reason why there's dozens of v vulnerability solutions.

And I would say, Robert, it's not that one is right or one is wrong, they're all a little wrong and a little right? That's right. Everybody, right?

Everybody has kind of nippled on the edges here, but no one, no one's really solved it. So, And, and enterprises feel that, so, oh, absolutely. So the one observation that, uh, you can reference this, um, only 1% of all known vulnerabilities have ever been exploited.

And that has been the case for quite some time. So if the prioritization models are working, why is it always 1% of vulnerabilities? So that's something to uh, something to contend with.

So the concept that we're bringing forward is a evidence-based vulnerability management. And the way to describe it is, any given vulnerability could have evidence of exploitability, it could have evidence of attacker activity and evidence of attacker breach and loss, financial loss. If you have all those three, those are the ones you picked First, Right?

If you don't have breach and loss data, you have effectively a Kev list vulnerability, which is fine, but that's the next down the list. You remove evidence of, uh, attacker activity, then you get into prediction and prioritization. So what we wanna do is concern ourselves with the ones you absolutely must fix.

Now, no ratings, no scoring, no color codings, you fix these and if you do that, you're better than 99% of everybody and you're not going to get hacked. You know, I'm reminded, I don't know, do you guys know Giddy, giddy Cohen? Uh, he sold the company, but it's not 20 years old.

Giddy was the first one I saw who generated attack maps of vulnerabilities. So it would find a vulnerability and then work backwards from there out to the internet to see is it reachable? Is it exploitable, is it fixable?

And how is it fixable? Is it patchable? Can you close a port down?

You know, what's the remediation path? And darn, I can't remember the name of his company, but Giddy Co was the founder. And I thought that was one of the best things I've ever seen in, in how do we tackle this?

Now we didn't have ai, we didn't have the, the reams of data that you guys had, but it was a, it was a holistic approach to saying, let's get out of the hamster wheel of just giving you a phone book of CVEs in South Sea next year, right? Which is was the kind of state of the art when I was doing this in 2003, right? Well imagine, imagine that's what you get, right?

You have like 50, a hundred thousand vulnerabilities. Some of these companies have millions of vulnerabilities. You go fix, fix a bunch of vulnerabilities the next day you have more vulnerabilities, you're not actually really moving the needle at all.

And we, we spent a lot of time thinking about like, like what, what if you have 10 vulnerabilities equal chance of a bad thing happening to each one of 'em, and you have a million dollars in the pot in the middle, any one of 'em gets you there. If you only fix nine, but you leave the last one there, like you've actually just wasted time. You Probably shouldn't have fixed, you Probably should.

I'm shouldn't have fixed any of 'em, which is a weird concept. And I think security's gonna have a really tough time like really thinking through that. And of course there's a lot of variables there, but, but I think one of the cool things about looking at the insurance industry is we're like, here is loss.

We are actively seeing loss in these places. Why don't we fix these first? Right?

And, and the nice part about that is now we're talking dollars and cents. We're no longer talking about, you know, some prediction model that, I mean, and no offense to those guys 'cause I think that is really very tricky and interesting. It's just that it's very hard to predict when there's only a handful of loans.

And depending on who you talk to and we have reasons to believe some of these numbers, it's definitely less than a thousand, let's call it that. So we're talking a fraction of a percent. So if you're gonna make a prediction, you have better be perfect if you're gonna get exactly those vulnerabilities.

Absolutely. Look, no one manages risk better than the insurance industry. That's what they do.

And I've never met a poor insurance company, right? Um, so I think that's a great model. My my question though to you guys is does it wind up being like the OAS pop 20 where it's like a static list and, and right, this is a list that needs to be constantly cared for in guarded?

That's, uh, that's a great question. Um, to lead into that, what we learned, uh, what we've learning is about 50% of the breaches that lead to loss have something to do with a remote exploitable CVE. So if we wipe those out, the ones that Robert was mentioning, we can reduce 50% of the losses.

That's huge. That's the impact that, that we wanna have. Does that list a thousand vulnerabilities?

How does that get updated? So right now, it, it's generally speaking a static list. That's why we know the vul management industry is getting it wrong.

Because if it's only 1% of VULs, that means the adversary is not forced to innovate to take on the next one. So if we get the prioritization right, we should expect the list to change over time, right? And that's our gonna be our bellwether if we're doing it right.

So if we see that list starting to That's right, that's a good thing. So, so if the list changes, we're doing it right and that's what our intent is, increasing Costs. Yep.

Let me ask you another question. So there's finding vulnerabilities and there's fixing vulnerabilities. I get what you're doing to help find the, the right vulnerabilities, let's call it that.

What, what is evidence hub to fix those vulnerabilities? So ultimately that's not our job, that's the customer's job. Uh, but we can make it easier.

Uh, and I think the major way we make it easier is we help them make their own business case to their own executive team about why they should prioritize both by reducing the amount of, you know, chaff a bunch of vulnerabilities. It'll never be exploited, have never been exploited by anyone. Now if it's a much small, a much more definitive list with known attribution, um, to claims data and we know what the claims losses are, now it's just a matter of how much, what kind of cost it is.

So if it's like a million dollars to fix a vulnerability, that'll cost you 5 million if you don't fix it, well that's a $4 million ROI That's, that is a very easy business case to make to your CFO who make no mistake, that is the real risk officer of the company. Not, not the, Not the CISO or any Of those people. Exactly.

Exactly. So I think that's really where our main focus is at. Now, we could always pivot more into that area later, but just by starting talking dollars and cents, I think that's a big win for the customer.

Absolutely. I, I, you know, another, another piece of this though is I used to call job security, right? The vulnerability, the guy who's responsible or gal, whatever the person responsible for vulnerabilities, vulnerability management.

The team, they've gotta be incented to hit the right stuff. You know, you know what they say, right? If nothing happens, we did our job.

That's not a hundred percent true, to tell you the truth. Nothing happens 'cause it didn't happen yet. That doesn't mean you're doing your job.

How do you, how do you help that work or show metrics that, hey, I am doing my job and we're doing a damn good job with evidence. So that's a fantastic question when we contend with all, all the time. 'cause if we're gonna reduce the set of vulnerabilities that matter, then we should get to VUL zero really, really quick.

Yep. So what we, what we want to be able to do right now, when, when companies get hacked, the standard PR answer is, the attacker was sophisticated. Please don't sue us.

We did everything we possible It was, is zero day. Of course. Where we wanna move people to is if somebody gets breached, it will only be by a vulnerability that no one has ever exploited, ever.

That's a defensible position. What more could they have done? They fixed every VUL that has ever gotten anybody breached.

Right? That's pretty good. That's better than that.

It's the zero day argument. Correct. I'm not even a zero day.

It just, no one exploited that one for whatever reason, zero day or otherwise, It happens. Now, let me just business model a little bit. Uh, back in the day, we used to sell it by how many hosts we were scanning.

'cause it was scanning, right? How many hosts we were scanning, how many ips, how many nodes, how many vulnerabilities? I how do you want to, you know, skin the cat today?

How, how is this packaged? Uh, Uh, the business model will be software as a service. You should be able to go to a website, putting your company in.

If it's scan, it's the straightforward thing around. And what we want to be able to do is we don't want to wow the customer with, look how many giant plates of red you have and all this red. No, no, no, no.

We want to help them in terms of dollars and cents. This is what you need to fix. This is what it's gonna cost to fix, and this is how much money you'll retire.

You're retire risk. We want to get to VUL zero, at least for the VULs that matter. That's the best anyone could ask for in this world.

Hmm. So you don't, I know this sounds old fashioned, no agents, no internal sensors, pizza boxes or any Of that stuff. We only need as much as the adversary does.

Right. Which is effectively not the hack side View. Yes.

Right. Again, our background is breaking into things. That's where we came from.

So we want as little as possible, we want to see what the actual risk is. Let's talk a little bit rollout availability. Robert, is it, can people go on right now?

Uh, no. Uh, we, uh, we, we literally just started fundra. We've got our fundraise, whatever it was two weeks ago now.

So, uh, it'll probably be a few months before we're ready for design partners to start really like actually using it. Uh, it'll probably take another six months, I'd guess before a fully rolled out UI is, you know, freely available to anybody. Uh, it depends a little bit on what we, feedback we get from the customers.

Um, 'cause one thing Jira and I really spend a lot of time doing is absolutely making sure our customers are just, this has solved the problem. If it doesn't, like, we have to go back and fix it. So that's the real question mark, is what rejiggering do we need to do to make it, you know, one of the things we really wanna focus on is speed, for instance.

Uh, like being really, really, really fast. Well, if it turns out that causes problems, you know, we're gonna have to do workarounds, you know, for whatever reason. So that's a, it's an unknown until we get there.

But, uh, the good news is we do have a lot of experience building these kinds of things, so, sure. Uh, so, uh, for those that are, that are interested, so, uh, we have a really good idea of what needs to be built, where we're gonna need help from the industry. You know, practitioners, home management teams, for the people we've been meeting with the last two years is what does it look like?

What do you need it to look like? We know what needs to be done, what do you need it to look like? So for those that are interested, reach out.

We want to hear from you. We don't have all the answers. We'll into that camera.

Where do they reach out? Oh, reach out. Um, root evidence, uh, dot com.

Sign up for, uh, you know, the mailing list. And, uh, we'll reach out. We will, we will meet with you.

We want to learn from you. We want to solve this problem. We're not ever gonna have all the answers, but we'll build nonstop until we solve it.

And with your track record, you got a good chance that's happening sooner than later. Guys, I can't wish you enough luck, success and, and you know, strong tailwinds as as you go forward here. If you don't mind, I'd like to just pivot a little bit.

Let's talk black hat. Yeah. So I first met you in Black Hat, I think in 2005.

I got hacked on my iPhone three, I remember. Yep. And it was, I, I forgot the dude's name.

I think he's still in jail, not for hacking me, of course. But he was an idiot. Anyway, but, and Jeremiah, I probably met you around the same time, but you started, I knew you more for black, half fifth, the Jiujitsu stuff with Hoffa and everything, right?

Dad had to start also around 2005, 2007 maybe, something like that. My, uh, my first blackhead was, uh, 2001. Yep.

Well, mine, mine was 2003. I gotcha. Yep.

We used to be at Caesar's in the hallway. Yeah. I had a booth overlooking the, uh, the Venus pool.

And if you would take a briefing from my guys, I'd let you use the, uh, binoculars. That's how long ago and wrong that was. But anyway, black hats changed over the years.

It's, in many ways it's not what it was, but it's something better different than what it was. You guys are both intimately involved in the whole week's worth of activities. Give give the share with the audience, if you wouldn't mind a little background on this.

Yeah. Uh, I was probably, if memory serves, I think I might've been one of the very first board members for the main conference. Uh, so helping select talks and make sure that they're of the quality that we want.

Um, but gradually, Jeremiah and I, I think we, he was also on that with me. I think we decided it was better to spend more of our time on the CISO summit. Uh, it is just really important to make sure that the CISOs are getting the kinds of information they need to get.

Um, and so fortunately there's a lot of people backfilled and did a great job, and that's why the main conference has done so well. But our focus has really been more on the CISO event. Uh, the Cyber Insurance Summit.

Uh, it's a micro summit and the Innovation and Investor Summit, uh, which is all these sort of micro summits that kind of float around the, the periphery of the con con, uh, conference, but are really important to sort of pushing the, the needle. Well, I, I think that's the model whether you go to the cloud native like CubeCon Summit or RSA or Black Hat. It's these satellite conferences or what I call them, or conference within the conference micros, that really, you really get a lot.

If that's your thing, you, it's a deep dive, a deeper dive than you're going to get going to a keynote or walking the floor and getting, you know, distracted by lights and buzzers. So it, it's great that you do that. Give us kind of a metric.

So for instance, Jeremiah, the CISO Summit. How many people are in there? Uh, so the CISO summit is fantastic 'cause uh, we like talking to ciso see what top of, what's top of mind for them.

And, uh, so the CISO summit, uh, this year was 400 CISOs all in the same room. So the way we do the CISO summit is a little bit different than the briefings. The briefings take submissions and then the review board picks the best of the best.

And, uh, you know, Robert and I have both given many talks there. They do a fantastic job. The CISO summit's different.

The CISOs have an agenda, they know exactly what they want to learn. So we get about 10 topics down to things that they wanna learn. And then the review board sources the world's leading experts in those topics.

And we invite fight them in and just let them loose to, to speak their message and what they know. So the content is, uh, super high quality. We have, uh, generals and, you know, all the people that are front line of the, uh, uh, the breach, uh, uh, salt typhoon breach and things like that.

Right. Things you can't get anywhere else. Absolutely.

I I had friends at the Investors and Innovators Summit. They said it was Greg Robert. Oh yeah.

People who are home maybe didn't see it, don't know about it. Yeah, it's, it's a brand new as of last year, uh, event. Uh, so this is the second time we've done it.

And I think we learned a lot of lessons last year and it was really good. So the content is basically a mix of people who are, you know, entrepreneurs getting into the industry. Maybe they have a company, but they haven't quite figured out how to take money or haven't figured out how to talk to customers, or they're sort of in that growth phase and they're just starting to figure their, their way through.

And then the other half is a whole bunch of very seasoned VCs who are trying to meet those same people. So it's a really, it's a really kind of magical thing, you know, it's like everyone's just kind of coming together and sharing stories and kind of like, who are you? And let me give you your business card.

It's like, just tons of deals getting made right and left. But, but it's also a lot of great advice. Really well-meaning advice because there's a, there's a peer group there too.

It's a whole bunch of other people who are struggling to meet the other people on the other side of the fence. It's great. It's great Confidence.

It is. No, it's great. I I stopped by ly.

I was, I was grabbing Michael Farham out of there. But, um, so guys, what are you doing in your spare time? You still doing your podcast?

Occasionally? Yeah. You got, Sorry.

Yeah, occasionally. Um, so I do, uh, demos, product demos. So companies will come to me and, uh, and for free, I don't charge anybody anything, but they'll come on there and they'll do, uh, uh, about an hour long, uh, presentation, 45 minute presentation.

I ask him questions with Trey Ford. He is my sort of co co-presenter. And, uh, but we just, we ask him kind of, I wouldn't say elbows out hard questions, but the kinds of questions that if you're sitting on the other side of the fence and you're a security expert, you know, if that answer was a good answer or not, you know what I mean?

And the nice part is, unlike having to put your email address in somewhere, you can just watch it and enjoy it. And, and if you want to do something with them, you reach out to them. And we've got a whole bunch of people who've be wanted to meet that company.

So it's, it's ended up being a great sales channel for a lot of companies. Good For you, man. Where, where can people get that podcast?

Uh, just look for our snake show demo day. Beautiful. Thanks.

Robert, what about you, Jeremiah? Uh, for hobbies? So, uh, I guess, uh, for the blackout related hobbies.

So, um, a long time, a long time ago, you know, I started, uh, Brazilian jiujitsu like 20 years ago. And, uh, rather than go out to the vendor parties after blackout in the conferences where there's crowds and noise and things like that, I decided to get a workout in. So I would visit Jiujitsu Academy.

So at blackout, rather go to the vendor parties, I would find a, an academy. And, uh, somebody saw me leave the conference once and they said, can we come? That was Chris Hoff.

And I said, yeah, sure. So we started trading, then like the next year, more people wanted to come. And then it grew to a, a life of its own.

Where now I put a 60 plus, uh, computer security people on the mat with UFC fighters and we learn and spar. It's like, it's a, it's a crazy event. No, it's, it's really cool.

It's, the pictures are fantastic. It's fun every time. So I love it guys.

Fantastic. It's great seeing both of you. com.

Check it out. This is gonna be something you can get, you can get in early here and and watch it. Robert.

Pleasure man. Jeremiah, two of my heroes in, in security. Uh, we're live, well, we're not live.

You're watching this recorded, but we were live when we recorded it. We're a black hat. Stay tuned.

We'll have more. Bye-bye. Hey guys, thanks.

From the Throne, we're here with Mark Curfew, who's co-founder of Crash Override. They just picked up $28 million in seed funding to address the problem of engineering relationship management. Mark, welcome to show.

Thank you very much for having me. Well, What exactly is the problem we're trying to solve here? 'cause it seems like we've been building software for a long time and despite our best efforts, sometimes it actually gets out the door.

But what is the inherent inefficiency that we're seeing in engineering that requires an engineering relationship management platform? Yeah, sure. So, uh, lemme start by kind of telling you the genesis of the company and how we found about that problem.

So both John der and I, my co-founder, had, uh, built startups before. Um, we're a little bit older than a older of people. And so we approached it a little bit differently.

Rather than come up with an idea and hoping people have the problem, which is what frankly a lot of people do, you try and convince them they got a, a problem that was a problem that was coming down their own. We sat and asked people what their biggest problem was, and we did a long time to do it. And we interviewed a lot of people.

And as we started ing those people, what became very obvious is that as people moved to the cloud and as people moved to DevOps, they had lost visibility of what was happening. So the, the sort of simple way to think about that is that when developers build software, they think of source code management, essentially as version control. It's when developer commits their code, they manage all the versions locally.

But what happens these days with DevOps is that code then leaves the system. It goes into CICD and it gets distributed around the cloud. Once it's left that version control, no one knows what happens to it, no one knows how it gets changed.

No one knows how other things get put into it, what the actual software that runs are or where it gets distributed. So as we started asking those questions, we were hearing horror systems, we were hearing stories about companies that had production outages, and they had no clue what the code was that was running on that production system or who they should go talk to to figure out where the bug was. We heard other stories that people had issues significantly issues, and they couldn't figure out what it was was because they didn't have visibility into the CICD system.

And what had changed, we started hearing about systems where people started, started saying, Hmm, these docker containers were being pulled from somewhere, but we don't even recognize where they are. And the build systems have been using what's called shadow engineering systems, systems that developers have put their credit card in and all of a sudden they had their own infrastructure, their own tools. So that's what engineering relationship management is.

And that's the problem of fundamentally that we're, we're solving. And the name ERM came from, we had a like bulb that one day, um, front and we hired some, started hiring a sales team as we started building out the go to market. And when we were interviewing those salespeople, one guy said, this is kind of like CRM, that's kind of like Salesforce.

They had the same fundamental problem. It used to be the fact that when people were, were selling, every rep had their own Rolodex. Every person, you know, was managing contacts and things that they're connected in, in spreadsheets and on bits of paper.

And as a result, the organization had lost visibility. They didn't know what customers that they have, who they should be targeting, all of those types of things. And so we realized that if you connect all of this data together, you suddenly get visibility and you have got a single source of truth.

So that's what it is. It's a single source of truth about everything you have, how it changes. And so you can go and improve, uh, you know, and frankly operate your engineering the way you want to.

Well, let's go with that CRM metaphor. 'cause one of the issues with CRM is nobody wants to enter the data, right? Or spend the time to enter that data.

So how does the engineering relationship management platform actually know what's going on and where does it collect the data from? Yeah, so we're in a lucky situation with building software engineering. The, the way it happens, of course, is through automation.

So what crash override does is we've developed essentially deep build inspection. So what that does is actually sits inside the CICD to not run as part of it. We actually are able to sit inside a building and watch everything.

We then connect up to the AWS and we go with you using A-W-S-G-C-P Azure, connect up to GitHub. And then, because we watch every change that happens inside the bill process and we know where the code came from and what changed, we know that where it gets shipped, we're able to connect those things automatically. Developers actually don't touch anything.

They don't need to do anything. Every time things flow through, we know about it. Um, and that often includes things that, that no one did know about.

So, for instance, things we found things where stuff happened in the build process, and we know, wow, these software containers didn't come from a corporate system, this code didn't come from the company's GitHub, but it's trying to be pushed into production. Or, Hey, this code was built by the company's system, but it's trying to be pushed to a cloud, which we don't actually know. So not only do we understand, you know, the, the, the space that actually people, you know, want to instrument the change Agile, but we also get to see, you know, the, the, the white space will put out to say, um, We hear a lot about bottlenecks and DevOps and everybody kind of nods their heads and says, yeah, we wish everything was ruthlessly automated, but it's not.

Is this kind of the root cause of a lot of those bottlenecks? Um, we see a lot of automation, um, for sure. But I think absolutely what you say, it's not automated.

Let's face it, engineering is kind of somewhat chaos and a lot of people encourage people, let's go, you know, create our own systems. We wanna have as least rules as possible, like, you know, run fast and break things and fix and fast and all of those sort of things. So are there bottlenecks for sure.

But I think our take on it is that what we're trying to do is help people build the highest quality software and get it out to market as fast as they can. So essentially promoting innovation, when you understand that visibility, you understand where you can improve, of course you understand where there are problems, but you can shut down those problems to improve efficiency. So, you know, for us that's, that's the kind of, uh, the kind of approach we take.

We hear a lot about platform engineering these days. There is engineering, relationship management, and the move to platform engineering kind of joined at the hip in some ways. Yeah, and same with DevOps.

So, you know, we, we look at it as coach cloud, right? So it's like developers creating code, code goes into ci, lots of different things happen, gets shipped out to cloud. So, you know, if you are building a platform deployed out to the cloud, you have to understand the complete spectrum of that stuff.

And it's not just the stuff that you produce, it's the open source that's coming in, it's the tools that you use. So we view it as both, you know, the code that's flowing through the systems, the systems and that you themselves, and you've gotta tie it all together, right? The context and the relationships between it all of this.

One of the things that's supposed important, and that's why sitting in this privileged position inside the CICD is, you know, is how you go do it. And how does get installed in the CICD system itself? I mean, how, how do you kind of get in there or become embedded in that?

Is it a plugin or how does that work Now? So essentially we developed an open source project as we were prototyping this out. And so that's referred to as chalk.

So the best way to think of chalk is it's like inserting an air tag on the code onto the bill system. So not only can you then track it, you know where it's gone, but you also know everything about what it is, what it is doing. And you install chalk at a build server level.

So it doesn't have to be going stored repo by repo. You certainly can, you can do that through GitHub actions if you want to go do it. But typically you install that at a build process.

And technically the way that works is with, with Docker as an example, once you've run chalk, you essentially alias, um, chalk to, uh, alias docker to chalk. So if someone wants to go run a dopper command, they actually call chalk chalk, then goes and calls Docker. So we essentially take control of that process, which means then we sit inside the build process, not as another thing that runs that other people could disable or not give us access to, to, to the tool.

And then we then stick a beacon that air tag onto the container so we know where it's been deployed, whether it's been changed, where it is. Um, and that's ultimately how we instrument things. We've had customers that have run literally, you can you go type one line, um, inside, inside the build system, every single bill that then passes through it, you know, gets, gets ran, and the time of trees inside.

And we will get, you know, thousands of, of, of bill reports that come back within, you know, 24 hours of some of the large customers. You cannot walk down the street, of course, without somebody leaping out to tell you about their great new AI thing these days. Is there an opportunity to apply AI to this data that you're collecting?

We are already, um, both, um, applying it ourselves, but also helping people figure out, you know, where they're applying it and get some controls around it. So, you know, an example of how we are doing it is that, um, you have large, large cloud environments. Well, what is production like?

That becomes a really interesting question. And then back to your thing, it's like, well that is, if someone marks this thing as being production, we built an algorithm that looks at the patterns of deployment, where code is coming from, who's deploying it, where it's going to, and we're able to infer where production is automatically. Is it perfect?

Absolutely not. Is it damn accurate and damn good and solving problems, absolutely. Customers that that use that are finding it incredibly useful to map out what is production and, and how, where does code go?

What is happening recently, which is also very interesting, is when we think about what code is most important, we have customers with, I think the biggest one has 200,000 code repos. How do you know what's important, right? How do you know what's actively being developed?

How do you know what you should be focused on? Like what are, what are things that are deployed out to prod? So as well as that, we do some analysis on the code, and one of the things that we've been developing is a thing called Tech id.

And what Tech ID is, is it looks inside of the code to figure out patterns of what's in there, what languages are being used, what SDKs are being used, they're using Stride that are using Salesforce. Are you using, you know, X, Y, and Z? One of those things that frankly I've been experimenting with recently that will make its way into the product pretty soon is where are you using general code?

So where are you using pl, where are you using Core? Are all of these things, because again, like a lot of the instrumentation is essentially surfacing hidden information inside of the mill process. And those tools leave hidden information inside the mill process.

So helping people understand, hey, you have code by the way that has been developed by a third party. They're not inside of your GitHub org. They happen to be using rep lit, and by the way, it's made it into production.

These are really important things to people. Other stuff like do you have a SQL server, a SQL scheer that was generated by chat GPT that is running in production. That becomes really interesting scenarios and those are the sort of things people are looking to get to Brooks with.

So from our perspective, it's not only are we using it to accelerate our own development, but we're looking how can we help customers that are using it to, to do that better themselves? Hmm. It's no secret that organizations have also been struggling a bit with DevSecOps over the years.

Is there some way to use this platform to kinda also discover or find or track a lot of the vulnerabilities that seem to be floating through our systems, but everybody's not quite sure where they are? Yeah, absolutely. So, you know, we are equally and probably more so an engineering company than we are a security company.

Our DNA has been in building security tools, but we've been engineers building security tools. So, you know, I'd like to think we're, we're somewhat the experts in the space. What you can, one of the, one of the scenarios that you can do with, with chalk and being instrumented in the CICD is, you know, what process are being run.

So not only can you see what tools are being run, you could actually orchestrate them to run underneath the herb with no one knowing. So you could automatically run, you know, tools that do SBOs, find out what open sources there, match them up against the vulnerabilities, you can run static analysis tools inside of that as well. And then of course, because you can correlate it, you understand, I've got code that's being pushed into production that hasn't gone through a security scan.

So those types of scenarios become, you know, very interesting. Um, there's also interesting and a cost piece of this as well, some of the security tools license stuff are based on a license and they're very expensive because we know, excuse me, how many people are pushing code to production? I can actually tell you how many people should be licensing that code.

And some of them will try and license you based on every single code repo. Well, guess what? A lot of those code repos actually aren't used anymore.

Great. Go figure out how to assign the money to the right places that you can make improvement versus paying this tax. Well, you shouldn't actually need to go pay the tax.

So there's a lot of interesting security scenarios that, uh, that kick in and, you know, frankly, some of them we're discovering with working with early customers. And, uh, yeah, fascinating. You go see it.

You mentioned vibe coding and chaos earlier on, and I can't help but wonder if we're gonna have more of these so-called citizen developers building applications and software. Aren't things likely to just get more chaotic? My wife's lost me to, my wife's lost me to buy coding for the last two weeks.

I, I'll be perfectly honest, it started two weeks ago. Uh, actually it started at a, at a board meeting. Um, Google Ventures were, um, were one of the large firms that, uh, were participating in the seed round.

And as a, as a side conversation, it was like, Hey, this stuff is really increasing. It's just incredible. And so a couple of weeks after that in New York, my wife was out, I'm sat there with a glass of wine like, Hey, let's, you know, let's not watch tv.

Let me go figure this out. And since then, you, frankly, she's lost me. It is fascinating far from panacea, gets a lot of stuff wrong all the time, creates all sorts of problems, but absolutely it is able to essentially, you know, it's essentially enabling non-technical people to go go software.

And it's enabling technical people to be able to create decent software, and it's enabling professional developers to have superpowers, right? It's, it's right through the spectrum. But if we think about that, instead of now having 20 million developers of whoever it is to building code, all of a sudden the world is able to build code.

If we think about a company, you know, it's typically the development team, now it's the marketing team, now it's the accounting team that can go produce code. So this visibility problem is actually getting significantly worse. We're at an inflection point.

So, you know, the vi coding thing is absolutely phenomenal, and I fundamentally believe it's changed the world and it's changed the world of how we build software, not really just vibe coding, kind of the, the, the non-technical people leaning in, getting the vi but the gen AI stuff, right? As, as well. Um, and that's both, you know, also with the ability to build tools.

So anthropic last week released a set of prompts to go do security analysis, um, using Claude. It is incredibly effective, like as effective as some of the stats, some of the static analysis tools that we've been using for a long, long time. So yeah, changing the world, fascinating.

And, um, there is absolutely really interesting things that we are, we are, we are doing both helping people use it more efficiently, helping people find out where it is, but as a company embracing it ourselves to accelerate what we're doing and make our products better, What's your best advice to people to get their arms around this thing known as chaos? Because I think in a lot of instances, it's kind of like a backache, we get used to it, and so therefore we don't realize it's an issue until somebody comes along and solves the problem for us. But where should folks get started?

Where's the, where's the, where's the point of entry for the journey? So, I mean, look, I'm an early, early stage startup guy. Chaos is what I love.

So, so, you know, maybe you're asking someone, uh, interested in shutting it down is not necessarily, uh, um, I, I I embrace it. And I think the reality is it happens everywhere. Um, even if you think in large companies, large companies now have pockets of teams and innovation is encouraged in small teams.

So to a large, you know, to a certain extent, even teams that, even companies that don't have chaos now are studied to create small teams of chaos. So, you know, the first thing is to embrace it. You know, it probably sounds self-serving, but to a large extent, visibility is understanding, you know, is the first step.

Once you understand what you have, you're able to decide what bits of chaos do you wanna embrace, which bits of chaos do you wanna figure out how to, you know, pull back and put some controls up. So, you know, an example of that is tools, right? So like I said, developers can put their credit card in now, and I can have, you know, the biggest world supercomputer A AWS, right at the end of MyCorp card.

I can have my own Docker system, you know, that sat there. I can have my own GitHub org, right? The, the the age of shared security services, you know, is, is a challenging one yet.

So, so from a security perspective, the security guy wants to make sure he's got the right security controls on the right thing, but from the engineering side, they wanna make sure that they have supportable systems, they wanna make sure that, you know, developers can go discover containers. They don't have to go build and maintain their own, as an example. So, you know, there are certain things that I think, you know, are, are powerful for efficiency and effectiveness for engineering in, in controlling the chaos.

And there are certain things that you wanna, you wanna let go, right? And you want to, you wanna encourage, like, you know, allowing people to, to explore and, and, and embrace, you know, clawed like embrace rep lip. Like you do not wanna be shutting that down.

Like even though it may be creating chaos, like it is undervalue reef changing the world, um, but understanding where it's being used and there are other things that you need to figure out, like, okay, now I need to stop putting controls on, on that thing. Um, but, you know, it's self-serving, but visibility allows you to go and make those decisions. And I think that's what we, what we see in a lot of organizations.

Well, folks, you heard it here, there is a method to the madness. It might just be called engineering relationship management platforms. We'll see how this all turns out in the months and years ahead.

But in the meantime, buddy, thanks for being on the show. Thank you very much. Appreciate it.

All Right, and we'll see you guys next time. Hey everyone, it's Alan Shimmel for Textron, and welcome to the first episode in a series we're doing that we call Shift Left Shift, right Shift Everywhere. I am really happy to be here.

I'm really happy to have these two guests I'm gonna introduce you to in a moment. You know, we're doing this series with our good friends at Adobe, and I know everyone out here has heard of Adobe, and many of you use Adobe products, but I don't know how many of you know how influential Adobe has been in the world of security over the years. When you, when people are trusting you with, with the, their files and their work, like millions around the world do with Adobe, they don't have a choice but to take security seriously.

And as we were talking with my guest offhand, off camera, you know, a lot of security innovation has come out of Adobe. Um, Adobe of course, is all about you, the technical people out there who are working in all of their products for graphics and documents and applications and everything else. And they have for a long time.

This whole series is gonna be focusing on sort of what's Adobe's view of security about what's some of the frontiers, some of the, you know, areas of security that we, we wanna shine a light on. And, and specifically as I said right in the title shift, left Shift, right shift everywhere. Where do we put our focus on security?

Let, I've got two great folks from Adobe to introduce you to who are gonna be talking about this with me. Let me introduce them to you now. First I want to introduce you to Pelli.

Yuli. I hope I got it right. I've got, I'm doing the best I can on names, but Pella's name is actually not that hard.

Pelli is the lead security strategist at Adobe, and we're thrilled to have a on Pelli. Welcome. Welcome to our podcast series here.

Share with our audience a maybe a little bit about your journey. Um, sure. So I've been in the security industry for 25 years.

Um, I started out working for a company called Anonymizer, which was sort of a commercial version of tour way back when. Mm-hmm. Uh, I worked Insecurity consulting for a while with had Stake and Symantec, and I've been at Adobe for 17 years now, working in all sorts of areas of security.

And, uh, when we brought, uh, Florian into the team, I decided to go and focus, uh, mostly on Shift Wright type projects. So I'll be representing the Shift Wright aspect of it. So you're the right hand.

Yes. I hope it's still right on your, this is my right hand. I sometimes it mirrors.

I know, but that's funny. You know, at stake of course is legendary, right? Chris w Ball and, and the folks there, they went to semantics.

So it sounds like you were involved in in all of that, you know, in the in I've also been in the security business 25, 30 years, legendary, legendary folks there. It's still doing great things. Um, but thank you for joining us.

Sure. Next, let me introduce you to, uh, Florian nut netting note noting, I know you gotta curl your tongue and in New York we just don't curl it so well. But Florian nerding, Florian pronounce it correctly.

It tell us it's a little bit about yourself, helped me or rescue me, Difficult name, my name's, uh, Florian Newing, or if you want to use the German ation because I'm originally from Germany, then it's Floridian, which is even more difficult. But let me also talk a little bit about what my background, I started my professional career in, uh, 2000 and and 10 at a small startup, which built, um, network firewall, the devices with a focus on being very, very user friendly so that anyone without networking or security ex expertise could actually set them up and have a secure net network for their, their office or their, their home. Even after a couple of years working as a software engineer there, I joined at, at Adobe, and I've been with Adobe by now for 11 years and, and for most of the, well, a bit more than half of of the time i, I spend in software engineering.

I am, and it's still what I, I'm at heart. I'm a software en engineer. I want to make the lives of, of developers better and really focus on pragmatic security solutions.

Roundabout six years, I, ago I joined the security org, started working to together with, with Palace, and I'm taking care of all things shift left. And so the cutoff point is basically when software gets deployed to the cloud or otherwise released to our customers. So in, in my scope is there's a lot of stuff from security training, security, awareness, code analyzers, and various aspects around secure by, by design, and especially memory safety.

Excellent. So you're the left hand? Yes.

Got the left and the right. Okay. I feel like the Pope, um, anyway, He's home from the hospital, so that's good.

Anyway, um, let, let us, let us talk a little bit about history. com in 20, uh, November of 2013, published March of 2014. A big reason that I personally felt compelled to do this was because I thought that DevOps offered us the best toe of, of getting security, right, of, of correcting a lot of wrongs, right?

I I, I grew up, or I, or my career in security, probably much like you, Pelli was on the right side, right? AF post-deployment, I helped found a company, intrusion prevention network, access control, vulnerability management, you know, all the traditional network security stuff. And the problem was we were, we were always the caboose on the engine, right?

The end of the train, the engine got pulled by the developer or, or someone else, right? It was too late. By the time we got involved, it was too late to often to fix a lot of the wrongs that were there.

And I always felt if we move further up the food chain further left, if you will, we would be able to fix these things. And what a perfect opportunity DevOps was, right? Ops and dev working together, let's get security in there and we're gonna move security to the left.

And you know, the, at the time the notion was, and I don't know if you believe it, I'll ask you both, that it was a fraction of the cost to fix a vulnerability or a defect far left than it was to try to fix it in production in the right hand, right? So it was cheaper, it was more efficient, it was, it was just everything was better doing it to the left. And why start just left of deployment?

Let's push it all the way left. Now, like both of you, we, we have friends who are developers, but the average security person said those developers, they don't care about security, they just wanna push out code, right? They get paid to about how many lines of code they publish.

But an interesting thing I learned when I got into this DevOps thing, a lot of the developers, and not only the developers, all the people on that left side really felt that the security people were like an anchor that was dragging them down. They were slowing us, we were slowing them down. We were the people who say, no, no, no, go back, go back, go back.

No. And I found it incredibly difficult to bring together what I used to call the, the, the cybersecurity, or we didn't even call it cyber back then, but the security tribe with the DevOps community, it was sort of oil and water. I was trying to make chocolate and peanut butter.

Pelli, you've been around if you were at at stake. You've been around a while. I know.

Yeah. What, give us your take on that. What do you, you know, was, was it an impossible mission to begin with?

Uh, I I don't think it's an impossible vision. I mean, part of, even as a shift, right person, right? Like my job isn't just to find as many bugs as I can.

My, I'm a feedback loop into, into florian, right? So, you know, we go and we try to look at patterns of, in within the vulnerabilities and say like, okay, are the developers having this consistent class of problems? If they're having this consistent class of problems, you know, what can, you know, Florian and I coordinate on and what can, uh, Florian help build to address that class of problems?

Like how can we shift the company to using a framework that's maybe a little bit, um, more secure by default so they don't have to think about security as much. Uh, maybe it's a pipeline problem. Maybe, you know, it's they're, they're having trouble keeping their amis up to date in, in the cloud.

So, uh, it's, I I found that developers tend to want to do security. Well, they, they, some, a lot of times they do find it sort of an interesting topic, but they're, they're just constrained by the realities of, of their situation, right? They, they have so much time and, um, to get things done.

So, uh, from my perspective, you know, I'm not just looking to find as many bugs as I can to get as many points on the board as I can. You know, everything's a feedback loop. Even if you're doing red teaming, the, the goal of a red team isn't to go nen or nen or we got in the goal of the red team is to then talk to the blue team and say, look, this is how we got in this, this is where you have gaps.

Um, if you wanna catch us the next time we do this, here's how you can improve. And so there's always a feedback mechanism in, in from shift, right? To, to make the shift left team, uh, more knowledgeable and enable them to make better plans, to make things just smoother for the developers overall.

Absolutely. com, Florian did all this DevSecOps, to tell you the truth. Give us your, you know, what, what's been your experience at Adobe primarily?

'cause that's where you've been to all these years, but is what I describe, was it true then? Is it true now? What, what's changed?

What's gotten better? So there are multiple perspectives on on that certainly DevOps, the, the ideas as fantastic. We have a group of people who really focus on, on the engineering aspects of building working software and operations people who then run it in production and take care of all, all the problems that happen in production there.

We have a feedback loop too. And if we now add security to to, to that mix, both sides need to, to do some of the work. But the challenge with shifting too far left is we, security people should not move all security work to the en engineers operators of systems because they are not experts.

We are the experts. So we need to make it as simple as possible for them to find these issues. And there are many different approaches of shifting left.

For example, you might shift left and say, well, let's do threat modeling at design time, because obviously it's cheaper to change the design that hasn't been implemented yet. Then while you have a architectural complete, um, system on, on stage ready to be deployed to production now and architecture change is very hard. It's, it's too, too late.

So shifting left in that sense is very, IM important, giving all kinds of feedback in an IDE on, on the other hand. Well, now you need to balance different aspects. Do you want to send all the findings to, to the developers only the sets that you care most about?

What is this the set, what, what security aspects really matter? And with my background as a software ENG engineer, I, I wanted to always help others of the engineers make pragmatic security decisions and Italy reduce security decisions. So the recent trend in shifting left is secure by design solutions that's, for example, started for cross scripting issues, um, with libraries such as React, where it's really hard to accidentally have, um, injection vulnerabilities because the framework by design prevents it.

And that is a very, very powerful concept that I want to see much more of. Yeah, the, the, the secure by design, that whole concept of secure by design does not get enough light, right? I mean we, we all, for instance, Pelli, I'm sure on the right side of things, right?

Does zero trust networks zero? The idea of zero trust security, right? Everybody kind of wraps their head around that talks about it.

It's, it's very, you know, very, uh, everyone, you know, buys into it, so to speak, the secure by design. I think people shake their head, but they don't necessarily drink the Kool-Aid, if you will, right? In that.

'cause at the end of the day, they're not quite sure what secure by design means, right? Y yes, of course we want to design secure software and we want to try to put in frameworks that take out you buffer overflow, SQL injection, you know, the OO watch top 20 or whatever, right? That hasn't changed in 17 years, but, you know, but actually implementing that is hard.

It's hard. And without, again, some ground rules, we, let's not let out state secrets and get us all in trouble. But how does Adobe do secure by design?

Yeah, let me talk a little bit about memory safety in, in this context because it, uh, showcases the fundamental challenges that we have have to deal with many of Adobe's products, like any company that that is more than 10 years old probably has lots of CNC plus plus code. You know, operating systems are written and c and mostly c, maybe some in CC plus plus desktop apps. C foundational libraries are all CNC plus plus just desktop apps themselves.

CNC plus plus. Look at any network d device at code running on other than the apps on on your mobile mobile phone, whether iOS or Android doesn't matter. The foundations is all c and c plus plus it's all memory unsafe.

And unfortunately we have learned that humans are not capable of reliably writing memory safe code. It's just too hard. So we need a a system solution and that is memory safe programming languages where a smaller group of of people is just focused on, on designing a system where it is very, very hard to have accidents like, like that If you use Java, Python, well these are not systems programming language.

You don't deal with memory safety issues. If you need to write highly performant code, well then you have rust or may maybe swift. Uh, the two most common choice there are certainly more than these two programming languages.

But if you now look at, um, the ecosystem where you have memory safety issues, it's CNC plus plus. You can't just rewr an entire application in a memory safe programming language. There's no business case to ever make that happen.

Even if we had a way to automatically transform, uh, tens of millions lines of code base into to rust wouldn't be interesting because the team that maintains the c plus plus code base couldn't maintain the rust code code base. They wouldn't understand the structure if we used AI to transform it, if that would be possible. So we need a much, much smarter a approach to memory safety.

And the first step is, again, feedback loops. We need to identify which parts of, of, um, the system are most vulnerable to this kind of vulnerability and does this vulnerability matter at, at all? And that is where the shift dry testing comes in.

And I'll hand it over in a moment to palace to speak about fuzzing and what we do there and once we've identified these spot that are safety critical will recognize a recurring pattern that especially areas that do, um, pausing and decoding of file formats are risky. And it doesn't matter if it's an image file format, an audio and, and, and video or a complex document or even an archive, it doesn't really matter. That is the key functionality that we need to protect because, and adversary is that sends you a file via email phishing via phishing, which is very targeted phishing.

And with one click, you open the attachment and then open it with an application and then the adversary achieves remote code execution. That is really the thing we want to avoid. So figuring out which code is executed during this one click attack that is most, most important and it's file pausing, decoding, and maybe a little bit of running logic, then you can take different mitigations strategies instead of rewriting everything in a memory safe programming language, maybe rewrite one safety critical component in a memory safe programming language.

Palace. Can you talk a bit about fuz? Yeah, sure.

So, so this is one of the areas where like you, the goal isn't necessarily always just to find as many bugs as you can. It's to do things strategically. And this is where shift left and shift, right?

Collaborate. So yeah, when we're trying to decide what to fuzz, we could do like just generic fuzzing and try to go after the entire application all at once. Um, but to do a more strategic approach, you would look at your adversary intelligence, right?

Like in, in the wild what file format types are attackers currently using to go and exploit things? You can look at bug bounties and you know, the people that you have in your, your bug bounty community who are contributing crashing bugs and looking at the techniques that they're using because they're often also emulating what they are seeing, uh, in the adversary intel community. And then you can go work with the product teams and go, okay, who are the teams that actually are responsible for this code?

We can go and you send a specific team into there, we can work to set fuzzing around that specific section of code and it can actually make the developer experience a little, uh, more predictable. 'cause you're, you're directly working with the team, you're working with one team at a time or two, maybe two or three teams at a time, uh, to do this type of work. They understand what, they understand the bugs, they're not context switching.

Um, like if you're just fuzzing the overall application, you're hitting different teams all the time and they're concept switching versus, you know, working with a team directly where they're like, okay, we're gonna focus on this problem for, for this quarter and we'll we'll work with you. We'll set up the fers we'll, we'll give you insights. And then, uh, they can start to see the patterns and the bugs.

And if they see the patterns and the bugs, they can say like, okay, well you, you can quit rank the fuzzer. We, we know this paradigm that exists in the code, so we're just gonna go tackle that overall and then we'll come back to the fuzzer once we've, we've addressed that. So, uh, you know, with with shift, right?

You know, I'm always looking for ways not only just to, to find the bugs, but also ways, uh, to do it effectively and ways to empower the teams to move faster. You know, I remember the first time I was exposed to fuzzing, so I think it was black hat around 2006, maybe, something like that. And, and what a, what a fantastic development that was for what the time, I don't even know if we called it AppSec.

Pelli, I don't know if you remember, but did we call it AppSec then? No, not really. It was still, I guess vulnerability management.

I don't know. But I mean, what a, you know, the whole idea of fuzzing the code and looking for, you know, the, the zero days before the bad guys found them, if you will, was, was just, you know, what a concept like, duh, why didn't I think of that? Right?

And I wouldn't be working here today. But, um, it, it, it, it really did help us a lot and it helped the developers fix code, right? Not in real time, but much earlier in, in, in the, uh, in the process.

But, you know, I I also, I feel almost like duty bound to say we have made a lot of pro progress on memory overflows and, and, you know, memory vulnerabilities in, in our code at Adobe as well as, you know, all applications we're, we're better at finding those kinds of, of, uh, of defects of vulnerabilities now than we were 10 or 12 years ago. We, we, we have, and we also have new, you know, you mentioned yes, the world's full of Brownfield, not greenfields, unfortunately, we have a lot of legacy code written in c and c plus and even C Sharp, but you know, we're seeing this at the Linux Foundation now, right? Lioness, lioness says we should be using rust.

Yeah, there's, there's definitely been a shift and you've seen it across the industry, so there has been progress, right? Like Microsoft's done a lot of work to introduce secure compiler flags. Yeah.

That can help secure code at scale. Um, Microsoft themselves have been playing with Rust in, uh, in their code and they've been putting rust into the, they've written a, a couple blogs about that. So things are getting better, but, um, at, at the same time, it's always a race, right?

So, you know, you're, you're always, they're always gonna find one more way or one more tactic. So it, it's always gonna be a bit of a progression, You know, it's good. I'm sure It's finding, you know, security is always constrained by the EE economies of building software and selling it.

So if you can't make money with it, well, even if it's perfectly secure and turning something off is usually more secure than running it. So we need to find an acceptable risk threshold, and for example, for our products, aggregate and and reader, the addition of sandboxing to really isolate the memory, unsafe parts. And yes, we have active content and, and, and there two from the rest of this system allowed us even before we had secure by design solutions like memory safe programming languages for systems use to reduce zero days and vulnerabilities in, in, in this area by large degree.

So there are many, many different techniques. And, and the key thing to always figure out is what is the best way, the most cost efficient way to mitigate risks at scale? And as security professionals, we always have a pretty large toolbox available and we need to help the software engineers understand what are the options and tells them about the different pros and, and and cons, both short term and, and long, long term.

A sandbox doesn't fundamentally remove the vulnerabilities in libraries that it protects. So we still have to, to fix any bug we might find. Whereas in a memory safe programming language, you have eliminated or reasonably eliminated a class of, of vulnerabilities.

Yes, rust, you can use unsafe, but I then you better know what you're doing. Yeah. And we're, we're sort of, uh, you talk about the industry changing, we're uh, at a place where, you know, like when I first started, like finding a bug was super cool kind of thing, right?

And now, uh, you know, and a large enough company, you, you have, you have tons of bugs, right? So like RS a coming up and there'll be a ton of vendors on the floor who are gonna be marketing, application security, posture management tools. Sure.

Which are, you know, take into account that you've got vulnerability feeds from all sorts of places. You've got your internal pen test, external pen tests, bug bounties, ds, sas, Kev list, um, cloud security, posture management tools, et cetera, right? So you have vulnerability.

You, you now have a wealth of vulnerability information available to you. And, uh, part of working together with shift left and shift right, is being able to look at that data and look at that information and say, how can developers most effectively spend their time to, to knock down as many vulnerabilities, uh, with as little effort as possible? Is it updating their baseline images?

Is it, as Florian mentioned earlier, switching language to like react or rust? Is there some sort of tool in the pipeline that we can build that makes, you know, keeping these things up to date more, uh, easier, uh, for the developers? Um, you know, managing third party libraries, you know, since right now we're at like, at almost at the other end of the spectrum where it's, we, we have a wealth of information.

Now the question is, is how did, how do we use that information effectively? Well, we're almost a half hour in and we haven't mentioned ai. It's time.

You know, is AI the answer to that question? Bella? Uh, AI definitely helps.

Like AI is, is another tool in the toolbox, right? Uh, so you know, you can use on the shift right side there, there are places to use it. And I'll let Florian talk about, uh, places in shift left.

Um, in, in the shift right side, like because you have all these different tools, you'll have the same bug finding for multiple tools. And the a common, uh, AI function is document similarity search. So you can do, you can do deduplication, make sure that you're not double filing bugs against teams.

Uh, there are tools, uh, to make reproduce, uh, the reproduction of tool, uh, the reproduction of a vulnerability, uh, easier. So they can take a bug report and translate it into a nuclei template, which, uh, utilize an open source tool for, uh, doing scanning. Yes.

That, that helps the development team in terms of reproducibility, uh, when they get a bug report. So there's definitely places where, where it can help. And we've seen, uh, places where it helps and also places where it expands, you know, the attack surface that I have to monitor as well.

Yeah, Expanding the attack surface is a good, good keyword. We are living in a world where more and more code will be authored or at least co-authored by AI systems. And these large language models, which writes this code for us, have been trained on publicly available source code, which of course has been written by humans and has sometimes a lot of security issues.

So you might find that AI generated code is not substantially better and maybe not substantially worse either than human written code. But since much more code will be generated than humans can produce in the same amount of time, we should probably think about, uh, addressing these concerns that the root cause. So can we get into the space where, um, AI generate code for us to directly influence how the code is generated and take care of security recommendations at code generation time?

That is as far left as we can, can go in and in the process, um, at least for, for code, we can could also use AI to auto generate code fixes. So if you understand a, a pattern well enough have AI after it was somehow detected, have AI rewrites the code and so that it's, um, vulnerability free, for example, from using string conation to create SQL statements to parameterized queries. That is, especially Im important when queries need to be dynamically con constructed because in that's the edge case that humans often get, get one.

We are also running other AI experiments, for example, on our block. You can, can find a post about how we think of AI for use and, and threat modeling. That is an an experiment that, that we are still con continuing to to this day to, to see can we recommend something where humans truly excel at with AI use to scale it across the entire company.

Because, oh, economics, again, you can't start model every tiny feature by a security specialist, but AI could, is it good enough? And the answer is still, still open, but let's, let's see how, how this space e evolves. I don't think it's good enough today, but it's getting better every day.

Certainly. And an interest thing we hear from security companies and developers is that today anyway, AI might be better at fixing bugs than it is writing code. So in other words, if you give a code that a human wrote it could find and fix vulnerabilities, bugs, whatever you want to call it.

And it does a better job than that. And then if you just ask it to write code for an application, then of course a human or someone else has, you know, something else has to look at that code. Um, but certainly we're not at the point where, where I think we can trust it to just write the code for us.

And, and, and security is, is, is, is at the top of that list. Very much so. Um, but you know, you mentioned something before about third party components and this has really been a bane of shift left and shift right of shift everywhere.

'cause we have to be in the repos. I mean, today software is assembled on an assembly line, like cars are, I assume it's the same at Adobe, you're not a right. Most of that code inside of these applications represents components that come, they're open source perhaps, or they, you know, they come from repos, container repos or, or or whatever.

And, and a lot of the security incidents that we read about or hear about are the result of third party vulnerabilities that made their way into code, not from the developer actually writing that code at the company, but from the third party component that was assembled into that code. This the software supply chain, this whole issue of SBOs software biller materials, right? And that's a left and right issue because you know what, when you're assembling the code, integrating it prior to deploying, yes, you wanna make sure your SBO m is is up to speed.

But that s om has to almost be a dynamic document that, you know, as things change, it changes and then pelli you on the right side of the house have to be able to reference that s om to say, Hey, does this thing need an update? Or is is a component here out of, out of, uh, you know, they found a vulnerability, we need to upgrade that component. Are you already starting to rely on SBOs to help fix or to help secure the software supply chain?

Yes. We, we do that is one of the projects i I lead. Okay.

So yeah, the basic idea is first you need to figure out where do is your visibility into the software composition limited, especially with cloud native applications, things that are developed in modern programming languages. Any one of these parts in Java, Ruby, JavaScript doesn't really matter, usually has a good package manager. So it's relatively easy to introspect a GI repository or a repository for the packages that, um, software depends on, on figures that out, even at deployment time.

Uh, cloud native security tooling can figure that out too. But there are gaps in older systems, especially CNC plus plus, again, just like memory safety is a, a problem there. The software composition is hard to determine automatically.

So we are working on, uh, on improving the ability ability, especially in these areas to understand which dependencies to have our, uh, CNC plus plus based products, how do they relate to internally and to external components. And that of course, this visibility then enables us to, to have a more standardized approach to vulnerability management. And Palace mentioned earlier things such as C Catalyst that is CSARs list, obviously known exported vulnerabilities, things that have been exported in the world so we can prioritize the remediation of these issues and a whole lot more pillar.

Yeah, and this is also a place where, you know, secure coding often gets talked about separately from just standard coding practices. And this is like an area too where, uh, you know, teams that have good development practices that have the ability to do automated, uh, testing in their environment to confirm, confirm patches, uh, the work that they invest into that actually benefits security. Uh, it's a mutual win for both teams because the more, uh, testing they have that's automated and can confirm something and, and get you closer to a continuous deployment model, the easier it is for them to test these third party libraries.

Like one of the things that a lot of developers, uh, have a challenge with, with testing these things is that occasionally there's, you know, a breaking change where you have to go and re-architecture code to, to deal with the new version and they're always scared of that. And the longer that goes on, the higher the probability of that occurs. And so, uh, a lot of times, you know, when we're partnering with developers, we'll look for opportunities where the thing that they want is also something that we want.

And you know, so if we see them like, hey, we wanna do initiative to improve testing just normal testing like unit testing within the organization, you know, we'll go and we'll back that and say, yeah, the security team believes that would be a good investment as well. So there's opportunities to look for, uh, partnering with, with organizations on that. And then from a shift right perspective, yeah, we have to keep track of all the feeds and one CBEs and, um, which ones are relevant, you know, are they on the KEB list, making 'em a higher priority, uh, those types of things.

So this is definitely something that we would monitor on the shift right side. Great guys, I've got one more topic area I wanna jump in on and that actually brings us full circle back to the beginning. I said the name of our episode here is shift left Shift, right Shift everywhere.

It's not enough to have one hand shifting right? And one hand shifting left. Those are two hands, they act independently and they're not necessarily coordinated, right?

Video directors say don't stick your hands out too far. You go out of camera, so I gotta keep 'em here, but so your hands are not necessarily coordinated. The idea behind Shift everywhere is coordination left and right working together, right?

Not in. Absolutely. Yeah.

Talk to me about how Adobe, other than having you both on the show with me, how Adobe is, is putting left and right together to truly shift everywhere. Uh, sure. I I can start that one.

Um, so one of the things you have to keep in mind too is we talk about shift left and shift, right? And that's important to the security team, but when you're working with the product team, they, they just know the security org, right? So, you know, the reason why we wanna collaborate and work together and, and come up, you know, make sure that we're coming up with like unified solutions and looking for patterns and looking for higher ROI activities for 'EM is they wanna hear from a security team from a single with a single voice, right?

They, they just need to know what they need to get done, um, and what needs to to happen, uh, to get there. And so, you know, with Florian and I, we we're in constant communication with each other every day, every day of the week, um, about some topic or another where they're, we're trying to collaborate so that when we go to development teams, there is a unified voice and I can say like, Hey, these group of bugs don't deal with them individually. It'd be better for you to do this thing.

And Florian can help you. Florian and his team can help guide you through that. And that makes, you know, just a better relationship between the security team and, and the development teams to, to know that we're not just coming up with work for them to, you know, busy work for them to do to, you know, prove we can find bugs, but that we're trying to actively work with them to, uh, get the most security from, from the limited time that they have.

Um, and, and Florian, do you have anything you wanna add to that? Yeah, um, we have so many different tools and as PET said, speaking with one voice is, is most important. So telling the engineering and operations teams what exactly is the most efficient and effective way to reduce their security burden, that is really important.

And this problem feel might feel simple if you only deal with one product, but at Adobe I'm dealing with many different products. So I need to rely on multiple teams that help both PE and me sends this message and amplify it at scale to many, many en engineering teams that use different text stacks, have different products, have different business cases, are facing different kinds of threats. So in really identifying what are the key things from a risk perspective to protect our crown jewels, and this might vary by, by product certainly is very I important.

And then PE and I work closely together to figure out what are these risks. We work with our security partners to amplify our message and we work with the security partners to pull in the specialized functions of our security organization to affect positive change. And the part of that is certainly also evangelizing for security to create awareness because, um, not all business leaders might be aware of the security threats a product is, is facing.

So really having a holistic perspective is super important. And there is a model that I use, how, how to think about the kind of the maturity of, um, the securities that that we have. And I found it on, uh, Colin Green's block.

See basic, I I am, when you classically think about shifting left, you start with the development process. So design right code, build test, deep deploy and and, and so on, and then left this at the beginning of that process. But instead you can, can have a different model that calling green called see the six buckets of security risk and see rightmost one where I start is exploited.

That is the thing we want to avoid. Then we have the bucket of unfound and most risks probably stay there. If you now add investments, you can shift things further left to found externally, for example, via a bug bounty program.

Further left, found internally, but manually manual testing, pen testing thing that rat teaming, oh, you can decide if that's external or internal, doesn't matter too much. Even further left, found automatically with an automated code analyzer. So solution and even further left to prevent it.

Then ask your safety question, what is your maturity? Where do you prevent risks? Where have your only capabilities to find them automatically or man manually?

And that is much more, more expensive. Then the economical question is not, can I do this at this time and moment, find a security issue, but how can I address the root causes of issues instead of only fixing symptoms? So it's a whole different way of thinking about a vulnerability management program and using all the tools you have at hand to make it better.

That was excellent. Thank you Florian. Guys, as I promised you when we started, I was gonna try to keep this under 45 minutes.

We're, we're hitting right up against it. I feel like we've barely scratched the surface though we have a lot more to go over and I look forward to continuing our discussions in, in subsequent episodes of, of this series. But I think we've laid a great, a great foundation here and, and defined a lot of these things.

And what's nice is sometimes we talk about this in such an abstract way because we don't have a real live company who's actually living and breathing this every day. Adobe is living and breathing this every day. And, and that brings a, a, a reality show, if you will, aspect to things where, hey, this is, this is what we're doing and this is what works for us.

So thank you both for coming on. Thank you to Adobe for participating in this series. Thank you for watching this.

I hope you found it interesting. Um, if you're watching a summary of this, click through, go watch the full, the full 45 minute version. It's great.

Until next time, this is Alan Shimmel for Techstrong. Thanks for what, what being with us today.