Securing AI-Native Development in the Age of Intelligent Threats

Hey everyone, it's Alan Shimmel. Welcome back here to Text Drunk tv. My next guest, it's actually his first time, I think.

Well, it is first time on Text Drunk TV with me, though. He's been at Datadog six years, so I'm glad he finally made it over here. Emilio Escobar is the CISO, CISO at Datadog.

Emilio, welcome. Yeah, thanks. How are you?

Doing well, thanks for having me. Pleasure to be here. It's My pleasure.

So am Amelia. I mentioned that you're, uh, CISO at Datadog now for, I don't know, six years or so, maybe more. Um, give us a sense of like kinda your journey to becoming CISO there.

Yeah. Um, I'm, you know, I think I'm definitely not one of them who had a career plan to end up where I am right now. It just happened organically.

Uh, so background is a software engineer computer science degree. I, uh, did a long stint with the federal government, uh, doing some, some software engineering there, but all of it was specifically purposed in, in, in security itself, different aspects of it. Um, did security consulting for a bit, uh, and ended up meeting, uh, the PlayStation team while I was doing that.

Went to PlayStation, ran the software security teams there for a while. Uh, then went to Hulu to lead the whole program for a couple years. Took that through the whole Disney transition and they did that called, uh, I by then already knew of Datadog.

Uh, Hulu was a, uh, a new Datadog customer while I was, while I was there. So I already had some connections to Datadog and, and when Alexi called, uh, had the conversation with him, it, it all felt like it, it was a great fit. And, and yeah, six years later, here I am.

Excellent. Excellent. Um, what a great story, man.

Right. Just so you went from Hulu, that was a cust a new customer of Datadog right? Over the Datadog at the time, right?

Yeah, yeah. Yeah. And I remember seeing the demo of Datadog's first security version of a security product, or a demo of their first version of security product while I was at Hulu.

Uh, this was reinvent, golly, like maybe 2019, I think. Um, mm-hmm. And, and seeing the, wow, this is, this makes sense.

But back then obviously it was very early. Um, and, uh, and yeah, it's, uh, that's when we got first got insecurity and, and, uh, needless to say, back then, I didn't realize I was gonna end up playing a major, major role in it. So.

Absolutely. That was, that was probably like right before COVID too. Yes.

So I joined the company during COVID, so I didn't, I didn't get to go to the office, uh, to meet people in person until a year after I had joined, which was, yeah. Well, it was a crazy time. It was a crazy time.

It was, it was interesting meeting people, you looking back on it, seeing People alive and like, oh, that's what you actually look like. Absolutely. I, we, you know, we went through that.

I, you know, 'cause it was, it was COVID. We were, we were still operating, but, you know, and I, I was hired like interviewing people that, and then hiring people that you never actually met. Right.

You know what I mean? It was a little weird, interesting times. But we're, we're in interesting times now, right?

Yes, we are. And I'm sure as much as it would security was new for Datadog then, you know, fast forward now, six, seven years later, well, security's not new, but everything we're doing seems to be new because of ai, LLMs agents, agent ai, and all of these things. It's really rewriting the book almost daily.

Right. Talk to us about, you know, Datadog, how to evolve Datadog, how to adapt and change and grow. What, you know, from where you sit is the ciso.

How do you see that growth? How, how does it play out? Yeah.

And, and it hasn't just been security. We've, we've seen that expansion even in our observability business as well. Uh, so what's been interesting throughout the years is to see Datadog from going from just a, an observability platform to be really a mission critical platform now.

Uh, because now we no longer just tell you, Hey, something's going on in environment, we allow you to actually act on it and security on top of that. So the, the mission criticality for us is definitely exponentially grown. Uh, and what we mean to our customers has, it means something completely different.

Uh, but where there's an opportunity, there's also, um, uh, threat actors that want to, uh, want to exploit that. So we, we definitely have been helping customers understand, uh, again, not just using AI to understand better what's happening in their environment, how to address it, to the point of, of having agents that can do the things for them, uh, but also helping them protect their AI applications as well as they're working on their journey to, to release AI application. So we have an SRE agent that helps you find root cause analysis.

Uh, we have a security agent that helps you triage, uh, all the thousands of signals the security teams have to evaluate. Uh, and then we even have a coding agent that helps you fix code. Uh, and if you really put it together, uh, there's a, there's a whole feedback loop that we cover, right?

We, we can have a, an error in production or we can have an issue of security that was introduced by a change, and now we have an agent that would actually make the code a configuration change or propose it to address that. Right? So really close end to end.

So, uh, this is being well received, but that means our responsibility to our customers has also grown with it. Excellent. You know, I, I was reading something the other day that 2026.

So 2025 we started experimenting with agents. Mm-hmm. But in 2026, people don't want to be experimenting anymore.

They want it to just work. Right. Right.

And, and, and let's be honest, in 2025, a lot of the agents, you know, they had that cool factor, but they really didn't work so good. Right, right. A lot of it was, yeah, I could see the potential, but this isn't quite ready yet.

When it comes to the Datadog agents, are they ready? Are they, are customers using them to deliver value today? Yeah, we see that, um, we see that they're working and they're delivering value, but also we continue to invest in them.

Right. So what's Olivia said this, our, our CEOs founders said this in an interview a couple days ago. It's interesting for we've been okay with humans being correct 70, 75% of the time.

Uh, what's interesting about AI agents is if an AI agent is correct, only 70% of the time, notice my statement, only correct. 70% of this time, we don't know, we don't accept that as a, as a good product anymore. So it's really fascinating.

So, um, while they're working and customers are seeing value, we still get a lot of feedback and we still continue to, uh, work on them. 'cause we know there's so much more we can do. Uh, but then we work in them in the sense of if there's a situation where the product they didn't get to cover, now we work on it, certain scenarios that customers are going through that maybe the agent probably didn't, wasn't part of that 70% that it got.

Right. We built this feedback loops that customers can tell us, Hey, this wasn't exactly what we want it to be, or this wasn't accurate. And then we invest in correcting that via the feedback loops that we mentioned.

But, but yes, customers are seeing root cost analysis. We use it ourselves as a lot. So we're a strict dog footing company.

So all these agents are actively helping us maintain, run, and protect Datadog as a platform. Plus we also hear it from a, from our customers that it's working for them. Excellent.

You know, I, um, I, I did a story last week, or it was actually a video that today, in today's world, even though everybody's using ai, everybody's trying it, everybody's using it, everybody's experimenting. It has sort of a scarlet letter, right? Where people say, oh, well, well, AI, it's only 70%.

Yeah. Well, humans we're only 60%, so we got a 10% improvement. Right?

But that's why they don't look at it that way. Yeah. It's only 70% if I'm gonna trust ai, it's gotta be 99, you know, five nines.

Right. And, uh, you know, I think this too will pass. I, I think it's sort of an artificial bar we are creating.

I think part of it is because people are, are fear outta fear, right? Because are is it gonna make them, Jo, make them lose their jobs, change their jobs, what have you? Yeah.

But you know, it, it gets better. It's getting better every day. Now, you as a CISO and, and talking on behalf of other CISOs though, is this making your job easier, harder, little, you know, some of both.

What Do you see? I would say it's, it's both. Um, on, on one hand, it's making it harder because the surface, the attack surface just keeps changing and growing, right?

So, um, not just the, what people can do with ai, say shadow applications or shadow it as we called it, or the notebooks of the, of or open cloud of the world. Um, but I, I, I, which inherently are nice projects to work on, they're interesting, uh, but they do carry some risk. But I think, uh, it creates that surface.

Uh, but also, and also attackers using ai, right? So a lot of the things that we see are using AI to hijack things like open source dependencies, things like developer, um, the extensions that they use, the developers use in their, in their development client, the IDs. Um, but then if you use it for defending yourself, then, then in a way you're, you're say, like what I say is the tide, if the tide rises, the tide rises for everyone and everyone gets lifted.

Uh, so I'd say it's both, it, it creates opportunities, it creates challenges. Uh, but ironically, if you're not using AI to fight those challenges, then you're gonna be behind already, if not, um, uh, so I always encourage CSOs and security teams to think about what they can build with ai. So also using agents internally for security purposes, also using LLMs to understand, uh, malicious patterns or malicious behaviors are, are critical for us.

So couple of things that we've done is, um, we actually use LLMs to detect, uh, not vulnerable code, because I think there's a lot of that out there already. And the, and the models are getting better at writing proper code, but we we're using LLMs to under to detect malicious intended code. Um, so imagine a, a popular open source dependency package getting hijacked where the code that it gets introduced by attacker is perfect.

It adds no vulnerability, no scanner will find a vulnerability with it. But what it does is bad, right? And what it means to do is let me extract all the secrets that I can find from the developer's laptop and send it to, to a server that I own.

Um, those themes are, are, are happening now. And you have to use AI to be able to stay on top of that. The react to shell vulnerability, for example.

It's a perfect use case of using AI to find it and look at the repercussion that it had, right? Where a lot of servers were exposed. We not only see that was the initial entry point, but then attackers get clever and, and, and use stealth techniques like our research showed to then say, okay, well if I can penetrate a web server and then have a proxy every request to a server I own, then I can exfiltrate all kinds of data and credentials to me.

Um, so I, I think the attack patterns are changing because of AI and all. I mean, we still see the phishing, the identity based attacks and all of those, 'cause attackers are always gonna go for the path of least resistance. But AI now opened a bunch of paths for them, and I think we have to open paths for security teams to be able to use AI to defend themselves.

So it's, it's both. It is both. I, I tell you another thing, I said it on text Junk gang the other day, even at this early stage of AI use, and I think it's still early, we're already on this, and I've been in security 25 years myself, right?

We're already at the stage in security where we're going to need AI to fight ai. Mm-hmm. Right?

Uh, these scalability, you know, I mentioned before the 600 vulnerabilities that Opus found the amount of just the, just the sheer, the, the better phishing, right? The phishing and smishing is so much better because they're using AI to write these things, right? We, you, you need it.

You need to deploy AI to fight the AI because the bad guys are using it for sure. Yep. Yeah, yeah.

Exactly. Yeah. Yeah.

Yep. And at the end of the day, data is data, right? So these models are really good at finding patterns or, or missed patterns in the data.

Um, so while attackers are using AI to create data, whether it could be a video of me saying like, Hey, I need, I need, I need a thousand gift cards. Um, you can also u use AI to figure out like, Hey, that's not actually me in the video. So it's, uh, it's really fascinating.

Oh, that, yeah, that's, that's a whole nother point. Yes. The other thing is, you know, look, I think we had evolved to the point where observability and security were two sides of the same coin, right?

Right. I think with ai, they're both on the same side of the coin. That's how tight AI makes them very tightly intertwined.

And, um, and we're going to continue to see that, right? Because besides the effect AI is having on security, it's having an, an equal impact in the observability side of the house, correct? Yeah, that's right.

Yeah, that's absolutely what we're seeing. And, and to your point, observability and security being, um, both sides, two sides of the same coin, I think AI is, is both the heads and the tails of that coin, if that makes sense. Yeah.

Uh, Yeah. No, it, it, it's the, it becomes the coin that, that, but that's what's going on here, right? AI is becoming the focal point of, of all of these things.

And it's, it's, it's a little, you know, it, it's, it could be a little scary because like you said, it's only 70%, not a hundred percent, but it's better than what we had and it's faster than what we had. Yeah. But, you know, we're still going through this phase where we're getting, we're getting comfortable with that.

Mm-hmm. Yeah. I think it's very similar to the cloud adoption phase, right?

Like, I remember when the cloud was booming, um, a lot of the fear was where I lose control, where is the data gonna go? Costs what have you. Yep.

Um, and there were always nuances that maybe, okay, the first time you, you made a shift, was it lift and shift? You didn't get it a hundred percent right? Uh, but those that actually saw like, hey, there's actually something here in the future that we can continue to work on, um, are now some of the leading either cloud companies or companies that run out of the cloud.

Um, so it's, it's similar of that. It's like when the paradigm shifts, fear is part of it and uncertainty is part of it. Uh, that's why I don't subscribe to a lot of, of the, of the FUD that you see around their own.

And especially with ai, because yes, attackers are using ai, but defenders so should, so I'm glad that we're actually talking about defenders using AI more so than attackers using ai. 'cause the latter, the, the, the latter doesn't really give you a way out. It's just more of like, be careful.

Um, instead of, we should talk more about what people are using AI for to protect themselves. Agreed. Emilio Emilio.

Where, where can people get more information about the data Datadog security, well, Datadog security offering, but also kind of, you know, what, what's going on with Datadog security and AI and, and all of these, like how can they stay up to date in the know? Yeah, absolutely. Uh, so we have a couple.

So I will say the blog, it's, it's, uh, it's a, a, um, a good resource. We have engineering block, we have a security block, we have security block that's called Security Labs. That's actually where, where my group, um, when we track threat actors, when we track certain attack patterns, malware, what have you, like, we, we pose our research there.

Our engineering block is really good. Uh, Datadog is, is is very open in the sense of, we not only give you product updates, but we actually talk about things that we learn ourselves and how to run a platform of our scale mistakes that we've made, uh, new things that we've done. So, for example, we talk about outages that we've had in the past.

We talk about how we profile a certain service to gain performance improvements, how we scale systems, how we manage the data volume that we manage. And then on the security side, we talk about how we do things. That's at our scale as well.

So our blog post is great. Um, if you haven't heard of Datadog and Security in the same sentence, I don't blame you. We're pretty new in the, in the space.

But I will tell you that um, about 50% of our Fortune 100 customers use us, uh, on the security side. And about 68 or so, sorry, the Fortune 500, um, 58, 60 8% of Fortune 100 use us on the security side. And we continue to invest heavily in security.

'cause we Datadog started to remove the silo between dev and ops. And, and we think we can remove the silo of security and dev as well with, with, with our platform. Amen.

Amen. That's why I got into DevOps, you know, all those years ago. I thought that's what needed to be done coming from security.

Amelia, thanks for coming here on Tech Trunk tv, man. We appreciate it. Thank you.

Thank come back, keep us posted. Are you guys gonna be at RSA conference at all? We will, Yes.

We actually have quite a few things planned for r ours a so we will definitely have a presence. So hope to See you there. You know what we are, uh, we'll be there all week on Broadcast Alley and of course we act, we have an interesting thing going on Monday there.

You know, every year we put on our DevSecOps event. I think Datadog's been involved in it in years past, um, this year it's a little different. We didn't call it DevSecOps.

We, it's securing AI native dev 'cause everything's ai. Right? That's, that's Actually a good name.

Point it. So it's Securing AI native Dev. I'm actually doing a panel if you uh, if you're gonna be there, you know, we'll, we'll, we'll have your people talk to my people there.

You, I'd love to have you do it with me that Monday. I would love to. Yeah.

Sounds great. Alright. Emilio Escobar, CSO at Day Dark here on on Text, drug tv.

We're gonna take a break. We'll be back. Thank you Emilio.

Thanks for having me.