Incident Response for Developers with Tanya Janca | SecOps Vision 2024

Hi, I am Tanya Jenka, and today I wanna talk to you about instant response for software developers and DevOps folks too. Um, basically incident response for everyone. So I wanna talk about what it looks like when there's a security incident and what you need to do.

So there's a team, an incident response team at large organizations that respond to an emergency, basically, but they need your help. They need your help in like not messing things up in telling them when you see something that's weird, et cetera. And so I made this presentation for, uh, one company where I was giving them training for their software developers.

And it, it was something that I had already done whenever I worked with dev teams. And they said, Hey, can you do this for us? And I said, yes.

And from then on every single company said, you know what? We'd like our devs to know what to do too. And then before I knew it, someone said, have you thought about submitting that to conferences?

And it seems so obvious when someone suggests it, but basically I'm hoping today you're gonna learn five things. And, um, let's go. Okay, so first of all, what is a, what's an incident?

What's a security incident? So it's an organized approach to addressing and managing the aftermath or when it's happening, a security breach or it incident. So the goal is to handle the situation in a way that limits the damage, uh, and reduces the recovery time and the cost.

We also want to reduce any harm to our reputation. We wanna, you know, make our employees comfortable and feel safe. Again, our customers, we wanna reassure them, everything's gonna be okay.

And so it's how we handle a security situation when an emergency has happened. An event or an incident. So what's an event or what, okay, so, um, the lady on the left, she's having a security incident.

The lady on the right, she's having a security event. So in simple terms, an event is when something strange has happened or you suspect something's wrong, but you're not quite sure yet and you need to look into it. Whereas a security incident is when you are certain something bad is happening.

So an example would be if you find your sensitive data for sale on the dark web, that is definitely a security incident. And that has happened to me before. It wasn't sensitive data, it was unclassified data, but someone had taken it and they should not have been able to do that.

And so we knew for sure something very bad had happened. Um, I have a trigger warning in this talk, and I'm gonna tell you the slide where that happens. Uh, something bad happens to children and it's abstract in the idea, but I don't wanna upset anyone if you're a sensitive person, that's cool.

Uh, I don't wanna make you uncomfortable. So when we get to that side, I'm gonna tell you, and you might wanna mute me, I don't get graphic or detailed or anything, but I don't wanna make anyone uncomfortable. So I thought, I've never given a trigger warning before, but I had some people tell me it made them uncomfortable.

So I've added it. Uh, 'cause that's not what you're here for. You're here to learn.

So if you're a sensitive person, that's cool. And just maybe mute me for that like 20, 30 seconds. Okay?

So your organization needs to have its software secured. This is software you buy, software you lease, like software as a service. Um, software you administer, software you create from scratch, all of it.

Um, software developers are our first line of defense for our custom software, and our custom software tends to present the most risk in general. Uh, if you look at the Verizon breach report, we're almost always number one cause of data breaches, um, which is not like a crown. I want us to continue to win each year.

Um, but basically if we don't have the buy-in of the software developers, US security folks are lost. We can't, we can't succeed. And so if something happens, sometimes we're gonna need your help.

And so that's where these five things come in. And so security incidents. So we're gonna talk about your role during a security incident.

What need to know is I'm gonna tell a few stories to help illustrate some of the points. And I, I basically, I'm just hoping that you learn these five things. Okay?

So first of all, your, so your role during an incident, the stuff that our team really needs from you. So the first one is probably pretty obvious, but people do not do it very often. And that's, if you see something, please say something.

Um, so it's better to have you report something that turns out to be nothing than have something wrong happening for an extended period of time. I remember, um, I had a guy call me and he said, oh my gosh, like these discs with the sensitive data were stolen out of my locked office in my locked drawer. I'm freaking out.

And I was like, okay, you know, like let's walk through some steps. I'm like, go talk to your administrative assistant that sits outside, see if she saw anything. And he go like, he, he's like, okay, great.

I didn't see her when I came back. So he sees her and she's like, oh, hi. And he's like, do you know where the things are?

And she's like, I do. She's like, the official secure courier came. Well, you were at lunch.

And rather than writing you a note, I was like, I'll just wait until he gets back and I'll tell him that the official person we'd been waiting for to come take the sensitive information and physically manually transport it 'cause it was that sensitive, had showed up. And she's like, I didn't think I'd miss you 'cause I sit right outside your desk. But then I needed to pee.

And so it was just a misunderstanding, right? So it was, it was totally fine. And he's like, you must think I'm so dumb.

I'm like, no, you followed the policy perfectly, thank you. If you hadn't told me, then you go, you, you freak out, you do all these things. Every single minute counts.

And so you just followed the policy perfectly. Thank you, you so much. You're awesome.

Security teams aren't gonna get ticked off at you that you reported a thing and then it wasn't a thing. You're doing what we need, which is you to tell us something. So if you have, you know, a certain API that's always just falling down and crashing and you don't know why, or you receive this giant cloud bill you're not expecting, and you're just like, where did this come from?

Anything where you're just like, I've, I started to look into it and it just makes no sense. This is super weird. Ask the security team because we have different ways of looking at things.

We have a different tool set than you. And they're, they're like, sometimes it's really fast. Sometimes like two minutes.

I'm like, oh, it's this and it is bad. Or, oh, it's all of our tools say it's, it's cool. It, it must be something like that.

We don't always know the answer just to be clear, but please loop us in. If it's weird, it's out of character. You know, like the picture in the airport of the penguin wearing a Hawaiian shirt and carrying a suitcase at the airport.

And it's like, if you see something weird, say something. 'cause that's weird. Penguins don't usually wear clothing or carry suitcases.

Um, and so please say something if it's just weird and it, and you can't explain it. Okay? So next, please don't leave the premises without telling us if we've involved you in a security incident.

So let's say I tap you and I say like, hi, I'm Tanya. I'm from the incident response team. I heard you work on this app, this app's currently being attacked or this app, um, it looks like it's been attacked.

So, you know, like the bug bouncy program reported a bug and you fixed it. Good job, you rock. Um, but I looked through the logs and I, it looks like that was not the first person.

So I need your help investigating this. We're kind of worried someone's in there right now. Can you help?

Uh, hopefully. First of all, you say yes. Uh, second of all, so if you're a part of an active incident investigation, please don't leave without telling us.

Like whether you give us, you know, your home phone number or your cell phone, or if it's the nineties, you give us your pager number. Um, like whatever it is, like, give us a way to contact you. This might sound wild, but like I, I've had this happen.

Like I was working with the dev and we're going back and forth trying to fix this bug and it was actively being exploited live. So I was pretty stressed out. And then I said, okay, I gotta go just brief my boss.

I'll be back in like 15 minutes. I just have to tell him what's going on. And then my boss kept me longer 'cause another boss and then another boss.

And anyway, I ended up being like 45 minutes. I go back to his desk and he's gone. He took his bus home for the day and he wasn't on call.

They didn't have his phone number. There was no way to reach him. And so I couldn't push that code with the fix.

So I sat there and I, I finished the fix and I just, I couldn't do anything. And oh, hey, Kurt can't get packed. I was such a stress ball.

And so first thing in the morning I'm like, hi, I need you to, um, so please don't, don't leave or, or, or tell us you're leaving and like give us options. Like he could have granted me access to release code. Like there's so many things we could have done so he could have still got home on time to do whatever the important thing was that I'm sure he had to do.

But like, communicate with us and tell us like, I am leaving the premises if we're actively working with you. Um, so this one maybe should be number two instead. So if we come and ask for help with a security incident, it's an emergency security incidents.

Like some of them are cheap, they're only a couple thousand bucks, but some of them are millions of dollars. Um, I believe that average ransomware incident cost is over a million dollars at this point. I've worked on security incidents where it's like half a million bucks from a web app being attacked and, and I'm lucky that none were way, way, way above that, right?

And so I need you to treat this like the emergency it is. So if you have two bugs that are due by Friday, that's awesome. That's nice.

Normally I, you know, I wouldn't bother you in your work and let you decide your own work, but this is emergency for our org. So if your boss is like, oh, do this, ignore her, I'll talk to your boss for you or the security team will talk to your boss for you, what's really important is that like you, you drop everything else, you do the things that we need. And often it's like, grant me access so I can go look at this or run through this scenario with me, run through the code, step through with it me, okay, I see.

Now I know what's due. Like ideally we won't need a lot of time from you, but if we do need time from you, I need it to come first. If your boss gives you a hard time, the security team will talk to them.

And if necessary, like my boss's boss's boss will talk to your boss's boss's boss, you won't get in trouble. It's truly important. Okay?

Um, okay, so next I need you to follow something called need to know. So do not tell anyone anything about what you're doing unless they need to know. I don't mean want to know.

So you can tell your boss, I'm working on a security incident and they told me I'm not allowed giving details to anyone, so you're my boss. And they said, if you need to know exactly exactly what I'm doing to talk to them, but I'm not allowed giving you details. Like just one of our apps got hacked or this app has been attacked and I'm working on it, I'm working hard on it.

And you know, like you can say maybe I expect to work on it 45 minutes, or I expect to work on it the rest of the day, or, you know, they're gonna brief me and drag me into a meeting later. But the exact details of the security incident, do not tell them. I've had many a managers say, well, I have the right to know.

I'm like, you literally don't that you are curious and that's cool, but like, then you tell someone, then you tell someone else, then you tell someone else and then all of a sudden everyone knows. And I, I'm gonna give you like a really dumb example where, uh, oh my gosh, I have so many examples. Um, one example where, um, actually I'm gonna save this for the next one because, uh, I, I think it'll go better.

But a few times, not very many times I've had someone not follow me to know and then a person told a person who told a person. And sometimes that can result in an insider threat being alerted as to what's going on and they leave the building and they run away and then we can't press charges. Sometimes it leads to a giant rumor mill that scares the employees.

A lot of the stuff that security people work with, especially incident responders, like some of it's really quite terrifying. Um, like I've done counter-terrorism work, I've done like heavier types of work at times and I forget how heavy it is sometimes. And like if I share like a high level of like, oh, years ago, like this thing happened.

So like the average person, it can be quite upsetting for them 'cause they don't realize how many times people like us saved the day and how much danger all of us would be in if we weren't doing our jobs. So please don't scare other people. Please don't essentially spread rumors.

Like do not tell anyone unless they absolutely have a need to know. If you don't know for sure if they need versus want to know, ask the incident responder who's managing the incident. Um, so quite often where I've worked, it's me, um, or, uh, previously when I was learning is my friend Eric.

Um, and so they'll brief people. That's their job to communicate on behalf of the team and make sure everyone knows what they need to know. And so just say like, my boss is asking, can you brief them on what you're allowed to brief so that they stop bugging me every four minutes because I need you to concentrate on the task I gave you.

Okay? So number five, please do not try to manage it yourself and be a hero. And so this is the trigger warning.

So you might wanna skip this slide. So you mute me now if you're a very sensitive person. And just be clear, I'm not casting judgment if you're a sensitive person.

Like I'm glad it's not a bad thing. Um, and then next slide, unmute me. So, um, one of the places I worked, uh, my like, uh, my mentor who taught me how to manage incidents, his previous workplace, one of the help desk guys had gotten a security incident and he's like, oh, I'll help out.

I'll do the incident myself. I know what I'm doing. I'm really smart.

So he started investigating. He didn't have special tools, he didn't have any training, he didn't know what he was doing. And he stumbled across images of child abuse.

And I assume you know what I mean when I say that. And um, what he ended up doing was ruining something called the chain of custody. So it no longer counted as evidence towards a criminal act.

We couldn't use it in a court of law. And so as a result, this person who did horrible stuff, who should be in jail kind of forever, that person got to go free. And on top of it, this employee, this poor help desk guy saw things he can never unsee and he has this huge burden he's carrying, like for guilt, for letting that person go.

And then awfulness of the things he saw that he can never perge from his memory. And my colleague and and mentor basically told me like, yeah, when I left years later, he was still in therapy all the time. He was like really, really upset with himself.

And so it is not your job to be a hero. It's not your job to carry this burden. It's, it's our job.

We have training. We know like, so like I didn't even, I did incident response just for AppSec incidents. And so that's not the type of incident I would even have responded to.

It's just software, um, which to point blank in my opinion is way less scary and softer. And so, um, I'm gonna tell you another story to like lighten things up a bit of another time. A help desk person thought they were being a hero.

Um, so this was another place that I worked. And so I'm Canadian and you've probably heard my accent. And at some point I said, instead of about, which I deny vehemently just to be clear.

Um, but I'm Canadian and in Canada there's certain stereotypes that are just true. And so I, I go to work one day I was at the dentist, um, I pride myself on like having good oral hygiene, yay. Me.

I was actually at the dentist this morning as well. Um, but so I had told everyone I'm gonna come in late. And, um, I was a CISO then.

And uh, I come in at 10 30 as planned and my team's like, there's a big security incident, go to the big boardroom. And I was like, I look and my whole team's there and I'm like, there's a security incident and you're not there. They're like, help desk is managing it.

I'm like, wait, what? And they're like, yeah, help desk is there with all the executives just go, go. So I walk in and all the executives are talking to this guy from help desk and he's telling us, so we have a headquarter building and then we have like a normal building, um, like that's small that's aside that maybe like 10, 15% of our staff use that's like way in the east end.

And so some people work from there, depending upon their job or if it's like a better commute or whatever. And that's cool, right? So apparently that building, which is not a smart building, it's not internet connected, it's a normal building made of concrete, that building according to this help desk guy was infected with malware and they should evacuate and it's scary and they're not safe.

And I was like, oh, hi, uh, I'm here now. And they're like, where were you? I'm like, the, the dentist.

And and they're like, we needed you and you weren't here. I'm like, I, I literally, I have on call, like I have a phone. No, no one, no one called.

And they're like, well, he's helping us. He's taking care of us. 'cause you were away.

I'm like, no, but you didn't even gimme a chance to respond. You didn't, you didn't call me yet. How could I know?

And like, it's in my calendar where what? I'm like, okay, put that down, what's happening? And he's like, I'm saving the day because this building's infected with malware.

I'm like, no, no. That's a dumb building it. You concrete doesn't get malware.

So no. And they're like, Tanya, you don't know. I'm like, no, I, I do.

And anyway, so I try to calm them down and, and he's like whipping them into a frenzy. 'cause part of incident management is you chilling everyone out, reassuring them they're safe, they're in good hands and you've got this. He was like, ah, they're like so upset.

And I was like, okay. So as a reminder, follow need to know, don't tell. So they told everyone, okay, so I leave the room, I explain what's happening.

We're like, yeah, that building can't have malware. We look at the network traffic. Okay, so Canadian, um, stereotype.

Uh, so what happened was is the Winter Olympics were happening and figure skating was on. Okay? So we love figure skating.

That's like a thing. All of us wanna see it all love figure skating. Um, and like this is normal that lots of us will wanna watch it.

And so what every office ever usually does is they say this giant boardroom's gonna be showing the figure skating. So it's skating, figure skating, basically like the Winter Olympics. Canadians totally love that, especially figure skating.

So we're like, it's gonna be in this boardroom. If you're gonna watch, go there, do not stream from your desk because if everyone streams from their desk, the internet disappears, right? Every single person can't stream all day every day, especially years ago when internet bandwidth width costs more.

And so, um, apparently some bigwig decided to go against this policy that we've had for years and was like, if you wanna watch the Olympics, you can take a vacation day. You know, if you're at work, you should work. I don't care if we're like about to win the gold tough, no one gets to watch it.

So all of them, like almost every single desk in the entire building, we're streaming the figure skating. And in four seconds we could see that we're like, oh, they're all going to this website. What's on this?

Oh, figure skating. Yeah, that tracks. And so then I go and I go to brief everyone and everyone's like, yeah, but what about the malware?

I'm like, there, there was no malware. This is how, or and they're just like, will that help desk guy told us what was going on? So eventually I managed to get the bosses calm, but the bosses had all told their admins, their admins had told everyone for months I heard about how that building had malware, how the security team had failed.

It was such a disaster. And like my team figured it out. I think it was under 10 minutes we figured out what was going on with our special tools because we know what we're doing and because we've seen stupid things like this before, right?

And like, ah, so please don't try to be a hero and manage it yourself because you undermine what we're doing. You don't know the best way to respond because you don't have training. You don't have the best tools to respond.

So you have an incomplete tool set and it is not your burden to bear the hard parts of our job. We agreed to that when I did counter-terrorism, although that wasn't originally what my job was supposed to be like. I knew well, I thought I knew what I was going into, right?

And so, um, that's not your job. You shouldn't have to deal with that. It's just, it's it's not even fair to ask you to do that.

And so please don't try to manage your yourself and be a hero. And with that, these are the five things. So tell the security team.

If you say, if you see something, if it's weird and you can't explain it, tell us. Don't leave the promises. If you're involved in an incident without telling us, tell us I'm leaving, here's a number to reach me.

Or do you have everything you need? 'cause I gotta go, this is top priority. Please treat it like the emergency.

It is follow need to know and not tell everyone all these details that are supposed to be a secret for many reasons. And one of it is to help everyone remain calm and not be afraid. And the last one is, don't try to manage it yourself and be a hero.

It, it doesn't solve anything for anyone. And with that, um, I want to encourage you to talk to the application security team. If you don't know, ask like a ask them what to do it.

And if you don't get an answer there, um, you can ask in the We Hack Purple Community, an online community that I run. I'm also starting to run the EM wrap online community. Um, 'cause they bought my little teeny tiny company.

And so the We Hack Purple Community is flowing, uh, or about to start to flow into the EM wrap community. And I'm gonna give you some links of how to join there. But if you can't get an answer to what you need at work ask in the community, you'd be surprised the answers you get, it is your security, your security team's job to enable you to do your job securely.

And so tell them what you need. If you're not sure, ask them. Uh, if you're like, this is super weird, can you come take a look?

Like, I realize it might seem like they're busy and don't have time for you, but I assure you they want to know these things. And with that, I'm gonna give you a couple of resources of where you can learn more stuff. So the first one, why is the button not working?

There we go. Um, the first one is a bunch of books. So I don't feel we can do security, right?

If we're not doing it right. And to me, software development is done best, but it's DevOps. I love DevOps.

I'm very biased. Um, and so the first four books are all about DevOps. The last book, Alison Bob Learn Application Security is my first book.

Uh, and if you are interested in learning more about the security of software, I feel like it's a great place to start. Me and my mom agree, it's pretty good. Um, oh wait, no wrong button.

There we go. Um, every Monday on Twitter, I use this hashtag Cyber Mentoring Monday since 2018, it's been a while. And I use it on Twitter, I use it on Blue Sky and I use it on Mastodon, on the InfoSec Exchange server.

And I, every Monday I like post a message using this hashtag and I try to help people connect. So I'm a terrible matchmaker. However, I'm really good at connecting people and helping them find each other and choose each other.

This has been used to help lots of people find a mentor. So if you wanna switch into InfoSec or switch from one area to another of InfoSec or you're just interested in learning way more and you're super curious, this is a really good way for you to meet people and learn. If you already work in information security, yes, maybe it's time for you to take someone under your wing.

I feel like a lot of people think they don't know very much, but they actually know a ton. And there's lots of people who wish they were where you are at in your career and they can't get there without people helping steer them. Or it'll take a really long time for them to get there.

And so mentoring someone point blank is extremely rewarding as an experience. It, it's so amazing to watch them soar and see them just be amazing and wonderful. And so, um, cyber Mentoring Monday can help you, um, join the SRE community.

So I am currently like trying to migrate the two communities together. Um, and so I was like, so we're in the middle of this giant project, but basically please consider joining the SRE community. And like, if you wanna take a screenshot, you could go to any of these.

So we have a newsletter where we invite you to lots of free events including training. Um, we have like a Slack channel where basically we just talk about like, we just help everyone with whatever they need for Sun grab. Um, and then we're building a community platform as we speak.

Um, that'll take a couple more weeks to get ready. Um, and then also of course, you may try our product. Um, and then there's the We Hack Purple Community, which is still live in Hopin.

Uh, and we haven't started the migration off of there onto the new one yet, but there's still new people joining literally every single day. Um, we have free courses, free events, like articles, conversations, and like memes. Um, and so please feel free to join us if you wanna learn more about AppSec.

And the last resource I would like to give you is me. Um, I do stuff like this all the time. I have a blog, a YouTube, a newsletter.

I, I'm, she acts purple on like basically every platform. Um, so if you are like, oh, I liked this. She seemed really not terrible.

Um, there's a lot more for you if you want it. Uh, and with that, I would like to thank you all so much for your time and attention today. Thank you to this amazing conference for having me and for Techron for organizing this giant amazing conference.

And, um, yeah, that's it. I hope that, I hope, I hope that if you see something, you say something to the security team and you help prevent a big emergency. I'm Tanya Jenka and I'll see you next time.