Jennifer Czaplewski – DevSecOps at Target

Hi everyone. I am Jennifer chiplesky and I am a senior director at Target. I have two areas of responsibility there.

I have endpoint security and product security but this is Dev secops connect. So we're going to go deeper on product security and I'm going to share Solutions actual Solutions. I've been to a number of talks over the last few days at RSA and over the last few years in general and sometimes I leave those talks thinking I agree.

We have Monumental problems. There are so many problems to solve and I want to know what am I supposed to do about them what have other people tried? So I'm going to share with you some things that we've tried.

I'm going to tell you why they work for us and maybe you can decide if some or all of those might work for you. So that's what we're gonna do today. It's gonna look like this.

I'll give you a little overview of product security. I'm going to talk about three things product intelligence security ninjas and something we call the right test at the right time and then a little preview into what we think is coming next. I'm gonna assume that most people here know what a Target is and that you've shopped it one hopefully recently, but you might not know that much about targets technology landscape.

We are actually a technology company. We have 7,000 applications. We have a scalable infrastructure using both private cloud and public Cloud.

We have invested heavily, especially over the last seven or eight years in building teams that are technology teams in-house engineering in-house security expertise and to support all of those applications so that we can continue to offer all of the amazing features that you probably engage with when you shop at a Target store. So let's talk a little bit about product security at Target and that word can be used in different ways and different companies and in different settings. So I always like to draw a little border around.

What does that mean for us? And the product security concept at Target is really grounded in two very important objectives and they are to enable the secure delivery of software at scale and all of the words in that statement are important. We are here to enable Our development Partners to build secure applications at the scale of Target a little louder now.

Thank you. So all of those words are important. But it is not just a technology initiative.

We are also equally focused on integrating security into technology culture. We can offer the best security tools in the world, but if teams don't have a culture of security tools won't help them. So we have two important goals six capabilities code analysis.

runtime analysis business logic analysis distributed security Advocates security health and awareness and knowledge and those six capabilities are supported by more than a dozen different services. And one of the more important things that we lead with whenever we're talking with all of you or with our development Partners or even in prioritization sessions or when we're trying to figure out what the next most important step will be is we fundamentally believe the developers are here and trying to build secure systems. So our goal is to support them in doing that.

So we want to meet developers where they work. Larry talked about this a little bit. We don't want developers and Engineers to stop doing development and Engineering to do some security tasks.

We want to wherever possible embed that security work into their day-to-day life cycle. We want to offer end-to-end Solutions not just finding security flaws, but being Partners in helping to resolve those security flaws, and lastly. We want the right way the most secure way of doing something to be the easiest way for that job to be done.

So now I'm going to start talking about those three solutions that have really helped us make a dent in the product security journey. And the first one is called Product intelligence product intelligence is a system that we have used that helps create transparency and accountability with our development partners. And the first thing you might notice about the product intelligence system is that it is actually a score.

This is a fake product. It has an 808. This score is a weighted algorithm.

We've developed it in-house and it measures a whole bunch of objective security criteria that would assess the security of all the applications that would make up a single product. So a product at Target is like an end to end business process. There's probably two to 300 business products.

7,000 plus applications. And so we measure characteristics of those applications and collect those into a score the score is like a personal credit score. So it ranges from 300 to 850.

It's a little bit familiar and it's four different categories the first category findings and endpoint vulnerabilities and we don't measure how many findings you have. We don't measure how many vulnerabilities we have for the findings and vulnerabilities that you have what percentage of those are closed on time 40% of the score? The next part of the score equally valued at 40% is are you using the right Security Services?

Are you pen testing on time? Are you using services like static scanning Dynamic scanning software composition analysis. So those two elements use of security services and findings are the bulk of the score.

We also have 15% for security culture and I'll go into a little bit more detail on that next and last is have you been the cause of a security event. If so five points or a 5% you can see the score and it's a nice point of Relativity. You can also see your score compared to your score in the past.

But if we only gave teams a score we're not really helping them solve security problems. We're just measuring them. So this to-do section is the most important part of the entire system.

It tells them all the actions. They should take in priority order to improve their score. Thereby improving the security of their applications.

So as Mark said I shared a little bit about this in 2019. But I'm going to share a little bit about what's different. So the first thing that's different is we are using a bright and shiny new UI that we've been developing over the last year are developers love it.

It's way easier to use it's way easier to navigate and we can embed more and more Security Services into this as a central hub. The other thing that's different is what's missing so when I was here in 2019, we told stories about you could see your score compared to scores of others and say if you're in the top 10th percentile or bottom 20th percentile and it drove a lot of friendly competition, but our teams got so good at understanding how to secure their applications more than 10% of all teams had a perfect score. So if you had a perfect score, you might not even crack the top 10 percent and people were frustrated it drove the wrong Behavior.

We are also asking teams to meet a minimum score for 2022 that minimum score is 750, but you could have well above a 750 and still be in the bottom 10th percentile and we want to make sure that we're not driving teams to Perfection. We're not looking for perfect scores. We're looking for good security health So in the system, it's unlike a credit score because it's completely transparent.

You can see everything that makes up the score. You can see all of the findings. You can change the headings.

You can click the three dots and put those findings in your teams backlog. You can download it to excel so we show teams all of this information for the findings that they have. There's a similar view that shows them all of the applications and the different security health status of those applications as well.

So let's talk a little bit about security culture. It measures if you have a security ninja and I'll tell you more about that program next but it also has a category that we call Pi fives. We think it's funny you can laugh with me or not.

But Pi fives is like a high five from the pi team and it measures the culture of the team the objective measurable behaviors that team members on this particular team would be taking to show that they have a culture of security things. Like how many people reported the most recent fishing simulation how many people from the team took the required application security annual training did anyone on your team find the Easter egg on the product security page which shows us that, you know a little bit about how to navigate the documentation. So Pi fives is 15% of the security culture score and then like I said 5% for a few caused a security event.

Okay. Let's talk more about security culture in the form of our security ninja program. Lots of folks have a security Champions program distributed security Advocates.

Our program is not a training and awareness program. Like a lot of the organizations that you will see our program is distributed folks who are actual extensions of the security team embedded in product teams. They aren't security people per say they are developers who are considered the elite team member on that particular team that is influential hopefully passionate or enthusiastic about security, but trusted advisors, they get a little bit of security training and then they are empowered to make security decisions and really help be that Ambassador for security culture on a team.

And what the security ninja program helped us do is address the historic relationship between our security team and our development team. It's not a contentious relationship, but it was almost like we spoke a different language. And so these security ninjas really helped to become the translator.

They helped their development Partners better understand that security is not just a gate that you have to cross at some point in the delivery life cycle and they help the security team better understand the developer experience things that we were offering that worked. Well things that we were offering that needed some work and so that's really become a Cornerstone of the success of our entire product security program. The program itself is meant to be exclusive as I mentioned.

We are saying no more than 5% of the technology team that we have about 5,000 developers total. So we have about 200 security ninjas today. No more than 5% We want it to be a destination something that people aspire to in their career Journey.

The program itself is tiered. You join as a white belt, but you can be a purple belt or a black belt over time. What does it mean if you're a security Ninja?

Six things and this is something different than when I was here. In 2019 or even since if you've heard any of my talks about our security ninja program, we used to have four responsibilities. If you are a security ninja and they were mostly centered around things.

We wanted you to know. To help create a culture of security on your team, but we've learned over the last three or four years. We really want to Pivot to what we should be doing to build a secure culture on a team not what you should know that hopefully then turns into something.

So yes, if you're a security ninja the first responsibility, please at least have some basic security knowledge. We want you to build and maintain that knowledge, but the other five things are actions identify themselves security problems and threat modeling is where this is where our program has also matured over the last few years. It's been a hard not for us to crack in terms of throughout modeling at scale.

It's a Time intensive process when done well, and so we're now leveraging our security ninjas. We ask them as part of their responsibilities to do two to three threat models per year. They share those threat models back not only with their teams not only with our cyber threat intelligence team, but also the leadership of the teams that they're on and it further creates that sense that they are trusted advisors influential people who are absolutely good advocates for security on their team.

We want them to promote security best practice within their teams achieve security goals a lot of times. This is pi score, but it can be other things teams have create security goals and lots of different ways. We want them to engage with the security ninja Community for us the 200 folks that are Target team members who are part of the security ninja Community.

It's a really active group. They have a really active slack Channel. They collaborate to solve problems.

We have meetings and meet ups and lots of different things. And so we want them to continue to participate there. And then lastly they are a very important important voice of customer for the security team.

They tell us what's working and they tell us what's not working. Okay. The last thing I'll talk about is what we're calling the right test at the right time and we can talk about that in terms of security tests that are automated or manual and this is a lot of what I've heard in panelists earlier today and in the talk right before this one talking about how we embed security things right into the developer life cycle.

So before I get in let me just ground you on what's probably a pretty developer pretty common developer lifecycle for all of you. But if I'm a developer at Target local code changes, hopefully with peer reviews, I upload that to GitHub build processes begin code is in production. What we are doing is introducing security testing into that flow.

The first one is an oasp project an open source product called sedated that looks for secrets in code before it gets to GitHub. And then once in GitHub two flow spin up simultaneously static scanning looks for code flaws and code that we've written in-house or first party code and software composition analysis looks for vulnerabilities and open source libraries. So let's talk a little bit more about secrets and code with sedated as I mentioned.

It is open source. And we have this in place because it is really not uncommon for a developer to accidentally upload secrets to code. But once that has happened we consider that secret to be exposed passwords need to be rotated anything up other applications that are associated with that account need to be updated redeployed.

It takes time. And so we have embedded into the flow before the code gets to get Hub a check sedated is really simple. It's a series of regexes that we've customized for our needs somewhat, but if it looks like there's a secret in code the developer gets feedback that says stop looks like you might have some Secrets or other passwords or other secrets in this code.

Are you sure you want to proceed? This is as close as we get to blocking the build at Target that we have because the developer can say this is a false positive. They can change their code to indicate that it's a false positive and proceed and the reason that we do that is two things one.

We believe that developers want to build secure code. And so we're going to trust them when they say it's a false positive we get an alert if that happens and so we can verify after the fact but secondly we are not staffed 24 by 7 if Friday at 6 o'clock, somebody wants to check in their code from the day and they can't because sedated has told them that they might have a secret. We don't want them to wait until Monday morning so they can proceed it's a speed bump more than a block but this really allows them to make sure that they're saving time.

So this has been a change for us that's been really successful and we know that for a couple of reasons first developers like it. We would know if they didn't like it because we'd hear a lot about it. But to the contrary we actually hear really positive things about the amount of time that we have saved them with Secrets rotation.

But secondly, our pen testers are finding fewer secrets and code since we've implemented this change so we know that we're actually reducing security vulnerabilities. As I mentioned we have sast static application security testing embedded in the flow as well as software composition analysis behind the scenes. These are vendor tools our developers don't know or need to know that to them.

They are just following their business flows. When or if something is found they get a notification in whatever way they choose GitHub issues jira tickets email or a combination of all of those things and they also know that once they close that issue the tickets are closed automatically for them. So we're really following that value of meeting teams where they work.

In the last area that I'll talk about is penetration testing. So remember this category is called the right test at the right time and I think a lot of people agree that penetration testing is a really good way of security testing, but it's also really manual and time intensive and so we have spent the last few years thinking really hard about. How do we make sure that we're testing the right things and using that expensive time as best as possible.

And the first change that we made is to be a lot more intentional about comprehensive tests. So rather than testing individual components whenever a team feels like they want to submit them to us. We will take a more thoughtful look and try to test entire business flows and what we've learned through doing that process is we find more of those design flaws and more systemic type issues that you wouldn't just find in a scanner.

But the other thing that we're learning is that they're really a lot more fun for the pen testers as well. And so I'm going to get into pentester engagement in my third point. So first of all, the comprehensive testing is good, we find interesting things and it keeps pen testers engaged but we're also leveraging our pen testers to help decide where to test when I first joined the leadership team of our pen testing function a few years ago.

One of our tests are said, I like being a pen tester, but honestly as soon as I'm done with a test two more have taken its place it's a never-ending pile of work and it can be demotivating and so we have given the pen testers a little bit more autonomy. They've learned parts of the business. We assign each of our lead pen testers.

Different area of business they met with development teams. They met with some of the leadership and they used their security expertise along with their interests to say, let's follow the risk not test things just because the right amount of time has elapsed but they're testing things that bring the most risk, they're testing things that have the most amount of change and so the result of both these comprehensive tests and those risk-based tests is that the engagement level from our pen testing team has gone up pretty substantially it allows them to develop new skills and have a little bit of autonomy over where they're spending their time. So we are not done for as far as we've come over the last six or seven years.

There's still much that we want to do. We want to continue to be data-driven. We want to continue to get even more efficiency gains.

And so we've got a lot of investment that we'll be doing in that space. Secondly absolutely investing in Team. All of this is led by really skilled engineers and security analysts.

And so we continue to invest in our team and the one plug that I'll make is that if anything that I've said today sounds a little bit interesting we are always hiring engineers and security analysts. And the third thing that we want to do is to continue to integrate all of the different services and offerings that we have our developers do not care and do not want to know how we're organized. They just think of security as one big group of folks.

And so we want to make sure that as much as possible anything that's done from a security perspective is really easy for them to do for example, you can get all of your endpoint vulnerabilities and that product intelligence system. But if you make some patching updates or Os changes and you want to see if that addressed any vulnerabilities, you have to leave the system and kick off those rescans somewhere else. We want to get rid of all of that and really make it a seamless easy to understand experience for anybody who's a developer at Target.

So I knew that I had a short amount of time and I tried to cram a lot into it. And so I think we have some time for questions if anyone has any. I wonder if there are a look at this driven development principles for security.

You know on the teams that I lead, we certainly think about unit testing and things like test coverage our personal team. We're not a big tdd or test-driven development team. I think it's more of the culture of the each individual team.

So from a security perspective, we offer ideas and ways that you would test for security but we haven't rolled out on the teams that I lead any sort of broad like tdd philosophy. We're we're developing just in a different way than that. Hi, I thought it was really interesting.

You talked about the developers can choose how they receive those messages. Is that something that you built within Target or is this thing you ask your vendors to to build for you for the most part we do use vendor tools behind the scenes a lot of the engines that kind of look for static vulnerabilities and software composition analysis stuff. We don't engineer that but we anything that the developer interacts with we have built so we use jira in-house we use GitHub and house certainly email and so those are the three most common ways that people will ask for feedback.

And so we've built most of that in-house So it's as Mark was saying it's love I love seeing how you program has evolved in since I first saw you speak here in 2019 it the one area. I thought jumped out the most for me today was your ninja program shifted from being mostly training to being more about actions that they could take. So do you have any guidance for what actions?

I mean, how do you measure? Yeah. I saw the list of the five things.

But how do you how do they know which actions sort of sort of satisfied that Um, so the program has always been meant to be security advocation and not necessarily just knowledge, but I think we were overvaluing knowledge over action. And so we do have almost like a Playbook when you become a security Ninja at Target you get a 90 minute course and security fundamentals and probably 40 minutes of that is what does it mean to be a ninja and so we talk about threat modeling we talk about how do I help my team better understand their vulnerabilities? And so we have sort of outlines of things.

There's a document that if you were a ninja it's like a two-page thing that talks about things you can do but that's where that Community comes in someone will pose a question and the slack room. This happened earlier today. I was looking at our security and just lack Channel somebody post a question about some kind of vulnerability.

There are 31 responses in a few minutes. And so everybody's like this is how I've solved it and so the community really helps people figure out knowing is fine. I think we're at RSA a lot of people know things here but doing is what we really want to see some value and that's where we've really third the security ninja program probably over the last two years Can you provide some more details on the automation of the manual penetration testing like which use cases good were automated how much time was spent in the automation efforts for that?

Sure. So our pen testers usually start with automation testing. They run a lot of the things using burp and other things that pen testers would normally do the real shift that we made is That's supposed to be just like kind of like day one of a pen test and you kind of crank through that automation stuff where the reason that we have an in-house pen.

Testing team is business knowledge history with the product knowing how something works and so we really leaned into that in a way that we hadn't necessarily in the Years prior. So it was less about more Automation and more about business thinking and putting a lot of different apis together to test an entire business flow rather than automated testing on a series of apis. Loved your presentation Jennifer.

Thank you. One question about product intelligence when you think about the roadmap and what you guys are planning for the next year, too. I was wondering if you could share anything with us that you are excited about that.

You know, we'll take it to the next level. Yes, absolutely. And I I just love it.

I am so proud. I hope that my pride for what our team has built. I'm the representative up here that gets to talk about it, but there are teams of people who build these things.

So I'm so proud of them and what we've built so we're really excited about the UI what we were using before was an open source product called super set and it served us well for as long as it could but it was a little bit clunky for what we were trying to do. So, I think what we're really excited about is making it more of a hub and not just a reporting mechanism where you would go get an update but it's a hub and it's where you can launch a lot of different security actions. That's what we're really excited about doing.

We're going to continue to bring in the right data points and kind of thinking about that over the next few years as well. Go so ditto on a great presentation. That was really I really enjoyed it over the last couple years.

My focus has been on the efficacy and toil related to audit. And so what audit okay risk all that stuff that sort of typically everywhere I've ever been is high tile very low efficacy, right very subjective in nature. The change record is a bunch of subject to discussions.

So I was just wondering what your relationship is. I see a lot of these like great tooling and devsecopting and it's great stuff and I like the ninja program, but I don't see any emphasis on how are we creating immutable digital at the stations to create a higher efficacy in in the audit process internally risk and externally sure. So like any organization, we're subject to multiple, you know regulatory bodies.

We have found so let me I'll paint a little picture of what it was like managing findings and vulnerabilities before the system and why our Auditors really like it. I wouldn't say that we've solved all audit problems, but we've really done a good job on some of those audit inefficiencies through the product intelligence system. We used to have an entire team of people whose job it was to pull the findings out of our GRC system email those findings out follow up with people on dates.

And so that process is now owned an accountable and transparent to the product teams. So in a lot of ways, this has made it much more efficient for us to satisfy audit requirements as well as security requirements. But that's a small subject, right?

Sure. Just hygiene test driven development, you know how you do branching stride all those things that literally show up variants of it in nist and PSI. It says, yeah.

So yeah, so that's another thing. I have a pet fee about not you but the sort of drowning in on just vulnerabilities because there's so many it's it's much more dangerous world than just Library dependencies. So I would say that our systems have solved some of our problems with efficiencies not all of them.

We are not living in an audit free land now where when an audit starts we just snap our fingers and things are done but a lot of the data is more Consolidated which makes getting through that audit process easier for everyone involved. Hi wonderful presentation. I have a question about infusing Security in the development side cycle, right?

So you have to get the buy-in and everything in a life cycle of a product. You know, you start you have fixed number of days. Now bringing in security.

It increases the timeline for a development cycle. What are the challenges or how do you? Address that so that the management buy-in development team has to buy in the test QA team has to buy in that.

Okay, you cannot deliver the product in like a month. It will be six weeks. Well what what tools or Dynamics would you use for that?

I think it's a combination. So the question the whale answer it is you know, how do we start thinking differently about security and recognize that if you wanted to build something. Well you want to build Security in from the beginning and not just try to get through at the end and check some boxes.

And so that's where I think the security ninjas and the fact that we've chosen hopefully The most influential security knowledgeable folks on the team to influence right from the beginning. They're in those design sessions. They're throughout modeling or hopefully at least thinking about the threats to the system as they're building it we're here to enable and so sometimes you know, if a team is building something as a proof of concept, then they can kind of circle back and start including all of those things from a security perspective and asking themselves those questions, but we have been successful in our cultural transformation.

Because it's really not a question for us anymore of like well now we got to go back and build it securely we're really encouraging teams to think about that from the beginning. Even if you're part of a hackathon we have hackathons. You're gonna get security feedback.

As soon as you check in your first batch of code static scanning finding software composition analysis vulnerability. So hopefully that gets teams thinking right at the outset how they want to make sure that they're building the system with security in mind. Thank you.