Establish and Mature Threat Modeling Practices at SAP Customer Experience | DevOps Connect: DevSecOps 2023

Hello everyone. I welcome you all to my talk for today, establish and mature threat modeling practices at s a P customer experience. In this talk, I would mainly highlight what is the kind of threat modeling culture we have established over the years at S SAP and SAP P customer experience.

I would also like to share my learnings and experiences from my journey of doing numerous threat modelings over these years. And last, but not the least, you'll also get to hear some best practices for effective threat models, which you can take back and implement in your organizations. And before we, uh, del into the slides, I would, uh, like to briefly, uh, introduce myself.

I started my cybersecurity career in the IT industry around 15 years back. And since then, I have had the experience of working with various security teams in different roles. And from, uh, from the past three years, I'm working with s a p customer experience as a security expert providing consulting to developers on all aspects of secure software development and operations lifecycle.

With that, I mean, uh, we provide the security awareness trainings. We conduct threat modeling workshops. We support them with the security scanning tools, open source scan tools, how to, uh, you know, evaluate the findings and so on and so forth.

And also support with incident response in the monitoring phase. With that, I'm really excited to share my ideas, learnings, best practices, and I'm here to, you know, uh, share my, share my learnings with all of you. Now, let's get started.

So what exactly do we mean by a threat modeling? What do we mean when we say we are threat modeling? Right?

You, when you search on the internet, you can get tons of definitions, uh, around threat modeling. Very heavy definitions, yeah. But in the simplest language, it is essentially what could possibly go wrong, isn't it?

And it always need not also focus on security. Everyone is threat modeling. These days when you lock your house and go for vacation, you have that in your back of mind.

Like, you know what possibly could go wrong, right? And when you're sharing information with somebody over the internet or even p physically, you also think about could this be misused? Um, you know, is it the right person to share this information with?

Or even when you are boarding your kid to, let's say, school bus to go to the school, right? So we are threat modeling in everyday scenarios. Now, in the context of DevOps and dev DevSecOps, what here we are today for threat modeling, a software or even software supply chain can get very intensive and intimidating in the DevSecOps or in the DevOps world.

Threat modeling is an iden, uh, is an activity to identify design flaws in the architecture during the initial phase of development for a specific feature or even the complete product. You can think of it as a blueprint eraser, that is, you can make changes in the blueprint before actually constructing the building. Now, the ways to thread model are evolving as we speak, right?

With the changes in the world with technology, uh, with the, you know, transition to cloud, with the evolution of generative AI these days with internet of things, cryptocurrency and whatnot, right? The tax surface is expanding day by day, day by day. And at the same time, the need to build a secure software without slowing down the development is also growing day by day.

From the pipeline perspective, threat modeling is started across the far left, or let's say in the planning phase, but it has influenced over every stage of the pipeline. Like, for example, the threats or risk that we identify form the baseline to develop security controls during the coding phase. They could also be used to provide inputs to the security testing team to develop test cases.

Then, uh, let's say in the monitoring phase, they could also be used to identify the focus areas where we need to, you know, uh, focus more on the monitoring. To that end, let me, uh, you know, uh, let me also, um, explain what are the different, uh, you know, pillars to do with threat modeling. Yeah.

So whenever you are talking about a threat modeling, you need to think from the attacker perspective, from the architecture, and also from the assets perspective. Now, with that, the three pillars that I would like to show you today is, you know, you need to have the architecture knowledge first. You need to actually understand the underlying technologies.

You need to understand how the components are talking to each other. So that is the architecture pillar, and then coming to the attack knowledge. Yeah.

So you definitely need to understand the, you know, um, attack scenarios, how different threats could be exploited, and, you know, um, what, what is the current, uh, you know, uh, attack, attack factors going on in the two days world and so on, right? And the third one comes the risk rating, which is once you have identified the, um, attacks or the threats you need to analyze their impact based on that, you will the risk rating and not just risk rating. You will also have to, you know, um, uh, come up with or, uh, provide, uh, provide guidance on how to build the security controls or mitigations to counter those attacks.

So these, these three, you know, uh, pillars form together a complete, you know, threat model exercise. With that, I would like to talk about the people and process aspect of threat modeling at S A P and s a P customer experience. Yeah.

Threat modeling, no doubt plays a very important role in our solutions for that. So we have a central team of security experts who are well trained in conducting threat modelings. Most of them are, um, certified threat modelings experts as we recognize them at s a p basically, who have gone through a formal, you know, training who have, uh, conducted few threat modelings as a colleague and then started leading and, you know, learning o o on the go.

Yeah. So they are then certified as threat modeling experts, and then they can go about and conduct, uh, threat modelings with the product teams. And then we also have, uh, security champions, as we call them, who are the representatives, uh, from the engineering teams who have gone through a, uh, 12 month security expert curriculum as we have, uh, formed here, which, uh, you know, uh, trains them in all aspects of software security.

And of course, threat modeling is an integral part of it. Now, these security champions act as multipliers for driving security topics in their respective engineering teams at S a P, uh, and customer experience, of course, we have threat modeling well built in the product development with that. I mean, every product team shall have threatened modeling performed for every new scenario or user story, right?

So with that means, you know, with tens and thousands of, uh, you know, uh, or hundreds of, uh, uh, new features getting developed each day, there is, uh, you know, at any point, uh, we have multiple threat modeling being performed, uh, you know, across this lobby. So we also, you know, some of them, we, uh, like for minor enhancements, we might take a decision that, okay, you know, uh, we don't have to go for a threat modeling. You can, uh, you know, uh, we analyze, um, you know, uh, risks and directly give guidance to the teams to, okay, you take care of these risks, and you don't have to really go through a full-fledge in very minor enhancements, I mean, to, to accommodate more security sensitive scenarios.

But even with that, I think it is very clear that, uh, you know, uh, there is a need to scale up threat modelings. Yeah. So every organization these days is, uh, struggling with resources and scalability, and we see decentralization comes as a natural solution, but we need to be very cautious before taking this decentralization approach.

We need to ensure that our processes are trustworthy and repeatable. Yeah. What we have done is we bring in people with the required knowledge from engineering teams, and they work side by side with us.

So after working with us for, let's say three months or so, they start figuring out the process. They start getting more confident in asking more questions and identifying threats by themselves. And what I have also seen is that when they go back to their teams, they start evangelizing security.

They feel more empowered and aware of what could go wrong, and they start guiding the teams. And when somebody from, you know, within them ask them, okay, this is better, and this is not, it definitely makes an impact. And this happens not because of any fear or anything, it's the mindset shift that is happening over the period of time while working with us, while, you know, writing the SEC reports for threat modeling and so on.

Now, this was about scalability, but I'll not stop here. I think I also want to bring one more point, uh, about decentralization, which helps us in bridging the skill gap. Yeah.

It's also, uh, a very important aspect that I've seen, uh, myself happening in the past, uh, you know, a few, uh, years. So security experts have, we know very good knowledge of security threats, what questions to ask, how to evaluate. We know that, but what they don't have is they don't have the product knowledge, right?

And the security champions from engineering teams have in-depth knowledge of the technologies that they're using. They have, you know, clear understanding of the intricacies of product development. And when we bring them together, finding threats becomes easier.

The result, uh, I have myself upskilled actually a bunch of security champions last year by mentoring them and involving them as colleague in the workshops. And I can clearly say the results are very impressive. And also the security controls.

The, uh, uh, we know that threat modeling is one of the security controls from, uh, nist. And as we are following the n cybersecurity framework for our processes, all of our products follow the recommended threat modeling methodology to fulfill maturity level three or four, as the case may be. And as central team, we own the process ownership, we maintain the process, cookbooks, as we call them, take care of the evolution of the evolution and improvements in the process.

We ensure that all threat modelings follow the defined process. We ensure that we have, you know, structured templates for the reports. Everybody uses those templates.

Yeah. Basically end-to-end process handling is managed by the central team. So that in a nutshell is how, you know, uh, we go about, uh, you know, uh, the threat modeling, uh, threat modeling as a concept in s A p now, uh, getting into a little, uh, in the next step, which is how do we actually go and conduct the workshops?

Yeah. So the workshop is basically a two to two, three or four day, depending on the size of the, you know, scenario or the feature exercise at least half days is that we block. And then it, uh, is, uh, you know, uh, attended by the product owner, the architect tech lead, the threat modeling lead and co-lead of course.

Um, and what we need to prepare before getting into the actual workshop is aligned with the teams UNA availability, et cetera. And then, uh, you know, get product feature documentation, understand the underlying technologies if needed. We also need to sometimes invest time into learning those technologies, you know, because we know there are, you know, hundreds of cloud services these days, and not everybody is on, on top of everything at a, any given point in time.

So, like, for example, uh, I was doing, uh, threat modeling for moving to managed Kubernetes service. And, uh, that was a little new to me as I spent two days to understand, you know, how does this, uh, Kubernetes service works, what, uh, how the communication happens within the cluster, what network policies do they have in place? Uh, what or what security gaps do they have, which I need to be, you know, discussing in the workshop, or I need to take care of Yeah.

Ask the questions around that. So, yeah, I mean, uh, this is just an example, but you will have to, you know, uh, learn and upskill yourself as the, uh, uh, you know, uh, situation demands. Now, let's, um, when the workshop actually starts, what happens?

Uh, the first and foremost important thing is you, you know, uh, create and conducive atmosphere for the team to openly share, uh, you know, their ideas openly share, uh, their gaps or whatever, you know, the product has, and they don't, uh, feel that there is somebody, you know, finger pointing at them or cross questioning them, or, you know, uh, finding faults with them. Yeah. So that is definitely not the intention.

They have to feel that we are helping them find gaps earlier that could potentially create a lot of problem at a later point in time. Yeah. So that's about setting up the stage.

Um, um, then as we are new to the product, we'll ask them to give us a POC demo, ask questions, questions, questions to understand, you know, how the communication happens between components or how authentication happens, how, uh, you know, they're taking care of something, you know, different things. Um, but the actual analysis happen when you start discussing the threats in detail one by one, depending on the applicability. Yeah.

So you'll ask them like, uh, you know, how, uh, the access is restricted or whether the admin, uh, functions have two F two factor authentication enable, for example, or, uh, how do you take care of, you know, uh, input validations in your application, uh, network policies and, uh, uh, are all your security, you know, relevant, even logged? And yeah, the list goes on, right? So in, um, and it is important that, you know, uh, whatever you have discussed, you take detail notes of everything, positive, negative gaps, not, you know, how, what security controls they have in place.

So everything is recorded in detail, uh, so that, you know, later on you can go back and check, uh, not everything you will remember for sure, and not after five months or six months when you go back, want to go back and see what was discussed, whether did I discuss this threat or not. Yeah. So the notes from a integral part of threat modeling.

And then let's, uh, conclude the workshop. Now, we are out of the workshop and we need to form the report. This also can get an, get very intensive exercise because, uh, sometimes during, you know, uh, analyzing this field that, okay, some questions were missed.

I need to get more information on that. I need to ask this question again. You know, read through documentation, whatever they have said is, is similar to, you know, uh, the reality.

And, uh, sometimes we also find gaps, uh, while going through the documents. Yeah. So I'll come to it, uh, in the next slides.

Uh, and then you form the final report, which is basically, you know, explaining all the threats in detail, uh, how the attack could be exploited, what is the impact of every, you know, threat that you have identified how, what, uh, you know, uh, suggestions or, you know, uh, recommendations you have to mitigate those attacks Yeah. Or threats. Yeah.

So it, that forms a complete report and with Yeah, of course, a lot of other details. And then, uh, when the report is shared with stakeholders, they might have questions, yeah, why, uh, why is this rated as a right? Or whatever.

Yeah. So you might also have to get into those discussions. Now, one more thing I want to, uh, you know, highlight here is, uh, we not only do threat modelings for applications, we also, uh, thread model C I C D pipelines and infrastructure in some cases.

Yeah. To give you a glimpse of, uh, C I C D pipeline threat modeling, uh, again, you start with the, you know, complete understanding of the complete architecture or data flow, and then you'll start evalu, uh, evaluating, like, who has access to the pipeline, who can push the prs? Do we have all open source scans for the pipeline, or which version of code is included in the specific docker image who build the image and the list goes on?

Yeah. So this is an example. Uh, uh, yeah.

So that was about, uh, you know, uh, the threat modeling workshop itself. And, uh, what I want to show here is an example tan diagram, which has three fundamental elements, agents, storage and communication. This is actually what we use to, you know, uh, represent the architecture for a threat modeling, uh, in a simpler form to understand the communication flow to, uh, you know, um, uh, to also define the scope of our threat modeling discussion.

Now, the exciting part, yeah. Uh, so I have condensed all my real time learnings and challenges from the numerous threat modelings I have done into this one slide. Yeah.

So in this digital era, we rely on the multiple hyperscaler resources and services for any product or application. And s a P is no exception. Yeah.

So when we are on cloud services, the shared responsibility model comes into picture that is, there are security mechanisms provided by the cloud vendor, but we are responsible to configure them securely. Let's say if we take, uh, you know, we are using Azure key for, uh, uh, you know, storing and managing our secrets. Now, we can't blindly, you know, close our eyes, okay?

Uh, now we have put secrets in, uh, Azure keyboard. Now it is Microsoft, you know, as Azures responsibility to store them securely, to encrypt them, rotate them, whatever. Yeah.

So although it is a managed service provider, we know they have, you know, uh, a good level of, you know, security controls in place. But it is also our job to analyze and see, you know, how do they encrypt? What encryption mechanisms do they use?

How often do they rotate the keys? How often, uh, you know, uh, what are the SLAs to even retrieve the secrets from the, uh, you know, from a K V when we need them? Yeah.

So, uh, all these things come into picture when we are using the cloud services. Yeah. So we, and, and, you know, uh, so last month there was something available.

I cannot, you know, uh, rely on that. So I have, there will be some changes happening day by day, day by day, and we have to be on top of changes or, or on top of the, you know, offerings from the cloud services and also our internal product standards. Yeah.

So product standards based on the external attack vectors keep on evolving. So we have to, um, be on top of that every day. Yeah.

So this is, uh, I think a key skill for a exec threat modeling expert or a security expert today. And another key aspect, uh, you know, for threat modeling is, uh, dependency on other applications, not just hyperscale services. It could be your internal, uh, organization, products, thumb, third party services, whatever.

Yeah. So when you are threat modeling, uh, with the product teams, they are only, you know, focused and only concerned about the, uh, code base that they own, right? So this is our application, and these are all the other applications which we rely on for so and so, so and so, so and so thing.

So if there is no gap with us, we are fine. Yeah. So they, but as a security expert, you cannot close your eyes, okay?

Uh, this team is not having a problem. We don't know how the, you know, uh, other product is handling this. Like, for example, um, uh, I am an application which uses the, you know, UI framework of a parent application.

Now, how the parent application handles the, uh, web-based, you know, protection mechanisms, I don't care. Yeah. So from, it's not, you know, it's, it, it's not a bug for me or it's not a problem in my influence.

So they kind of keep their hands off, but we will have to go and investigate how the parent application, you know, does the web protection mechanism, what validations do they provide? What documentation do they have? Is this sufficient?

Do we need to ask more questions or whatnot? Yeah. So this dependency aspect, we cannot, of course, threat model the whole, you know, uh, application or the third party service itself, but we at least need to have sufficient evidence in the form of documentation to ensure that the end-to-end scenario is secure.

Right. Moving on, um, as I spoke about the NIST cybersecurity framework, uh, you must be also aware that it actually asks us to revise threat models every six months, right? And I just explained the volume of threat models that we handle in every day scenario.

Yeah. So with that, um, to revise a threat model in six months, it totally makes sense because, you know, what we threat modeled, uh, six months or one year back, there might be more enhancements done to that scenario. There might be new risks introduced.

Yeah. So totally makes sense. But to, you know, accommodate those revisions of threat models along with new threat models becomes a huge challenge.

And resource constraints are, you know, similar, uh, to what I said already, and for the both of them, I think, uh, the solution that I spoke about, uh, the decentralization approach really works well, although it's still not a hundred percent, you know, um, sufficient to, uh, scale up so many threat models, but at least it, you know, kind of provides us some breathing space. And I also spoke about, you know, keeping up with the, um, evolving security threats. So that is very important.

So yeah, mentioning it again and again, and now, uh, I as promised, last but not least, uh, there are some simple and very effective best practices that you can utilize or make shoes off to make, take your threat models to the next level, starting with the people aspect of it, what you can do, uh, mindset. I already spoke, I will log again, elaborate. So it's highly, highly, highly, you know, uh, uh, important that you set the right, you know, uh, mindset and the set, the right atmosphere.

Maybe by adding more humor, uh, to the workshop when you start or sharing some success stories from past, the team has to open up. They have to speak for their product. If they try to hide things in the workshop, it's very, the, your job will get, you know, difficult.

You can't dig into every code or everything without them coming and telling you. Yeah. So this is the basis of a threat model, no doubt.

And the next one is also my favorite. Ask as many questions as you can to understand the future, to understand the whole communication flow, to understand what each component is doing. Yeah.

So for you to have a fair understanding of, uh, what you would need to focus on, where you need to ask more questions, you know, what threats could be potentially be there. And of course, you need to have a good understanding of your organization's product standards, uh, or security and data privacy requirements. I'll come to the last point, uh, at the end.

And we have, uh, also, you know, uh, uh, uh, some recommendations from doing the threat modeling itself is, as we know, security is a lot about how do you handle the edge cases. Like if, um, I have a field which can only take six characters. Now what happens if I enter more characters?

Yeah. Does it go into an real estate? Does it throw some exception?

Does it, uh, worse, you know, execute some script for you or whatever. Yeah. So, uh, if I can handle so many requests at a time, what happens if I user, you know, if there are thousands of requests coming into the application, at the point does it go down, uh, does it, you know, just stop working or, you know, slows down whatever.

Yeah. So focusing on the edge cases, asking questions on the limits, you know, okay, this is the limit, and what happens if you go beyond the limits? So these are, you know, very, uh, crucial to catch in threat modelings and, uh, the dependent applications, as I already mentioned, uh, the product team will always try to get, uh, get their hands away, but make sure you create, uh, you know, find gaps.

If you have identify gaps, create tickets, make the team follow up with the dep, you know, the other team to get those fixed before releasing their scenario. Unless that is fixed, the scenario for the customer will not be, will not be secure. Yeah.

So we don't want an insecure scenario to go out just because I am not responsible for this floor. Yeah. So this should not happen.

And, uh, this, uh, that's why I'm bringing it again and again. Yeah. So it's also very critical that we, uh, take care of these, you know, dependency, applica dependent applications.

And then, uh, eh, the document part. So, uh, it happened once that, you know, uh, not all threats can be identified verbally, asking questions and, uh, answering, uh, uh, looking into documents really helps to also identify GA gaps. I have found them multiple times.

So in one case I can say that, you know, uh, the, uh, the document was saying that mean, you know, uh, stores the cryptography key on his local machine, and then some steps and some, something happened now, but, uh, it doesn't say, okay, once you are done, uh, you know, transfer, uh, the key to the required place, you delete the key from your local machine. Yeah. So that could be an issue later on.

Yeah. So if the key remains with the admin, uh, it could be misused in whatever way, so that nobody will tell that very, you know, upfront in the discussion. So sometimes, and many times if you go through the documents, you are, uh, you can find flows, or at least you can find pointers where you can ask more questions to find threats.

And then last, uh, but not, uh, least is, you know, uh, as I said, not everything can be answered. So it's always good that in case of a web application, you take that application at your, with you and try to do some quick pen tests Yeah. To see how the validations are done.

Is everything looking fine? Or you can find some gaps. Of course, it does not mean that you do a complete, uh, pen test for days and days, but yeah.

Uh, uh, you know, a quick two or three hour review would, uh, give you a fair idea of the security level and you can raise, um, you know, uh, tickets for that. So, uh, coming back to my point about nobody can find all possible threats. So like, you know, uh, no matter how many threat models you have done, no matter how much of an expert you are, or no matter how many, you know, um, uh, security threats, you are aware of how many technology knowledge you have, nobody can ever, ever, ever find a hundred percent of the threats.

Yeah. That's a fact. So that we need to be aware of, and when we are forming the report, we always try to, you know, get involved.

Okay, let me find one more. Okay. This is looking this, uh, you know, little bit, uh, uh, fiy, let me explore this, let me explore this.

But there is no end to a threat model ever. Yeah. So you can go on and on and on ever, always, but you need to take a call, okay, this, for this effort, I have found so many threats and it makes sense to now, you know, uh, pull the plug and just, uh, conclude the workshop or conclude the, you know, the whole threat modeling exercise itself.

Yeah. So, uh, be also aware of that, you know, being a perfectionist here will not help you so much in terms of, of course, uh, you know, you might do one threat modeling in two months, whatever, but with the volume of threat modelings, it might not always make sense for you. Yeah.

Okay. That's all. So with that, thanks a lot for listening to this talk and I really wish that this brings you, uh, a lot of value for you to go back to your organizations and make your threat models more and more effective.

I would be also really interested to hear from you if you have any stories to share, if you have any comments, feedback on the session or whatever, or if you simply want to connect with me, uh, I have the QR for my LinkedIn, please scan and you can connect, connect with me or write email to me or whatever. Yeah. So thank you very much.

Have a nice day. That's all.