Rosalind Radcliffe & Thomas Lawless – Securing the Software Supply Chain for Cloud-Native Development

Hello and welcome to this next session as part of the virtual Cloud native conference. My name is Rosalynn Radcliffe. I'm an IBM fellow and my new role is in the CIO organization responsible for the deaf secops transformation.

And today we're going to be talking about securing the software supply chain for cloud native development and I have with me Tom Lawless Tom. Would you like to introduce yourself? Good morning, everyone.

Thank you Rose Lynn. My name is Tom Lawless. I am a senior technical staff member within IBM CIO responsible for our developer experience.

And so first we'll start with a little bit about the IBM CIO organization. We are the organization which runs the it which runs IBM. So we provide things such as the HR systems payroll systems all of those fundamental systems to the organization of IBM and if you think about IBM, we're relatively large organization, which would translate to our it organization is probably also pretty large if we look at some statistics about the organization, you know, we have about 6,000 Developers.

If we think about that scale and scope and then you can see the number of applications GitHub repositories and it's important to know that we support all of the applications that support IBM from the large it standpoint and they run on various different systems. So this is not the entire landscape that we have. We have some systems that aren't yet in GitHub.

We run applications running on zos. We run their party applications. We work with SAS systems.

And so we have lots of self-developed applications through the Spectrum and if we think about this kind of organization, it's no it's a very large organization a lot of work to do and we're doing the same thing as everyone else trying to transform to a more Cloud native approach and if we think about all these applications and then we think about the platform that we want to support this if we move on to the next slide We'll see we're trying to be a showcase for the hybrid cloud and truly demonstrate how a large it organization can build upon the hybrid cloud and use the hybrid cloud and so as I said, we have some SAS Solutions running in various different places. We have things running in IBM cloud. We're trying to bring this standardization across this portfolio.

So when I have a developer experience, I have a similar way of building whether or not I'm targeting anyone of those various runtime platforms. We're working to abstract enough of the infrastructure from those developers to make it simpler and easier to do development, but recognizing that we're going to need the various different kinds of Hardware platforms to help optimize our capability and help optimize. What were the service we're providing to IBM.

in this particular instance, we're going to talk about this developer experience and in the the devsecops transformation that we're doing so if we continue on So Rosalina, as you know, we are asking more and more of our teams in terms of how they build and deploy their applications to our hybrid Cloud platform in our teams are struggling to keep up with the amount of work in the amount of change that we're asking them to do in this area. We know that if we continue to do this per team and per application, we will continue to see them struggle it'll impact their productivity and given an organization of our size. We're talking about a large amount of redundant effort a large amount of developer wasted hours in recreating the wheel.

Going back to your comment about just the number of GitHub repositories that we deal with 70,000 is a large number take any one of these bullets any tool that we would integrate into a cicd pipeline to help us get better visibility help our teams achieve the goal of say open source vulnerabilities, even if we spend one hour per repository and enabling that tool as part of a cstv pipeline that 70,000 hours that we spend that year within our organization. Just turning the tool on that does not include analysis. It does not include remediation.

And when we think about this securing this supply chain, we can all think about the log for J vulnerability that hit in fourth quarter of last year. I know within the CIO organization. We had people who had to come in and and deal with this during their vacation.

But the biggest problem was we didn't know where it was what there was no high level view of which applications were using long per day. So every single application within the organization had to report was log4j used. Therefore.

Did they have to do something or was it not used and therefore they didn't have to worry about it not having this visibility into law for Jay made it very obvious to the organization, but this is only one small piece of the picture. Having visibility into all of the code repositories understanding what's going on with all the individual teams. Is a key critical component in a large-scale organization and then making it easier for teams to make these changes and deal with this Challenge.

And so this along with the experiences that were going on caused us to want to create this developer experience team and Tom is working in that developer experience team and we'll give you a little bit more of that. So our developer platform initiative is meant to help our developers become more productive while we try to solve some of these problems at our organization level and we look at our developer platform. We can break it up into four major pieces.

We have our automation catalog in our goal there is to unlock the expertise of our subject matter experts break down the silos in between teams and can in create a culture of inner Source contribution with the work that they are already doing. We want to get that into a more centralized location encourage them to share it so that the rest of our organization can benefit from their expertise. Pipeline execution management is is our ability to run ci/cd automation on any platform that the CIO supports going back to Roseland statement a few slides ago.

We have IBM Z. We have IBM power. We have x86.

We have a lot of IBM product in red hot product in our stack and we want to make sure that the automation we have in our catalog can run wherever our developers need to platforms support. As those pipelines run they collect a lot of data. They produce a lot of data a lot of evidence about the quality and security of our applications and we want to start to collect that data in a centralized location that we call the developer data lake so we can begin to provide better insight to the developer from source code to deployment to put them on a path of continuous Improvement as well as Drive some more visibility across our Organization for areas of risk like vulnerabilities.

Finally, we want to incorporate all of this into what we are calling a developer experience a dedicated developer experience you integrating these other three components to improve their productivity and again, encourage the continuous Improvement of our application source code in terms of quality and security. So if we look at each component in a little bit more detail the automation catalog again are our play here is to encourage the contribution of high quality automation through innersource. Inner Source, if you're unfamiliar with the term inner sources much like open source, except it's within within our company within our organization.

So we want to encourage the sharing of the code that our subject matters have a subject matter experts have already created or planned to create in the future. We are anticipating a lot of automation because again, we have a large platform. So classification of that automation becomes really important.

So we understand what it's purpose is where it's capable of running and through classification. We can help Drive discovery. The publication process is meant to ensure that each piece of automation that's contributed to this centralized initiative is itself high quality and secure.

Running my own Jenkins instance with you know a ci/cd pipeline in it. If it breaks it only breaks me. As we move to this model a task that is used across, you know hundreds of teams.

If that breaks, you know, we we have another productivity problem that we need to solve as well as if it has a security vulnerability in it. We open ourselves up to a different type of software supply chain attack that we need to be need to be aware of and put measures in place. Discovering configuration is where our development teams go to understand what automation is available to them and how they incorporate it into their repository incorporates.

Not the right word actually because of our size because of the number of GitHub repositories that we're dealing with we don't want each repository to have a complex pipeline configuration to build and deploy build packages deploy that that particular piece of source code. We want the repository to describe its need. And I'm I am a Java microservice.

I need to be containerized that I need to you know be deployed to the platform. That's really all we want at the repository level so that we can interpret that need today. And then we can evolve our interpretation of that need over time to reduce the amount of time that we spend maintaining pipeline configuration per Repository.

Finally we want to be able to transform that automation anywhere that we need to run it. And I'll talk about pipeline execution in a moment, but we will continue to have teams that will be using our centralized ci/cd initiative that we're creating. But we'll also continue to use their own existing tic platform until it's time for them to migrate over but we need to make sure automation is available to them wherever it's being wrong and our transformation component is what we envision enabling that.

Pipeline execution management again is is our ability to run automation wherever our developers need support. So that starts with integration the the most straightforward integration there to think about would be integration with our source code Management Service right with with GitHub for example, so when it you know change is made there's events that we interview we capture through integration that allow us to orchestrate a pipeline that results in task execution, but we are building a platform for developers. We want to make sure that our platform is extendable by developers so that there is not only a sense of ownership in it across our organization, but they can continue to drive additional productivity enhancements based on their own team level workflows Integrations.

Give us the opportunity to do that in we would imagine that we'll see Integrations with other tools like slack for example, or other source code management systems and really we want to make the potential they're almost unlimited wherever developers feel they will get a Activity Improvement. They can contribute it integration through winter source. Is an interesting one for us because of the number of platforms that we support.

We have a goal as Rosen spoke about our stock. We really want our developers to work at the top of that stack. And not be so concerned about where their code actually runs.

We want the platform to take care of in order to enable that we can use an example of containerized applications in order to enable a container to run on any one of our supportive platforms. Whether it be ibmz IBM power or x86. We would look to open shift to run our containers where it makes sense for that workload based on the characteristics of that work.

But we need to be able to build a container image that supports all those platforms so that the platform itself has the ability to run it where it makes sense to run it. Building a multi-architecture container image means that we need to build the image on each one of the CPU architectures. So we wind up with a fairly complex CI pipeline that would you know results in these multi-architecture container images.

We want our developers to be aware that we're doing it, but we certainly don't want them to have the configure that per repository or worry about the nuances of doing it across multiple platforms. So our pipeline orchestration component is meant to facilitate that type of multi-platform cicd pipeline where we can delegate pieces stages of that pipeline to the platform that needs to execute those particular sets of tasks. And that flows into task execution so you can imagine Pipeline orchestration and we use tecton for this by the way, right?

We we want to use open source and IBM IBM products Red Hat products, wherever possible tecton is a really powerful workflow engine for us and we can use tectonic multiple levels with with this concept where we use tecton to orchestrate this overall kind of conceptual pipeline. We can put a little custom code in between to then facilitate a tecton pipeline to run maybe on Z to build a z a nest 390 container image of you know, that matches that CPU architecture same thing for power same thing for x86. So the combination of pipeline execution Pipeline orchestration and task execution enables us to to facilitate these complex.

Yes you pipelines so our developers don't have to You know as as those pipelines execute we want to be able to capture the output of the pipeline execution and give it back to the developer to drive transparency. They should be well aware of what we're doing. We don't want to hide anything from them.

But going back to the requirements that we have for CSD pipelines. We also want to be able to capture the evidence of an application or repositories Quality Security and compliance so that we can automate that aspect of the developer role as well. So we look at our developer data like it's three major components.

We need the ability to aggregate data. Now this may be data from a pipeline stage running on multiple Platforms in parallel. It may be data from a service where engineering metrics are collected or vulnerability data is collected that we want to pull in through API or maybe files that are generated during pipeline execution that we want to pull into the data Lake in parse so we can we can help the developer make sense of all of this data that is being thrown at them.

Once we have the data aggregated we can start to do analytics and Reporting against it in this not only helps our developers but it also helps our engineering organization as a whole so we can have a more holistic view about the Quality Security of our higher application portfolio. Finally we want to make sure our data is actionable. We want to put our data to work for us and for our developers that may be an automating compliance and audit, you know audit reporting.

It may be in putting guard rails into the pipeline. Well, it definitely will be putting guard rails into the pipeline to ensure like our developers don't don't accidentally introduce a new vulnerability into production. We have a lot of potential with with this concept and we're excited to see how it evolves and matures over the next, you know a few years.

Finally the developer experience is meant to improve the productivity our developers by putting a our big play here is the developer-centric web experience that sits on top of our developer data Lake. One of the you know, we've interviewed our developers. We've asked them.

What can we do to help you like what we understand that we're asking you to do a lot. What can we do to help help you be more productive to make your experience better? A big piece of the feedback we received was we are currently doing a lot of context switching.

We have tools all over the place that help us with engineering metrics code quality open source vulnerability open source license management and we are constantly switching context from tool to tool to get a mental model of what our application looks like in terms of, you know, quality and and compliance and security. And in some cases we're asking developers to report on that maybe even as spreadsheets, right? There's lots of lots of areas of improvement here.

Our developer portal is meant to help in that area by by using the data that we've aggregated and normalized in our developer data Lake to give them a single view of their application in context of all the data that we have. And again Integrations is also another key play here for us. We're building a platform for developers.

As I said, we want to make sure that we can integrate this data back into any tool that developers use and would find a more productive experience with so Integrations becomes another extension point for the platform. If we think about what we've just been talking about we're building this developer platform to improve the overall productivity of our organization within the CIO organization now, honestly, we want to build it and prove it within the CIO organization and we want to make it available to the broader IBM So that IBM as a whole can benefit from this capability when you think about this developer platform, we're trying to make sure we're capturing the right data to allow the individual teams to be more productive so they can identify the only their own waste within the system and remove that waste so they can they can follow the continuous improvement process. We're bringing Security in from the very beginning.

We honestly have a set of compliance pipelines already and these compliance pipelines though. They aren't doing the full build and deployment process because those teams have some other pipeline today. They're allowing them to hook into the security scanning that's available as a step.

This need for security the need to build an s-bomb for software of the future this requirement around all of the certification one needs for the software running within our containers for these Cloud native development applications is such a burden on each of the development teams that by doing this developer platform. We're hoping to remove that but as Tom described we're trying to do this in a way that doesn't remove all of their flexibility. We still want teams to have autonomy.

We want them to have flexibility. We want to make it easier for them. And so by having this doing this through an inner Source model, we can allow them to contribute additions.

They can get their flexibility but not be responsible for running the pipeline itself and not be responsible for all of the activities required. We really want to empower the developers, but remove this daily task this extra waste from the system of the developer experience or really trying to make building Cloud native development applications build applications, no matter where they're going to run more effective and more efficient. And as Tom talked about this multi-art container and the platform making more of the decisions for the developer the intention with this is that the developer writes their application they do they get to focus on their application the testing for their application.

We can't forget testing that's automated testing as part of the pipeline. That's one of those quality Gates that's part of this pipeline, but this All of that the things that the developer should be doing so the development team is responsible for those activities that are really required of them to build the capability for the business and then pulling away those capabilities that underlying platform itself as well as the developer platform to make this experience much better allow teams to move faster and allow IBM as an organization to adopt the ever evolving standards when it comes to security log for J was an eye-opener for many people. We had been working toward open source scanning and making sure we have the right security capability throughout the organization but each team doing it their own way wasn't giving us the visibility or helping the teams and so by really am bringing this into the platform and An empowering the developers to continuously improve we can ensure we're really driving that business value for the organization within the CIO organization proving this out on a relatively large-scale organization and then scaling it to the entire IBM company over time allowing them to also be able to adopt this capability adopt the automation that has been created to make it easier for them in building their tooling because we all all of us no matter where we are have to meet a certain level of security a certain level of standards.

Hopefully this session has been helpful, and you've learned a little bit more about what we're doing to try and improve the developer experience to make it easier for our developers and to help us in this continuous Improvement journey, and I hope you enjoy the rest of your day at the cloud native virtual event. Thank you.