Enhancing Supply Chain Security with TestifySec’s Cole Kennedy at KubeCon Paris 2024

This is Textron tv. All right guys, we're back at CubeCon plus Cloud Native Con here in lovely Paris. And we're talking to Cole Kennedy, who's CEO for testify sec.

And we're gonna be talking about, well, the failure to communicate in the land of DevOps. Cole, welcome to the show. Hey, thanks for having me.

We have all these tools. They generate all this data, and I've talked to developers about this issue a million times. 'cause they, they hate this, right?

They're like, somebody comes along and says, can you update this? Whether it's a project management app or whatever it is. And they go, why do I have to update all this stuff is in this thing already.

I have this tool here. Why can't you just collect it? And they get frustrated.

'cause they don't want to sit down with somebody and say, you know, here's the update. Or they don't wanna have to come out of their thing and say, you know, let me send you a message about a thing I just did so then we can have this asynchronous conversation. Yeah.

That is all a waste of time. So how do we kinda uplift this whole thing? Well, I think, you know, we can look at, you know, the recent past and when we started working with DevOps, right?

It was all about how do we communicate, how do we communicate more effectively, right? Uh, you know, before we had these DevOps methodologies, there might have been that email that you sent to your production team saying, Hey, here's a binary, let's go deploy this thing. Right?

But over time we started developing APIs to help us out with this, right? Um, Kubernetes, Ansible, Terraform, these are all APIs that allow us to communicate much, much more effectively. Um, the problem is, is that, you know, we don't have these APIs really, or these APIs are very new when it comes to communicating about security, which is I think a lot of these tools that they're talking about, right?

I see folks they like, use even Discord servers now to kind of communicate about a project. Feels like we've done a lot of unnatural things to figure this out. So, um, how automated can automated get as we go forward?

Because you guys got the framework now, but what comes next? Right? So, well, the framework, first of all, I'll do a little plug, it's in the, in total A is the API that allows this, a lot of this to happen.

It allows you to create signed metadata about the different events in your software delivery lifecycle, right? So when we have this trusted telemetry to communicate these events, we can evaluate this in an automated way, right? So pushing all these events into an API, whether that's a security scan, a two party code review, um, you know, or, or some other sort of a sign off or other tool that you use once you put all this data into an API, uh, that, that data can actually be used and be on automated or be evaluated in an automated way.

Um, so, so that's really, you know, where, where the industry should move to if they wanna fix some of these issues around communication with DevSecOps and speed up these manual processes. We've had this issue for years is, and we're now at a CubeCon event. Yep.

Is there something about cloud native and microservices that kind of forces this conversation? Do we get to a level of complexity where people go, I can't deal with this anymore? Yeah, I think that's what you're seeing is that we're moving much, much faster because we have these APIs around DevOps, but we're still slow when it comes to security because we haven't efficiently, we haven't been able to, uh, um, make that communication more efficient.

And so when we have microservices, right? That's even more applications that are moving even faster into production. So security can become a really big roadblock unless you have a really good methodology to understand the status and the risk of your artifacts.

And when you have more artifacts because you're using microservices and you're using DevOps, you have more releases, right? It only compounds the issue. And, and so this is why you're seeing a lot of the SSOs, a lot of security engineers are, are very over overworked and this problem because on tractable, unless you kind of change your methodology in how you're doing things.

So in, in a way we're kind of creating this immutable workflow that we can understand and we can verify that this took place with this time with this component. Mm-Hmm. Who, um, who wakes up in the morning and says, we gotta go solve this issue.

Is it the security side? Is it the DevOps team? A-C-I-O-C-T-O?

Yes. Well, I mean, you know, it's up to the ci o the CCEO of the company to understand where their risk is. And we've seen a lot of organizations have failed to understand how much risk is this in their software.

Um, so it starts from the top up, right? And then, you know, as an engineer, we have the responsibility to put out secure software that protects our users. So that's from the bottom up too, right?

And so it really has to be a whole organizational effort, which again, comes to the problem of communicating, right? And sometimes it's very difficult to have that cross organizational communication, especially when it's a, a manual form of communication. It seems like there's a lot more regulations talking about securing the software supply chain.

We had the Biden administration and issue an executive order for the federal agencies anyway. Um, do you think we're gonna see more of that type of regulation that's gonna require companies to go address this issue a little bit more? Because right now I feel like it's still a, you know, it's a vitamin in the sense you should do this, but it's gonna become a headache real soon.

Yeah, I think we're starting to see it change from should to shall, um, you know, the, the this, uh, just released that self attestation, um, PDF, that now if you're a CEO of a company, you, you need to sign that thing or have a designee that's authorized to sign that, sign that thing. And, and that form, you know, puts down on paper that you comply with, uh, secure software development, um, practices when you're pushing off software. So if you're selling to the US government, like these, these are, these are shall now you must do these things in order to do business with the government.

Um, so I think we're gonna start seeing this filter down to other organizations such as, uh, large banks and, uh, other industries working with high compliance, right? Because, you know, a big part of, uh, of their threat is that software coming in from their vendors and they really wanna be able to control that. Um, and right now it's very difficult for them to do that because we get back to that communication, right?

It's tough for them to communicate what the status of, um, or what the vulnerabilities of the software that's coming into their system. They just don't have the system set up to do that. I don't think everybody knows who you are, but, um, you know, how did you get into this?

Where did you come from? I mean, not everybody wakes up in the morning and goes, I know I'm gonna go solve this security software management issue. Yeah.

So my co-founder and I, uh, Mikel, we were working, uh, for joint special Operations command, um, developing application to help the war fighter down range. And I actually got sent overseas week before Christmas to go deploy this application. Um, and I got there and because it was on a different network, this was a NATO network versus a US network, the compliance rules were completely different.

So we weren't able to deploy that 'cause the software because we didn't check all the boxes. Um, so I ended up spending Christmas away from my wife and my one, my 1-year-old daughter, and we completely failed the mission. The war fighter didn't get the software they needed all because of this compliance issue.

So, so this really gave my co-founder and I the impetus we needed to say, Hey, what do we, what do we do to solve this problem? Right? Um, and it ended up being a lot more complex than we thought it was.

Um, so, so, you know, five years later, um, you know, we were working with, uh, DOD platform one for, uh, consultancy called Fox Code. And we ran into some of these similar issues, but this was for a nuclear weapons program. They, they had these requirements that, that code it needed to get checked, right?

But, but this didn't fit into the paradigm that Platform one had created. Um, so we reached for this open source project called in Toto and, and started implementing there at DLG platform one. And it worked great, but it was a, it was a very academic project.

Um, so Mikhail and I, we started working that, working with that project, adding the features needed to work with enterprises. Um, and that's when the company we're working for actually got, got sold to IBM. So we're like, Hey, you know, we think we have an idea here.

We think we know how to solve this problem. And, and it's a problem that does need to be solved. So, so we started, uh, testify Stack, uh, with, with the, uh, you know, the purpose of, um, providing everyone with secure software.

So first of all, let me ask you, is your wife still talking to you? Because most people, you know, are the 1-year-old and you're overseas and it's Christmas, you would probably be getting a lot of dirty looks for a long time. Yeah.

Yeah. She, well, now that I'm bringing her to Cube Con in Paris, she's much more happy with me. Um, but yeah, it was, uh, you know, solving these problems does take a lot of support, um, from your family, especially when you're getting, getting deployed.

Um, it was a, you know, that was a, it was a difficult job. Um, but, you know, it's something that, you know, had a profound impact on myself and profound impact on my co-founder. Well, besides, you know, what happened on your family side, but with the work, do we underappreciate the amount of stress that we create because we don't have the right kind of handoffs and the right ability to verify that this was created?

'cause it seems like there's a lot of, you know, I don't know what I got, or I get there and suddenly I'm missing a component in your case. But, um, you know, have we become our own worst enemy sometimes? Yeah.

And that's all goes back to the inability to communicate. If a CIO could communicate with the developer and says, this is exactly what you need the software to do in order to be compliant and, and do that in a way that the developer understood, right? A lot of these problems would go away, right?

But what we see is, okay, the software's ready to go in production, but now we're missing step seven, eight, and nine, uh, because we have all these regulations because this is going into a bank, or this is going into a weapon system, or this is, this is going into some other high compliance areas and we just can't, we just, we gotta do the right thing, right? We, we gotta be compliant. Right?

The Army has, uh, an acronym called snafu, right? Um, have we accepted that too much in software? 'cause it feels like sometimes, uh, our teams are willing to put up with a lot of things that maybe they shouldn't.

Yeah. It, you know, as a software engineer, we're taught to move very fast, right? Getting shipping things is the most important thing.

Um, and, and with that methodology, right? We forgot about, you know, basic engineering, right? We should, when we create something, we should test it to make sure it works, right?

And, uh, I I think now we're starting to see all that come full circle, uh, with that, with, with DevOps now we're turning in DevSecOps taking that security. And so I think things are changing. Um, um, but yeah, there's a lot of work to do, right?

We need a lot, we need that verification, that formal verification of our software in the processes our software goes under in order to understand the risk associated with it. Yeah. So what's your best advice to folks?

'cause you've been here, you've lived it, and you know these teams. So, you know, when you go visit a customer, what's that kind of thing that comes to mind when you go, folks? Man, if you just thought about this one little thing, it would make a world of difference.

Yeah. It's all about observability and, um, the software delivery lifecycle is missing that component. And this is one of the things that we worked on with the CNCF software Supply chain Best practices paper, as well as the CNCF secure software factory reference architecture is we found that this, this, this observability component just missing from all these different CI systems.

Um, so can, can I tell you an analogy kind? Sure. So, I mean, if, if you had a daughter and she was getting married and she wanted this perfect cake for, for her wedding, right?

You would bring that set of instructions to the baker and then a couple weeks later that Baker would drop that cake off at the wedding. Well, what you've done is you've established a trust relationship with that baker. If that, uh, that kitchen, they, they baked it in was dirty, or the, the, they didn't wash their hands or they use the wrong ingredients, you really don't have any way of knowing that until you eat that cake and get sick from it, or don't get sick to it.

Um, so if you didn't want your wedding to be ruined, I would recommend that you hire an observer, um, to watch how that cake's being made, right? Every time that baker washes their hand, they write that down on a piece of paper and they sign it, right? Every, they, they, they look at what is the oven set to write that down on a piece of paper and sign it.

They, what ingredients go into it, right? Write that down on a paper and sign it. So now when that cake is delivered, you can deliver those, that envelope with all those steps in it.

As long as you trust that signature on that envelope, you know how that cake was made, right? You don't have to trust that baker. Now you can trust that observer to know that, hey, this wedding is not, not gonna get, uh, get ruined.

And we need to do the same thing with our software, right? We need an observer to watch the process of our software being made. And then when we do that, right, we can look at those observations and as long as we trust them right now, we can run that software with a good sense of assurance that, hey, it was built the way that our regulators tell us we need to, to build it.

Or the way our cis o tells us we need to build our software. Right now we know it is because we trust that observer. So what are the components to make that happen?

Because if it's observability, right, I need some ability to collect it and some ability to ob analyze it. So what are those things, right? So, uh, we, we are, we maintain two open source projects at Testify SEC one is witness and the other is IV Vista.

So Witness is our observability tool. Um, this collects information around the CI process. You can either embed it directly into, uh, your CI process or, you know, we do have some integrations with GitLab and GitHub that make it a lot easier to do, but then you need somewhere to put this information.

So that's why we create IV Vista. Uh, so IV Vista is a storage database that stores these in total attestations and allows you to query them. Um, so with these two components, you're effectively able to observe the entire supply chain, have a great place to store that information and query it.

And then witness also has a policy engine that allows you to evaluate all this evidence to determine whether an artifact is compliant or not. Um, most of the time this process takes days, right? 'cause you're exchanging emails back and forth.

We can reduce that time with those tooling to, you know, you know, under a second, you can't walk down the street without somebody talking about ai. Is this a foundation for right now? I'm gonna go query something to find something, but at one point, will it start to just tell me that there are certain things that are a high risk proactively in a kind of a danger will Robinson kind of way, but let me know that there are issues in there that I'm not even aware of.

Yeah, I I think AI definitely had to place a play in, in the area of security, but exactly what you said, right? To bubble up information that you might not normally see. Where I don't see AI playing a role in security is where we need to make deterministic decisions about, um, the risk of a software artifact, right?

So yeah, a decision made by an AI engine is, isn't necessarily, at least not today, is not gonna pass the muster for, for most, uh, compliance officers, right? So we need some way to do that deterministically. So that's really what we're focused here on set testify SEC is, is not using AI to, to evaluate these at the stations.

However, uh, we use a deterministic model to do that. And then we'll use AI to help the user sift through the information and find out, you know, that, uh, you know, some of the, uh, stuff you're talking about, those events, probabilistic is a fancy word for gambling, right? Yeah, yeah, yeah.

Right. It is. And regulators don't like that, right?

Exactly. Buddy, thanks for coming on by. Hey, thanks for having me.

All right.