Antithesis CMO Patrick McBride on Evolving Testing Methodologies and the Multiverse Debugger

All right. Hey, we're live back here in Cube Card. It's, uh, Tuesday night, which is cube curl night on the floor.

So when I look out there, I see popcorn machines getting ready to set up. I think I saw some hummus, but they'll usually have a lot of stuff here. Um, I think that starts at five 30 or something though.

I don't know if I'm gonna make it to five 30 here, Patrick. I might make it to the, to the first part, but, uh, yeah. Yeah, it's been a, and it's been a day I Get you Sale, But let me introduce you.

This next gentleman here. Well, he's gonna tell you the company he's with now, but I've, over the last 20 years, 20 would be 2005. Yeah.

Maybe 20 years. Yeah, probably, maybe more, maybe more. Um, my friend Patrick McBride has held a variety of roles and had a very interesting career, and I think we'll start right there.

Patrick. Welcome to Text Drunk tv. You know, I, I gave you the buildup.

Tell, tell him share your story. Yeah. Thanks.

Having me in. Um, yeah. I, a long, long time ago in a, in a, in a place far away.

Dallas, too far Away. Yeah. I, uh, I actually started off as a software engineer, so I wrote code outta school.

I was a finance and, and a CS major. So I could go one of two ways onto Wall Street. I could either write code for financial services company or, or go trading.

And I, I, uh, I took the former. You ever regret that? No, Not at all.

Not at all. Uh, I, I've had a, had a great, you know, great time in, in tech as you know, since we've known each other for so long. Yeah.

But yeah, I, I, I graduated, I, I worked at, you know, one of the large insurance companies and wrote a lot of code. Um, I would say I was a, a a, a mediocre coder. I was prolific, but, you know, but I was really good at figuring out what to build uhhuh.

So that kind led me down more of a product path, helping, you know, companies do that, which got me, you know, into, uh, one of the early research firms. I worked with our, our buddy, uh, Mike Rothman, back at a company called Meta Group. Yep.

And, um, you know, so, yep. Yeah. Came in there as an analyst and, uh, had had a lot of fun.

But then I, you know, with my product roots, I, I ended up advising a bunch of companies and ended up taking, uh, a marketing job, a CMO job, which I had never done before at an early cybersecurity company, and learned the ropes from, uh, Mo Rosen and, and some other folks, other People I know. Yeah. Good People.

Mo Mo most taught a lot of us. And so, yeah. Yeah, it's, um, yeah, had a good time doing that.

So now I'm, I guess, a seven time startup offender. I just love early stage companies. I love the, the excitement.

I like when, you know, people are just wanna run through walls. Once It's in your blood, it's, it's hard to get out of It is, I couldn't go back to, uh, to an easy job. And, you know, I do like getting dunked in the kind of the deep end of the pool every once in a while and having to learn stuff, swim, Sink.

Exactly. That. That's kind of where I'm at now with, uh, with Antithesis, uh, where I'm at now Through that.

Yeah. So you mentioned two names. I just wanna shout out, of course, I don't know if you talked to Mike recently.

I did The Candyman. He's the Candyman. Yeah.

Mike's doing candy and Mo uh, Mo sold his last company, but he's already CEO of another company. Mo Yeah, he's he's a multi-time offender too. Yeah, he's a serial.

Yeah, no doubt about it. It's, uh, he owes me a glass of wine over at our famous restaurant in our, our favorite restaurant in Reston Town Center. So Absolutely he bo but both stellar people.

So shout out to both of them. So, antithesis, look, I know you a long time, you were very big in cyber. We didn't call it cyber, of course then I know it.

Security infosecurity network security. Yeah. But you were big in security.

Tell us the anti, the antithesis story. Yeah, so the, um, you know, the quick thing that they do is they test stateful distributed systems. Um, so really complex Serbia systems that are particularly hard to test.

And kind of the backstory is really interesting. Um, the founding team at Antithesis, antithesis, uh, did a company that, that you may not know, but you use it every day. You got an iPhone in your pocket, right?

Sure. Um, it was a company called Foundation db, and, uh, it was a, Actually I know them. Yeah.

So the, the foundation guys, you know, built this really cool, um, distributed database that had some really nice acid properties, um, that people couldn't, you know, said, Hey, you really can't do that. Well, you can't put all that stuff together. They're, they said, hold my beer, and went off, off and did it.

Well, in building that, you also have to figure out how to test it. You know, they, they had property and guarantees that they wanted to make, so they spent a whole lot of time as they were building the product. They were building the whole testing methodology, and they were trying some new things, you know, the traditional, you know, write your unit test, write your integration test, and, you know, and see what it, you know, brings up was just not gonna work in an environment like that.

So they all, they all ended up at, you know, apple and, and then migrated from there to different places. And I think the, you know, around the Bay, and one of the things that they all realized was testing was pretty crappy, you know, even for some really notable companies with big systems. And they said, Hey, let's, let's go and, you know, take what we started at, at, uh, at Foundation and take, carry it forward.

You know, testing has always been, I I, I had this conversation once with two guys. One was sort of a, a classically trained, you know, went to school testing guy and one was a maverick dude Yeah. Who self-taught, but the, you know, where they met was for too many organizations.

And, and the, actually in the market in general, everyone had this vision that testers were for like junior programmers or failed programmers, people who weren't good enough, right. Less than, Right. Yeah.

And that of course, therefore testing wasn't gonna be great because the people we had working on it weren't, weren't the, Weren't professional. They were a little bit of the redheaded sub trial. They were of vike.

Yeah. It was always, yeah. They're, you know, but the fact of the matter is most testing is done by a cadre of professional testers.

Yeah. And what we run into all the time these days is the, the actual app development teams that are building this distributed applications are highly involved. You know, even if they weren't using something like antithesis, you know, they, they have to figure it out, you know?

Yep. They're, they're highly involved in the testing because when the pager goes off at 2:00 AM It's, they're the one who Has there, they're highly involved in the answer. And, you know, depending on what kind of an error is, if it's a, an outage, it may be a bad, you know, couple of hours, a couple of days in the office, if it's a, a correctness issue, the database is, you know, saving the wrong type of, of data in the database or incorrect data in the database.

That's a, that's a rough month or a couple of months in the office. And so yeah, they're, they're, they're far more engaged than, than they were in my history back when I was writing. Oh, absolutely.

There were separate groups and they're much more integrated. I Mean, look, there are developers who say, look, I'm, I just want to code. I don't want to test.

But also the, the, in the inter intervening time, we've seen this whole automated testing where even today, the professional testers don't necessarily test, they write the coverage, right? Right. They decide what to test, how to test, when to test, and then a lot of the testing today is automated.

It is, yeah. You know, we, And, and we, we really carried that to the nth degree. We kind of flip testing a little bit on its head.

So the, with traditional testing, you have a couple of issues. First of all, typically the guys that are writing the test are the engineers, right. That are writing the original test.

A, they don't like it, and b, if they're writing a test, they're trying to figure out, you know, what, what they're testing for. So they've typically thought a little bit about what could go wrong. So they're writing try test coverage to try to figure out those things.

Well, if they're already thinking about, they were probably pretty conscious when they were coding uhhuh to try to not make those things go wrong too. So you end up kind of in a testing for the things that, that you actually built tended to build Well, for, it's the stuff that you, you know, forget to test. That is the stuff that, you know, makes the pager go off at 3:00 AM So one of the things that we've adopted is, is something that we call, it's, it's called, you know, formerly property based testing.

But what you end up doing is instead of writing individual tests, you write a set of, uh, instructions about what the system should do or shouldn't do. You know, it, what, what state things. If, you know, it should never give the user a 4 0 4 error.

Uh, it, you know, if we're doing a debit, you know, of some amount, it always do a credit of some amount, you know, up in an application there. I get it. You can get way down in it.

And then what we did was take, you know, take a lot of the burden off of the engineers. You still have to think, I mean, you still have to work through the properties. I get it.

And, and, and, you know, it's a mind shift for folks. And then we take, uh, that it basically put what is effectively their production environment. We take that their containers and run it on a, a, a hypervisor that we built, um, that's, we call it, it's a perfectly determinative deterministic hypervisor, which I'll come back to the imports of.

But what we end up doing is we put load under that and we'll run it massively parallel, you know, on a pull request or overnight even. And since we're running it in such a, a massively parallel thing, we can test months worth of production wall clock time in a couple of hours over overnight. But in that time, we're not just running it in happy path.

We're throwing all kinds of faults at it. A network fault, a race condition, a service drops a disc fails. We'll, we'll throw all those kinds of things at the system so that we can get all those edge case types of issues and, you know, types of faults that you don't really typically see in your happy path coding.

You know, that's, that's a problem. You, the, the, the other problem is you're, you know, the engineers write tests for the things that they think about and they get tested in a, in an environment that's much more pristine, not at all like production. So, right.

Those two things together make it, you know, hard. So it's a one part property based testing, one part chaos testing, and one part massively parallel inspection, you know, of, of the, of the code base. And, and so the, you know, the hypervisor pieces, you know, it was, took them a long time to build.

Um, the more interesting piece of technology was the piece of technology that decides, you know, what pieces of, of, you know, where it should branch and, you know, how do we get through the whole code base and where should we inject faults? That's, that's really where a lot of the math and, and science came in. And, but the hypervisor being perfectly deterministic is important too.

'cause finding, finding a bug is only half the problem. Okay. Got it.

Then you gotta figure out what the hell happened and, and fix It. And how, well, how do you want to fix it too, because Yeah. You know, there might be multiple paths to a fix.

It's not always just a patch. This determinism element is, is important. So we, you know, we, we, we actually, I have the engineering team named this.

I loved it. It was, um, uh, you know, we, in our hypervisor, you've got a debugging tool that's, we call the multiverse debugger. You can actually multiverse.

Yeah. Multiverse, debugger. So it could goes through quantum, you can literally rewind time quantum, right?

It's quantum, yeah. It's, it's totally quantum. So, so you rewind time to, you know, a minute or a couple minutes, you know, before testing and, and we can show you graphs of, you know, you expecting a particular fault.

We can show you literally graphs that when the, it was a zero probability that would happen, and then it stepped up, oh wait, something interesting happened there. Usually it'll, you know, be another step up. And then it was, you know, a hundred percent going to happen.

We know, you know, you know, since it's perfect. So we rewind and then you can inspect exactly the state of the whole system, all the logs right. Where right where it was, and really figure out what the hell happened.

It's Like a time machine there. That's fun. Yeah.

It's it's a debugging time machine. Exactly. And, and you know, with perfectly preserved and state, so you can repro, uh, a bug.

Exactly. And then you can literally, you can go into a, a, a terminal session and, and throw stuff at it and see if that change it, then rewind it, try it again, try different things. So it's, uh, love, you know, debugging that I've never, you know, when, when I, when I saw and saw what they were doing, I just had to join the company.

We haven't done that. So, you know, speaking of which, Patrick, we, we didn't, what's your role at antithesis? See, you know, this is where I get in trouble.

Since I'm running marketing now, my, my credibility went down like 30% or, or whatever. He's not a Marketing guy. I'm, yeah.

I'm a, I'm an, I'm an old, old engineer, you know, a strain, strain or stress on the old, but, uh, running On the business side of things. Yeah. I'm, I'm, I'm working with our, our go to market team and, and you know, helping make sure that we get to things, nothing To be ashamed of.

I do it too. The more important part of it, the cool part of it and, and contextually for the conference is I get to work with some of the, the cool projects here. Right.

Um, one of the things that was a lot of fun that, that we had earlier was, um, uh, one of our, one of the maintainers for the ET CD project, which is a critical component of Kubernetes. Uh, we helped, we, we reached out to those guys and worked with them to put that under test. Oh, that's cool.

And, um, and they did, so Merrick, their, the, the maintainer who was, was working on that, um, you know, joined us on stage in the beginning of the morning and then did a, a talk, uh, to, you know, really take, take them through how you use the deterministic simulation environment, you know, to test CD. Um, and, and we had, you know, the other thing I, I got a chance to work out, uh, was, you know, working out an actual partnership with the CNCF team, so that we're offering now, uh, to put all the, all the folks that are all the projects that are incubated, uh, or further along and graduated, uh, under tests now, we're not gonna take 'em all on at once. We're taking, you know, we're out here searching for our first two and we've had a lot of people come by the booth and ask us about it.

So we're, you know, we'll just probably do it two at a time for the, the next bunch of quarters. But those projects, yeah. We, we, we feel we use a lot of open source, so it really matters to us.

It matters to a lot of our end customers. And so being able to help the projects, It's also being just a good community member too. Yeah.

We, we wanna be a good citizen in the community. Absolutely. And it's, it's in our best interest in lots of ways, and it's in lots of company's best interests.

So I, I think that's why you see all these people here that you do, man. Yeah. Patrick, you guys did announce some news here though.

Yeah, that was the big one. That, that was the big one where we just putting all the, putting all the projects under tests. Um, you know, we've got, we've got some other, you know, other fun stuff up, up our sleeve over the next couple of quarters that, uh, I don't wanna get you in trouble they'll ever to put you back to coding.

No, they, yeah, they, they, no, nobody lets me touch the code base anymore, I promise. No, I'll let you do that anyway, man. Did we mention the website?

Ah, www dot antithesis. It's an antithesis. com or do ai, You can do or do AI as as well.

We, we, we own that domain as well, so Absolutely. Either way, you're, you're, you're coming to the right spot, Patrick, thanks for coming on, man. Alan, great to see you.

I appreciate it. com. You know, know, I, I think that I'm going to give me credit on this one a time machine, right?

Yeah. For, for your testing and your, your states. It's, it's an interesting way of thinking about a very complex kind of, uh, process.

Yeah. Very cool. We're gonna take a break.