DevOps in Banking – John Rzeszotarski, Dexcom

This is Textron tv. We are running, you know, running all of our containers. And this was back in the day where people weren't running containers in production.

Like this was not like a common thing. So for a bank to do it, it was, it was really gutsy Banking is a very traditional industry. Less than 10 years ago, you would've never heard the words DevOps at any major bank.

When our guest, John Zaki began pushing for DevOps at KeyBank, no one took him seriously. It wasn't until a major outage that they started to listen for A bank to be suffering for a day as a very, very big deal. People expect to have access to their money.

Join us as we dive into John's insights on the DevOps movement and banking. Looking at how the industry has changed and how to get your foot in the door at a major institution. Banking is such a traditional industry, and DevOps is more like newer age, Like right.

Methodology. How, how did you get into this? How do you mix these two together?

Yeah, and I mean, so we're going back, we're going back quite a ways. And it, you know what, I think one thing that, that always brings these stories, uh, uh, a lot of weight is a major incident. So when I was at KeyBank, we had a, we had a very large network outage, and it, it affected us pretty much the entire day.

For a bank to be suffering for a day is a very, very big deal. People expect to have access to their money, people expect to have access to their services, and those, that type of downtime is obviously not a allowed, and it's also regulated, right? So you'll, you'll, there, there will be a lot of regulation that'll come after major incidents.

So one of the things we did was we put together a, a group that went and studied everything that we did from a development perspective as well as like our existing infrastructure, like our network. And what we realized was that an organization like KeyBank way back in 2015, they had kind of grown up through acquisition. And so they had like pieced all of this together, which is what you'll get with a lot of these very, you know, uh, legacy type of, of companies.

And, but there really wasn't that taken back of looking at it, of holistically of like, wow, what, where are we and what have we done? And so there was multiple things that came out of that, whether it was network re-architecture, whether it was different chain swapping out our platforms, rev, making sure that we're not ping ponging traffic back and forth across data centers like mad. And then they, other big piece of it was this just concept of, of DevOps, right, of configuration drift that could still exist within the environments.

And really breaking that down to driving automation to be able to do the things you wanna do. I would say that's, that's kind of how we kind of kickstarted it. But what really triggered it was we, back then in, in 2016, we were going through an acquisition of a first Niagara.

And we want, we are also in the process of rewriting a lot of our digital platforms. And, you know, most banking companies, they, they will go after large companies that they can get support from, right? Because they don't necessarily get the best dev development talent, but that typically leads to more traditional development, traditional architectures.

And one of the things we wanted to do was we had, we had a couple, a really good architect, we had a really good director over our retail development at that point in time. And we said, let's go do something that's a little bit more cutting edge. So we actually started implementing Kubernetes back way back then, like 2016.

And the reason we found the value from it more than anything was it solved our biggest problem, which was trying to do zero zero down releases so that we could safely release while we, in the middle of the day while we were getting traffic. And we tested this and tested this and tested this on top of our new system. And it also helped us accelerate essentially the, uh, the release so that we could get this new digital system in right before the acquisition.

And it allowed us to react to that acquisition when things didn't go as well as we planned, but we were able to make major releases in the middle of the day without affecting our customers and ultimately, like really making that a lot more successful than, than, than it what it could have been. So you started this at KeyBank, correct? Yeah.

Was when you started the Yeah, we'll, yeah. Yeah, that's kind of a, that's where I really jumped into it anyway. But you know, when, when we went to P N C, obviously things have, you know, they, they had a somewhat of a, a group that had been really working to try to spark a lot more infrastructure automation, really doing things more around get ups on a release side.

The, the, the parts that we, that I, when I went into p and c, one thing that I realized that was a big difference from KeyBank was it's much more regulated and they're, they were a little more risk adverse. So at, in that aspect, what the thing we wanted to solve was really bringing risk to the table with DevOps. So if I'm going to do these releases and I'm gonna provide, you know, infrastructure and, you know, de development and testing all with a DevOps release, how do I then tie in risk management?

How do I tie in making sure that I'm providing all of the evidence that needs to go into a specific release so I can still get these fast feedback loops, and at the end of the day, I can still get to some form of a continuous delivery type of release process. Lots of times people don't really understand the details that go in those changes, and I think the, the secret sauce for a DevOps movement associated to a regulated organization is being able to bridge that gap between like what, what a lot of executive leaders think of a change versus what the actual technical changes itself. So you've, you've been able to go into these big banks and change them.

Have you seen, how has the banking industry as a whole adopted DevOps? I think it's actually probably done pretty well. I don't think you hear a lot about the stories from it because, you know, most, most banks don't like to share that type of information.

They like to kind of keep that under the hoods can maybe even consider it a competitive advantage in some instances. But there's a lot of really great talks out there, whether it's done by Whethers Fargo, bank of America, chase. I think a lot of organizations have really just said like, look, this is a, this is no longer a, uh, differentiator.

This is now basically a requirement for us to be able to move in a safe fashion for banking. It's all about resilience. I need to be able to provide high availability for all of my services.

And, you know, legacy ways of, of doing software development and release just does not, does not equate right. You have to do it in a more uniform standard way, declarative way. And I think that's really at the heart of what, you know, we, we try to engineer with a lot of our DevOps practices.

Mm-hmm. So can you talk more about your particular role as a, in these banks and what, what your piece, what you're part of the the DevOps movement was? Yeah.

Yeah. So when I was at Key, I was actually in the enterprise architecture organization and, but it was, it was kind of a group of rogue EA guys that wanted to go make a change in how we do things. Uh, the funny part is there's a lot of stories that start out like that.

I think it's an architecture group that has some very strong technical competent people that are just want change. They're just dying for change. They just wanna see it done differently.

They wanna see it done more effectively and they're tired of, this is the way we do it because that's the way it's always been done type of statements. So that's kind of where I came from, which was this, you know, enterprise architecture group, which I have never gone back to since I've, since we've kind of left that and, and, and, and then I kind of led up a part of the infrastructure organization at Key in order for us to be able to really, you know, build and maintain those types of solutions in a new way. And then when I went to p and c, I was kind of hired in to run, you know, various large parts of the organization on the infrastructure side.

And everyone that is doing infrastructure inside of a banking organization understands the pains of making sure that everything has to be reliable, but also understands the pains that it's very hard to change, change the paradigm for how things are completed. And, but they want to, right? So all they need is a catalyst.

So, so I feel like I've been able to somewhat be that in in, in most of the roles that I've been able to play in banking. Do you have any like wild, crazy like DevOps implementation stories that you're okay with telling people? Yeah, I mean the, the, the one I always like to tell typically was the KeyBank story that I mentioned About, You know, really that was, it was, it was just such a crazy acquisition that we had.

And, you know, the day, the first day that we were migrating the new customers over, it was an enrollment process and it didn't go smoothly. And so we said, all right, let's go make a bunch of changes to the front end of the application to make the ex the experience and much more improved so that users were able to get through the flow. And, you know, we started making changes in the middle of the day and we had just basically released this new system that was running essentially Kubernetes, that we were running, you know, running all of our containers.

And this was back in the day where people weren't running containers in production. Like this was not like a common thing. So for a bank to do it, it was, it was really gutsy.

And I, I, I mean, I'll tell you, I was nervous sitting in the command center making, praying to make sure that these containers came back up. But it was actually incredibly successful. And we put so much work into the things around unit testing.

We put so much work into the things around the automated a p I testing, we felt very confident in all of those specific services. DevOps clearly is like a part of the banking industry now and people who want to get into it. How, how would they, what skills would they need?

What would they need to do to go into this industry? Yeah, so that's a really good question, right? Because like the, the best part is like you'll see DevOps engineer all over the internet now, right?

I'm gonna hire a DevOps engineer. I think most of the time that's like looking for either some, somebody that knows how to like, write really good, get actions or knows how to do Jenkins, like left and right. It feels like that's typically like your, what's kind of classified as a, as a DevOps, which is just funny because I feel like the DevOps movement was like trying to break that down and say like, no, that's not, that's not it.

It's, it's more of a verb, it's more of a thing that you're trying to do. But I think you need all aspects really, right? You and, and not one person needs to have to be able to be qualified to do all those things.

But if you're very, if you're a, a solid developer or engineer and has a, have a great way to build fast flow as part of pipelines, that should be great. It doesn't mean that you, that, that, that you can't be a great Java engineer or a great React engineer that's gonna build business services for your applications because typically those end up making the best pipeline engineers at the same exact time. I, I don't know if, if you've heard this or not yet, but Kubernetes has kind of become a bad word.

It feels like lately in, in, in like the last few conferences I've been in, it's been used as a means because, because companies have come in and said, well, we're moving everything to Kubernetes. So just this top-down push of saying like, all these things have to go out on Kubernetes and Kubernetes is gonna solve all of our problems, even though that's not a business strategy, that's got nothing to do with your business strategy. That's just got to do with like running infrastructure in a certain place, right?

And, and you're moving away from, you know, instead of like figuring out exactly what's the right thing for what that business strategy actually is. And in this aspect, I say another role that you have to hire for is some, is folks that understand things like basic networking, understand things like basic configuration management and, and, and how to use a declarative way of building and managing infrastructure. And so I think, I think at the end of the day, I, I don't necessarily go look for the guy that has all the, you know, potentially Kubernetes certifications like certified Kubernetes administrator, et cetera, unless I ha, unless I actually know that I absolutely need that type of a position.

Instead I'm looking more for maybe a little bit more generalist from a development perspective on the infrastructure components themselves. People that have expertise with things like Ansible or Terraform or Cross Plane or those types of things would warrant for me their ability to be curious, their ability to learn. Cuz the tech's always gonna change.

It's always gonna continue to change. And so it's really trying to stay in front of it is my honest opinion. But I think you need a little bit of all of this, right?

So I say all of those rules out there are all, to me, DevOps, and they're all important and, uh, we didn't even touch on testing, but testing's another area that once again to provide true great test automation, you typically need really good software developers for enterprise testing or, or other types of roles like that. But you also need those infrastructure backgrounds that are gonna work with them so that they can run those tests on every commit. I could tell you that I know a lot of banks are out there that are hiring for those types of positions.

And I think any one of those to me would be ways to jump into doing things around DevOps for, for these types of organizations. So that's how you get into banking. Why do banking though, as a part to any other comp technology company out there?

Any other company in general that's doing DevOps? Well, the, that's a really good question, right? And I think it's, it's, it's like anything else.

So number one, banks are great companies to work for. They are, like, they will, they, they typically have very good entry, entry job position pro programs that will rotate you through the organization. So you can learn parts of the security aspects, parts of the development aspects, and then on top of that, they do a lot for their communities, right?

So I think that's another really big aspect to get into a large organization like banking. You know, from a development perspective or from a technical perspective, it is one of the biggest challenges. These are some of the most complex systems you'll see, and in most banks you'll see a lot of legacy systems that you have to design through or you have to design for.

So I think it's a, you'll, you'll learn more about things around security, you'll learn more around things around the complexity of true technology and architecting through that complexity and trying to pro provide a simplistic solution. So to me, from a technical challenge perspective, it's a great place to learn and it's a great place for you to grow your skills. And you know, they, I absolutely do think that that banks will absolutely reward you for, for making su for, for coming in and trying to change the norm, right?

And my expertise, putting myself in uncomfortable positions in the banking world was, was always kind of more of a celebrated as success. And so I, I would definitely argue banks are a great place to go. And then on top of that, one really good thing to learn is how to build solutions in a regulated industry, which can pay dividends when going to a startup where they typically don't.

Right? A lot of, you know, a few of the startups that I've had a chance to work with, there's, uh, giant gaps around designing for high availability, giant gaps in, in, because it's more about getting the product and getting the product out there and getting the pro and getting feedback on the product and that's great, but at some point it does need to turn and making sure that you have all of the right regulation, all of the right security, all of the right change processes that need to be in place and the change management that needs to be in place so that from the, from the time comes, you're able to meet SOX compliance, ISO like this 853, all of the, the compliance that will eventually hit you likely as you grow to be a larger organization. So I think having that expertise coming from a bank is a pretty big deal.

Do you Have any like wild, crazy like DevOps implementation stories that you're okay with telling people? Yeah, I mean the, the, the one I always liked to tell typically was the key bag story that I mentioned about, you know, really that was, it was, it was just such a crazy acquisition that we had and, you know, the day, the first day that we were migrating the new customers over, it was an enrollment process and it didn't go smoothly. And so we said, all right, let's go make a bunch of changes to the front end of the application to make the ex the experience and much more improved so that users were able to get through the flow.

And, you know, we started making changes in the middle of the day and we had just basically released this new system that was running essentially Kubernetes, that we were running, you know, running all of our containers. And this was back in the day where people weren't running containers in production. Like this was not like a common thing.

So for a bank to do it, it was, it was really gutsy. And I, I, I mean, I'll tell you, I was nervous sitting in the command center making, praying to make sure that these containers came back up. But it was actually incredibly successful and we put so much work into the things around unit testing.

We put so much work into the things around the automated a p i testing, we felt very confident in all of those specific services. I, I really, I really do appreciate you coming on here, John, and, and having us talk with me. I I do Really No worries, Amanda.