Dwayne Bradley, Duke Energy Corporation | DevOps Experience 2022

Hey, good morning, everyone. This is Dwayne Bradley. Welcome to the devops virtual experience.

I'm glad glad everybody joined in joining me today. So I want to give a little bit of a presentation today to talk a about what my group period Duke Energy is doing and what we kind of see is, you know, a distributed intelligence Vision that we can lay over top of the electric grid in the future. And give you everyone a little bit of a background on Duke Energy because most folks may not know Duke Energy.

We're one of the largest investor on Utilities in the United States where headquartered out of Charlotte, North Carolina. I'm in a microgrid lab that our group has that's just outside of Uptown Charlotte where we get to actually do a lot of experimentation and stuff. We're one of the largest investor owned utilities.

2 million customers that we serve from Florida all the way up through Indiana that you can kind of see on the service territory map there. 2 million customers is kind of the endpoint. It's not really the number of people if you scale that up.

It's a roughly 25 million people that we provide electricity services to in the United States, which is about 13% of the population and you can see that we cover six different states. One of the things that we're really doing is we're trying to push, you know forward what we're doing in the electric energy industry and where we're going to take things and there was recent presentation that our CFO Brian Savoy made. Interview with Bloomberg.

There's a link there that you guys can follow and watch that interview. It's pretty good. But he announced that we're going to be investing $145 billion dollars and that's billion with a b as though.

It's a very large number of the next decade to upgrade, you know, our electric systems, you know, that's from everything from generation to transmission and distribution that we're going to be, you know, providing upgrades for over the next next decade. We also Duke Energy also has a natural gas. 6 million gas customers that cover, you know.

Five different states in Tennessee got included in that mix there as well. com and look at look at what we provide. If you look at if you look at our commercial and regulated Renewables, we actually have service in about 25 States across the us.

So we have you know, a very large footprint we're pretty, you know producing a lot of power we produce and you know interact with our customers and a lot of different areas. The group that I work in we're called emerging technology office. We're basically a small research and development group that got stood up inside of Duke Energy and around 2008.

And our kind of goal is we're chartered to look at like that. Five ten even 15 years in the in the future. What are some of the technologies that you know, our company needs to be aware of that may be coming down the line or what are things, you know industry trends that may that we may need to be on the lookout for that could impact our business operations.

We also look to say is there different ways of doing things that we've done in the past that moving into the future. We may need to look at how we how we do some of those things to better serve our customers. We you know our group we're not really a group that actually builds real products.

A lot of what we do here is building out proof of Concepts, you know doing that research that early research to see does these ideas actually make sense or not. Is it one of those things that our company would need to you know, dig a little bit deeper in so we'll do a lot of that research. We'll you know, figure out what does work if it's something doesn't work, you know, we actually tell our business customer's like yeah, we probably you know, don't need to pursue that or we need to wait a while to see if that market matures a little bit but we do everything a proof of Concepts from Hardware to you know software.

And actually we've actually even built some specialized robots for doing some of the things in our industry that you just wouldn't find elsewhere. But we also we test, you know vendor products as well. We you know vendors will bring products to us to say.

Hey, can you guys, you know take a look at this does this make sense? You know, they'll bring it to us before they actually even release it to the public just so that we can you know try out and see what you know, see does it work does it actually solve a problem that we that we really have we also we look for things that maybe you know a mature product in another industry that wasn't really targeted for the electric utility. In those kind of whether that software or Hardware bringing in to kind of test it and hand fit into our mix in the future.

We're getting to a little bit of the history of how we actually got to where we're at today. We really started on this whole distributed intelligence notion back in 2013. That's where you know, we started seeing things like, you know, how do we need to do interoperability between devices?

How do we do, you know compute at the edge of the electric grid, so we started doing some of that experimentation very very early on one of the things that you know, when we started digging into our existing systems, you know, we realized that we kind of have that castle and moat You know vision of everything's where you know, we try to surround everything with firewalls and you know virtual private networks and you know things of that nature just so that we can kind of keep you know, the grid secure in how we actually use it today. You know, we and 2015 at the Sands. I see a security Summit.

We did a demonstration where we kind of mocked up a substation. That's the rack that you see there in the picture of a four feet or substation and we had a like a small scada system on the side of the stage. We proved out.

It's so super easy. If you can get into the network, you know, if you can get past the castles in the Motes you can kind of own the network and we took some off the shelf, you know open source software. As freely available to anyone and we you know did a substation break in where we got into the network.

We plugged into the network and we basically took over all the devices inside of that Subs that virtual substation that we created just to show that you know, the what we do today is very, you know, it's very secure from that standpoint, but if you can get in you can kind of own it. So we doing some that kind of research around cybersecurity. in early 2016 we helped ratify new standard for the grid industry.

It's called open field message bus open FMB and it's a way it's a interoperability framework. It's really about how we can go through and put common abstractions on a lot of the things that we do in our industry and to make that interoperability a reality going forward and we were continuing to push on that whole notion as well. We've done things around testing passive monitoring tools, you know where you can take these tools and plug them into a Spam port on your network and you kind of see all the traffic that's going on to help you dig a Little Deeper to you know, see.

How is your how is everything functioning? Is it working the way I expect it to well. These tools work great if your application speak unsecure protocols because these tools are really about like Wireshark on steroids where they can really dig down into the packets and see what's going on.

But that is only good when you're using protocols that don't do cryptographer encryption over the wire. So, you know, it's one of those they would still help you tell you know, what boxes are talking what box but that really doesn't, you know provide you any real insights at the end of the day because our our industry has a real problem with really trusting the device that we put out there do we know that it's the device that I put out there and right now that answer is you know, pretty well. No, we kind of put devices in our Network and when we put them there we just natively trust them because we put them there.

But we started looking into you know, what does it mean to actually really identify the things that's on our Network and that's where we started digging into. You know, what if we had, you know TPM chips in kind of everything that we deploy so that we can guarantee that we're talking to the device the that we're really that we think we're talking to and so we've been on this whole notion of how do we you know, take some of these things and do things better. Well, we've also, you know, we started researching tools like kubernetes and we did fairly early on especially in our space where kubernetes was still kind of this thing in the cloud.

Where you know it was before kubernetes started being drugged down to the edge, you know, where you have lighter weight distributions, like k3s micro Kates and things of that nature. We started looking at it very only early on saying hey, can we use kubernetes to help us deploy applications out near the edge electric grid if we can do that we can you know, use better protocols. We can secure things much much better going forward.

So, you know, we started pursuing kubernetes. What does that look like? With our ultimate goal of getting to the point where when we put something on our Network.

It's zero trust. We always have to cryptographically identify every actor that's on our Network so that we can know that we that we trust it and that's going to be a real driver, you know, especially when we get to this world where we have distributed energy Generation all over the place with things like solar energy storage wind these new technologies that are coming along and really ramping up to be deployed. Give a little bit of a different view into some of that early work when we started looking at things on our and our industry.

Everything is really we have all these siled systems, you know, where we deployed devices out in the field. A lot of times these devices require are their own networks. They require their own head in software in the back office in order to be able to talk to these devices.

So you wind up with all these devices out in the field that have a lot of functionality that wants getting used for one use case. and you know, we we do what these things where we you know integrate all those with like Enterprise service bus in the back office today, but those kind of Integrations are very expensive their time consuming, you know, and when and every You know utility has their own mix. So what he's head-in systems are in the back office.

So there's no real interoperability at that point and it makes it difficult to to use those systems. Well, what we envisioned is we really envisioned to kind of flipping that model upside down a little bit and instead of having these head end systems pulling data in from all these devices. What if we turned around and we use published And subscribe to have the devices published the information that they have and we did that.

We've kind of layered on adapters on top of that so that we can take those Legacy protocols and turn them into publish subscribe. But we also distribute basically compute nodes. Out into the field where these you know new adapters.

They talked to the compute nodes. That way we can have a peer-to-peer sharing between you know, all these devices out in the field. It gives us a place to actually deploy applications to and by doing that we could also do the same kind of thing these head end systems that these vendors provide the vendors could subscribe to these messages and everything worked the same from their standpoint, but it gives us that way of really having interoperability amongst all these devices in the field.

With what Duke Energy is, you know, we have some very high level goals that come from like our board, you know level our c-suite on down where we're really pushing forward with the notion of doing 50% CO2 reduction in Generation by 2030. That's only seven years away. And so that's a lot of work, you know to change up our Energy Mix in seven years.

Well in you know by 2050 we're saying that we need to be Net Zero across the board that every you know, we don't need to be producing any CO2, you know coming out of Our Generations. But in order for us to get there, you know, it's one of those things where we're having a lot of these distributed energy resources that are popping up all over the place. It's not this centralized generation that we have that we have today where you have the large-scale generations and you know nuclear hydro plants, you know coal plants natural gas plants where they're generating large-scale electricity and pushing it through transmission through distribution all the way to the end houses.

That's you know a model that's it's not going to go away because we need some of that base load generation from those plants, but we're in this whole notion now where we're having energy being generated everywhere. There's people with solar on the rooftops. There's businesses that are putting solar up on top of their roofs so that they can you know, reduce their energy usage so that they can, you know have more clean energy and we see that that is kind of a wave going forward where you know, these kind of pockets of islands that can kind of self manage themselves that they you know, these like a microgrid we can kind of, you know, have these microgreens popped up all over the place.

They can kind of be autonomous they can you know, do what they need to do because they have their own generation mix there, but they can also be tied to the grid, you know to have that stability when you know, the Renewables aren't aren't available. There's some companies that are already kind of pursuing this notion of you know, kind of distributed microgrids everywhere. One of those is a company called block energy where they are.

It's our development of product where it's along the lines of a community microgrid where you have a community scale, you know energy storage where that energy storage is also connected to the grid so that it can you know pull in power when it needs to but then there is, you know, all the homes in the neighborhood have solar on their roofs. It's contributing back to the whole neighborhood and it's, you know, providing AC and DC power into into homes. So we're starting to see, you know, some of these kind of companies come to the fruition and they're they're moving things along as well.

Getting back to the whole castle and moats terminology that I used earlier one of the other things that our group really kind of talks about it from a networks perspective from an application perspective and our industry. It's everything is kind of about like a coconut. It's very hard on the outside.

But if you can get inside where all the good stuff is at you can kind of on it. So, you know, we've kind of have that. That mentality of you know telling people it's that's how we operate today because that's the tools that that we have at our Disposal today.

We're still using a lot of Legacy protocols in our industry. All these Legacy protocols like modbus dnp3, there's really no security on those protocols. Everything is kind of clear text on the wire so that you know, you can see what's going on.

You can kind of take over those devices you could spoof data very easy. And we're really we've seen seeing all these Trends where we need to say. I need to actually, you know do distributed until I need to push forward for the grid.

I need to be able to you know, put things out on the electric good to let things kind of self-manage but that's you know, a very different different way of thinking than what our industry normally normally goes through. So we we have this whole notion, you know, why are we really wanting to push forward distributed intelligence and that really comes down to the fact that we think that the grid needs to be operated differently in the future. And in order to do that.

We need to start pursuing these things like, you know zero trust we need to make sure that the devices that we're talking to are who we who we think they are. So we're pushing on that whole notion we're pushing on this notion of being able to you know have applications deployed anywhere on the electric grid, we will be able to deploy those applications to the actual location that need some instead of having some centralized system in our back office, you know controlling those In order for us to do that though, you know, we've really got to make sure that we get these common abstraction layers we get these, you know these interfaces so that distributed intelligence actually makes sense going forward and that the picture that you see there. That's one of my co-workers that's that's actually his son on top of on one of the top of the rocks in the Swiss Alps so that they they climbed it.

That's what we were trying trying to shoot for. We're trying to get to that peak of where we know how everything is actually operating and we can actually guarantee it. If you look at this picture, this is kind of a picture of the feeder that's coming out of the Rankin substation where our microgrid is located.

2 megawatts of solar on their rooftop. So we're we use this as our our testing ground for, you know doing some of these new things that were pursuing. And this is kind of how we would like to see things look in the future where we have you know clusters of these compute nodes running all over the grid that we can control and we can send applications to to have them self monitor have them self-operate have them, you know, join in the network to you know, help out basically help out their neighbors.

And you can see where we use a lot of these open source tools that most of all of us know about today, you know, we use kubernetes for deploying applications into our microgrid. We you know use get getlab 4, you know, building our applications. We use Fleet for managing those applications being distributed into our clusters, you know, we're using, you know, Prometheus for monitoring to see what's going on under the hood.

We use a lot of these open source tools that grew up in the cloud, but we've kind of pulled those tools down to where we're like, I've got a different use case, but they actually really still work really well. If you look and dive down a little bit farther those compute notes that I'm talking about that are basically one of the members of our clusters. This is kind of how we see these things, you know, the layers of these things being built, you know, we're very much want to make sure that I can trust that physical device that I'm talking to that really gets back down to that TPM chip thing.

You know, that's kind of us an essential piece of Hardware going forward that we see And that way only this trusted Hardware can actually get onto our Network. If we've got trusted Hardware we can make sure that we have trusted applications that we can deploy on that trusted Hardware. So we can do the you know, the peer-to-peer sharing in the field all of the you know, the certificate management, you know has been one of those real pain points in our industry where we really have a certificate for a device.

Well, if we're deploying applications everywhere, we've got to manage more certificates and but that's one of those where we're using tools like spiffy Spire to generate certificates for everything every application that we that we deploy. And we very much believe that you know, instead of deploying, you know, this five 10 20 years long certificate that we can use these new technologies and new tools to have you know, certificates that rotate, you know constantly, in fact, the microgrid application the kubernation that we have operating our microgrid every application that we deploy out there. We're rotating certificates once an hour.

Just because we can it's not one of those we may never do that in a real production scenario may not rotate on that fast, but we're doing the proof of Concepts here so that we can show others that yes. This is something that that we can do. Once we start enabling, you know, these compute notes being placed everywhere.

If you look at some of the communication patterns that you see this that we actually deploy, you know k3s is our kubernetes. That's what we're using today. Inspired those two components are kind of given on every box that we deploy.

And then we start deploying all of our application containers all those application containers talk to Spire to say hey, I need a certificate then. They also at that point they can start communicating between each other. But all of the certificates that are generated from Spire are actually cryptographically tied to the TPM chip that is on that actual box so that if I'm communicating if I see data coming from that open FMB adapter on the box, I know that it's good because it's it's certificate.

It's tied to the TPM on that chip or on that that compute mode. If you dig down in a little bit different View. Inside of these compute nodes, you can see the we have Spire our Spire agents running there.

We have K3 s that's running there every pod that we deploy gets an Envoy sidecar today. Envoy already knows the API to integrate with spire. So we have the envoy as soon as the pot is deployed.

Envoy reaches out to Spire says hey, I need a certificate. Spire will generate a new certificate hand it back to Envoy and at that point that container can actually talk and by doing that everything is a mutual TLS encrypted between all of the pods that are being deployed on that note. So not only are we encrypting traffic coming on and off of that compute node, but we're also can encrypting traffic inside the compute node and everything that we deploy at that point doesn't even have to know that encryptions in play.

For instance like the open FMB adapter that you see there gnats have published subscribe. tool that we use neither one of those actually knows encryption is a happening in between those actors. So it's a way of us, you know layering in the security and making it really seamless and kind of hidden from anything that we actually need to deploy.

a couple of visuals, you know a couple of pictures here so that everyone can kind of see the scale of things that I'm talking about. This is the actual microgrid test lab. That's just outside of Charlotte, North Carolina.

I'm actually sitting in this this building right here right now is I'm talking this morning. But you can kind of see there's just there's a few major components that we have here up in that Northern side there. You see those are that's really the micro grid.

That's the components that you know in Encompass the microgrid all the the different pieces there. We're also building a DC based microgrid this, you know, the solar panels for that here. It's actually inside the microgrid lab inside this corner right here.

So, you know, we're doing experimentation of what it what does it mean if we keep DC power if we don't have to convert everything back to AC then we can keep everything DC we can make things more efficient. 5 mile feeder extension here at our facility. With some of the experimentation that we do we can't put these things out on a real feeder yet like that ranking feeder that you guys saw earlier.

You can see that we can't deploy things on there because there's several thousand homes in businesses on that. We don't want to put those customers in the dark with you know, some of the experiments that we're doing, you know, if we put ourselves in the dark here. Okay fine.

That's not that big deal, you know, we can get things going again, but we don't want the impact our customers with some of the things that we're doing. You can see there's a little different view of the actual feeder extension. There's a lot of poles that's out here in there in the backyard where we have you know, that one and a half miles of electric distribution Line running there.

We have equipment that's installed on all these poles. We're installing load Banks so that we can simulate what the load would look like on a test feeder so that we can do experiments with with software. If you dig down into the actual components of the microgrid, we have a six node kubernetes cluster running in our microgrid every, you know, Orange.

Box that you see there that is one of the locations where we have one of our compute nodes. We've got a compute node at the solar. We have a compute node over into the grounding the battery battery energy storage system.

There's compute node there. The main point of common coupling the PCC is in that microgrid islanding switch. That's where we can you know disconnect from the grid and actually continue operating here inside the lab.

But we actually have you know kubernetes running on this so that we are actively deploying applications via kubernetes to all these compute nodes in order to one get data off of these devices to encrypt all of that data. And so that we can do you know, we can make sure that we're having a good security posture here. But we're also layering in the applications that kind of manage the microgrid.

We can deploy those anywhere that we want to in this cluster. We can have them redundant. So we're actively you know, building out these proof of Concepts today so that we can show what does this distribute intelligence future actually look like, you know, we we bring folks into the lab a good bit just so that we can You know give folks tours and let them see the things that we're doing.

We bring in not only Duke folks, but we bring in other folks from outside the industry and all the time just so that we can say these are the things that we believe need to be pursued in order to get us to that point where we can reduce our carbon emissions where we can manage the micro or we can manage the entire grid much much better. To you know, some of the things you know that we've kind of talked about today. We're really on this whole notion of How do we put you know build a platform that can provide zero trust out of the box that you know, we can deploy applications where we want to we think we've got a pretty good proof of concept running here with that functionality today.

It gives us very good situational awareness of seeing what is going on in our microgrid where you know today, you know, If that's a very difficult task that you can't always see how things are functioning until things don't work. So. We're really pushing on that notion of building out.

What does that platform need to look like? We are you know very much driving this whole notion of interoperability so that you know, I can build an application that works for Duke Energy with our favorite mix of vendors. I can take that application and run it at another utility with their own group of vendors.

Maybe same kind of Hardware, but I can run that application without actually even having to touch it because it will it's running off of these. Off of this interoperability layer that we're putting together. But the thing is, you know, I said earlier, you know, we're gonna be spending 145 billion dollars over the next decade but things are not going to change as quick as we need to out in the field.

You know, there's a lot of Legacy devices out there that are not going to be ripped and replaced anytime soon that you know that we still need to we need to use those we need to use those assets. But if we can put these interoperability layers on top of them we can get to the Future that we think that will that we need, you know while still leaving those those devices in place. And it's really we think you know, we need more peer-to-peer sharing where that's what we one of the things that we don't really do today.

Everything today has to be pulled back into centralized data and centralized systems in our back office and then turn around and and you know, send set points back to the grid to change things. We really feel like that. We need that peer-to-peer communication to let these new distributed energy resources, you know communicate amongst themselves to pull off what we want them to do.

We can kind of send them those rules and say you behave this way, but you need to figure out how you gonna how you gonna get there. If we can get you know, these things put in place. It really gives us that you know, that point of where we can have distributed Intelligence on the electric grid and that's where we're really trying to to push right now to say, how do we how do we get there?

And we really think that you know, what we've done here in our lab really pushes not only Duke Energy to to get to that distributed intelligence future. But also, you know kind of set a roadmap for other utilities to follow as well to say these are the kind of tools. These are the this is the way that we need to start thinking about how we're going to manage the electric grid going forward.

And with that I want to thank everyone for your time. I appreciate you attending today. I hope the rest of your day, you know at devops experiences good if you want to reach out to me, you see a couple of different ways of contacting me there, you know, we will be providing the slides later.

So that invite wants them we'll be able to to download them. But, you know, feel free to reach out if you're happen to ever be in the Charlotte area, you know, let me know we do tours. We bring folks in, you know to show people first hand here, you know, take them around the lab take them, you know around the microgrid to see what these things look like a lot of times, you know, like saying, you know, seeing is believing, you know, it's so we we really try to pull folks in to be able to do that and show them, you know the scale of some of these things that we're talking about.

So with that being said, I'm going to wrap it up and I sure do appreciate everybody's time today.