Aaron Rinehart & Brian Mengwasser – 5G is a DISH Best Served Reliably

Welcome to our session with myself Aaron Reinhardt. I'm from verica verica is a company that that was created by the founders and creators of chaos engineering and Netflix United in health group. I am Aaron Reinhardt.

I'm the co-founder interesting advisor of Erica and and I'm joined by well I think is one of the most exciting products going on in Silicon Valley and the world of tech is Brian malacher Edition talk about a little bit your background and sort of what what's going on a dish. Yeah. Thanks Aaron.

It's great to be here this text wrong event even great to be here to be with Aaron in our partner Erica. So I'm Brian mckesser from Dish Wireless vice president head of marketplace and at design and you know, you may be familiar with Dish Wireless as a television company, but I'm here on behalf of a new division within dish Wireless within dish called dish Wireless, which is about building the future of connectivity. We're building a virtualized 5G Oran hyper distributed Wireless Cloud, which is a bit of a mouthful.

But what does that actually mean? Right? It's it's a fundamental upgrade of the cloud and an overhaul of what we've seen previously in telecommunications.

We're turning everything into a software-based solution. We're running everything as possible and agile cloud-based manner. We're pushing the limits of both the software the Telecom stack the standards and the cloud in order to make this happen.

And so, you know Aaron we were talking about you know, the way that I see this is actually transforming the way that we move data around in the world from you know, where where we've been previously which is maybe like the era of sports horses and carts into something that's much more modern the modern, you know railroad that's gonna power the next Industrial Revolution. I completely actually, you know, they work the work going on a dish is to for just that the engineer in me. It's just appeals to me just because you literally are like revolutionizing Telecom.

I mean, it's moving from the world. We're really kind of struck me is the exciting product, you know, both from it basic engineering perspective and a security perspective was just the scale and complexity but also moving from a world of physical based radios and attacks like to you know, running on software and then you know, like the slicing stuff you're doing with Amazon and that's that's exciting, you know be able to for me my prior life, you know working as a chief security United Health Group like that. That's a great capability to be able to slice off.

Yeah 5G and create might be my own carrier in a way right? Like that's that's useful like when you take medical devices iot drones, I mean, you know, you know, you want to talk a little bit about some of that in terms of like what Us what is open ran like like I'm not sure everybody here maybe knows what that is because I am really excited when when I sort of learning about that that I'm like wow, this is a product. I want to be a part of yeah.

Yeah, there's a lot to unpack here. I mean that the technical nature of what we're doing is super ambitious and let's let's get into a little bit actually, but before that though, I do want to explain the why right? Why are we even bothering to create the you know, the modern railroad maybe horses and carts are working just fine, right and It's because we fundamentally believe in the next generation of applications that require movement of data or movement of intelligence or you know generation of closed loop Solutions in it in a much more complex manner than we've seen before that will drive, you know the next generation of technology for us.

so You know, how can we you know, how can we hope to kind of enable those applications? If we don't first modernize our Tech stack because you know what we saw in Prior generations of telecommunications is we're we're actually behind we're very slow right? We're getting you know into you know back in the early 2000s.

We're getting into Rich text messaging and you know different ways that we want to use Communications for phones, right and we couldn't really even keep up right we had to build kind of around, you know, the tech stack on top of it. We kind of try to hack it in order to make it work and that's just, you know, based on how Telecom was was created in the first place. But now we want to move forward into the next era where we're gonna have so many more complicated use cases so many more, you know, revolutionary technologies that you know our machine or machine enabled and we're we're smarter this time around hopefully, which is that we don't want to be an impediment to that progress, right?

We want to be a catalyst and enable her to that progress. And so In order to to achieve that why right achieve the vision of becoming a catalog Catalyst for the next generation of applications. We have to you know, look internally first right and reinvent our Tech stack and that's where you know, Aaron to your question, you know, we come with some of the key technical tenets of what we're doing.

So virtualization completely radical Concept in Telecom, right? Because Telecom has always been you know proprietary embedded software on proprietary Hardware that was tested sometime, you know five years ago and it has no none of what we expect from a modern stock. Right which is agility, you know, observability performance and so forth.

um Open rant. So I'm unpacking my my yeah, what is that? What is Oran?

What is open? Right. What is it?

What is that? Yeah, so open RAM and the key word here is open which we extend. Well beyond the rant but the idea is that rather than getting a complete solution from one vendor.

Let's say for the ran where you have radio and you have software that's talking to other software, right? That's kind of a tightly integrated stack. We said no we want to completely break apart all these interfaces so we could get and let's start using more modern terminology.

We could get application a from one source application B from another rather than having to get the entire stack from one place. What does that do for us? Well, it changes the dynamic so that we can get best of breed again.

It's getting back to one of our goals, which is the agility that we can kind of bring in new Solutions. We can change them we can upgrade we can do Canary upgrades or rollbacks in a much more open ecosystem. It also helps us cultivate the ecosystem because now we're not signing, you know, five billion dollar deals at a time with huge companies right with huge RMS that have Working on this, you know for forever right?

It's opening the door to a startup who has an idea and they maybe they're going to refactor a network function. Maybe they want to bring some machine learning into the into the network, you know with this open interface concept. We've opened the door for them to kind of plug in test with us, you know in a live setting without having to do, you know, 10, you know, 10 years of re-engineering or billions of dollars of reengineering, you know, we have that that modularity I think the last thing from a tech perspective, that's really Transformative and frankly challenging for us is how do we use?

Or how do we build this system in a distributed manner? So obviously we you know, we want to we want to Leverage The Cloud as much as possible but turns out the cloud is actually pretty centralized right even if you think about all that, you know what we work very closely with our partner AWS all the AZ and lzs that are out there, right that's not really enough for us because as a result of you know, latency anytime you're trying to communicate between an iot device or a drone or something like that. We're talking milliseconds of latency in order to authenticate and communicate with the radios.

So we need to actually bring that software local to all the debate all the devices. So what we've really done, you know, as a precursor to building the network is build a cloud, which is I think the largest and most complicated hybrid Cloud out there where we have thousands, you know scaling to maybe 20,000 individual many data centers that we've distributed around the entire country and then we've connected them to let's say a hundred local data centers or lzs. And then every you know, AZ region of AWS gcp is or, you know, every other Central Cloud so, you know building this Cloud fabric running our applications on top of it, you know in it obviously completely virtualized but very open manner is you know, how we had to look internally to say.

Well we want to be a modern stack and we want to be a catalyst for future Innovations. How do we need to revolutionize what Telecom is done and upgrade the cloud and you know what? I really you know what I really love about like what you just said in the work is like yes, it's bold.

Yes this audacious. Yes. It's It's it's some people may see it as a radical change, right but it's like but it's actually somewhat pragmatic and generic right?

It's achievable. It's such I think I use this word before it's achievable engineering. It's not some pie in the sky, you know, like, you know, you know pontification of like we're gonna like something that doesn't exist.

Right? No, it's this is like the components are now there right to do this, right and it's like, you know just be telecom industry that like you said doesn't like traditionally doesn't change that often to but to now have an ecosystem where startups and come in and they can they can build new kinds of radio functionality maybe improve parts of it. No one expected and maybe take build different offerings and that are maybe Telecom base or radio base or I don't know new ecosystem of services and keep abilities that you know, the tech world needs.

I mean this this just kind of brings a new building block. General to work with but the fact that like what I love about dishes approach to it is it's open right? It's like it's designed.

It's like you're not just building it to provide a base service you're providing it as a building block, right which to me is super exciting. That's the most exciting part for me. I think the most revolutionary, you know, that leads us.

On a path towards our vision of being this Catalyst because we want to be exponential in multiple ways. Right? We want to embrace and build a huge developer ecosystem that can do things with us.

you know at a scale that's unimaginable today and in Telecom, I mean, but you know many of the folks who work with in a Partners here, you know are used to like maybe 10,000 strong Engineers of developers working on Telecom, you know Hardware or software but you know the developer community that we all work in now is millions strong, right and we want to take something that you know, maybe we design we tested for like two or three years, right? We want to bring that that time scale down to you know, certainly days if not, you know minutes or seconds, right? So, you know getting to kind of an exponential rate of change because we've got we've engaged a much broader ecosystem.

We're shortened the time frame we've opened everything up right. We're really trying to cultivate this sort of yeah exponentially growing Garden of new Technical Solutions, whether it's to improve the platform or to develop new Innovations on top That's that's super exciting. You know, I'm I you know, it's it's, you know, the founder of me and the engineering me I just love the idea of like, I'm the I'm the kind of person that sees a blanks you to paper as an opportunity, right?

But to me, this is like, you know, this is like I could like you never know what people are gonna do with something like that. That's what that's what's the cool thing. I know that's gonna happen.

People are gonna do things with the the that infrastructure that you wouldn't have expected to be exciting, right? That's what I'm looking forward to see, you know, let's let's pivotal a little bit about like because of the work we're doing together between you know, where this started I guess with, you know with with Erica's doing and what this in terms of this new ecosystem. Let's hope what are some of the challenges right in building this and maybe we could talk I can explain sort of maybe what bear is doing maybe to help out with that or maybe yeah, too.

So so the, you know all the EXC Of this is paired with the with the daunting mix right of a great great. We want everything to be open and you know agile is aggregated distributed and and that scale right but that comes you know, its own set of interesting challenges that that we have to tackle I think. You know, the the biggest one for me is is the scale right?

Is it distributed nature of it? And it's definitely a situation where it's going to get worse before it gets better. But that's our you know, that's that's our contribution right to really develop and figure out how to make you know, how to evolve the tech stack and how to make this work.

But if we think about the scale, right we put we put numbers to it we're talking about like I said Thousands growing to double-digit thousands individual data centers in different locations. Right and let's think about that from a latency perspective. Right?

These are distributed around the country right different temperatures different climates, you know, we have to we have to worry about not only the physical distance but also temperature distance and other things that don't normally come into play when you're thinking about you know, okay and you know a software based application in the cloud now, the cloud is is in the real world, right which has real world implications. We're talking about thousands of clusters right that we have to manage and you know, update dozens of applications all coming from different locations. And again, our our goal is maybe it's maybe it's hundreds of applications coming from, you know, new startups developers that we want to kind of plug in and test live in the network.

And just a sheer volumes of data movement either for control plane or data plan. We're talking about exabytes. per day currently scaling to much more than that, you know in the future as we as we target getting 50, you know plus million up to you know hundreds of millions of devices talking to the network so that that scale right brings another level of complexity and I think the only way that we manage this is by Benefiting from from the attributes of the cloud that that were leveraging.

So for the first time ever we can actually get observability of each component of the stack independently. We've never had that before in Telecom, right? It's all been one integrated thing and you just get metrics coming out the top we get observability everywhere, right?

We can make changes right we can afford to maybe intentionally drop some packets right that will help us understand how the system will work in certain conditions. And this is exactly where where you know, we're thrilled to partner with America and you know, we started this conversation very early because we knew that this this complexity was coming and the only way to do it is take a continuous posture of how we test measure understand verify, you know, how the network works and then take that information and improve it. Definitely and that's I mean that's I mean like that's that the partnership with dish was just a couldn't be more perfect match.

I mean we Casey my co-founder in CEO of Erica he comes from you know, one of the only other comparison cases in terms of scale and complexity to other than the hyperscale was himself would be Netflix, right? That's where chaos engineering was born was born enough to speak because of this complexity and you know, it's but what's great about this. That the evolution of cast Engineering in the continuous verification is that it's it's where we're putting proactive.

We're trying to proactively verify because when when you would do such a system that's thousands and millions and millions of things changing rapidly, right? It's very hard to to make sure that all that stuff is still working and you don't want to be in a situation where it's everything's just not working and you're responding you want to be proactive. Well, like we're doing with you guys.

We're pro, you know, in terms of like, you know a system at this scale is gonna be using two components kubernetes and Kafka right? If you look at most open Rand deployments, most of them are large large scale kubernetes Employments and you know coffee essentially, you know, and Eric and I mean America's sort of specializes in actually continuously verifying that those two pieces of infrastructure can you know our our stable? Are secure by proactively making a practically introducing these sort of experiments and what we call verification verifying that it can serve the communication between Services can handle the latency or that SLO is actually true or not, you know and before it fails and then you're struggling to figure out what happened.

We're being proactive verifying it verifying that works. So in terms of dish, I mean with that, I mean what I like about what we're doing is you can see where the value is Right meaning what we're like we're you know, the customer is not gonna work with you. What we're trying to do is make sure to customer does not have a bad time right by being proactive if that makes sense.

Like yeah, and she's proactive you I mean proactive is exactly the right word. Right? I think there is no other way.

This is the way right if you have five components that are physically connected in your basements and you never touch them right you're like resiliency posture is that you lock the door and you don't ever touch it right? Maybe that works, right? That's what we've done in Telecom forever.

Right but now that we're talking about millions of devices. The only way the only way is to be proactive and you know get that get out there and understand right generate more information understanding for yourself rather than wait, right and maybe we have a big outage right and I do want to underscore the criticality of the infrastructure that we're talking about here, right which is running. Critical potentially life threatening systems right Public Safety in the future.

We may have things like drones and you know remote surgeries often talked about as like one of the example use cases, but you know, you can think about a lot of cases where you really need to communicate, right and it's not just a frustration like oh crap, you know, I'm in a dead spot. I can't I can't, you know access my my blog post or something like that. It's it's really critical information that needs to be moved.

And so we don't feel that it's responsible for us to take a you know, a reactive posture right? We're putting out this platform here. We're trying to catalyze Innovations and we expect that critical applications will run on top of it.

We need to take that responsibility and be proactive so that we're making it the best that it possibly can be to support. In you know, and what's you know aspect of this I guess we're here talking about I guess app second apis. I mean I'd be like, I I personally look at you know security as not a separate thing.

It's the system is secure it's not it's not it's not like the security system over here and the rest of them over here. It's it's an attribute. It's a component.

I mean, it's a it's the system like yeah, the system is secure it's not right and you know what but this kind of scale and complexity the engineering itself fundamentally is a challenge right? But it's it's achievable challenge is what I love that word. Like I love that.

That's how I like to frame this right, you know, but it's like, you know security apps SEC at that scale, you know is is a challenge and there is no other way other than I mean this is where I started sort of in sort of breaking ground on security chaos and Jerry was applying those same Concepts being proactive verifying that some of the chaos monkey chaos engineering and play that security because you know, We build and deliver Security Solutions and components but security itself, but mentally is context dependent and you must know what you're trying to secure object a service a user what have you and you must understand the context about it context rapidly changes in a world driven by software, right so that it's a challenge what what happens we derive a context about the security and then we go forth. We build and deliver a mechanism. We call them controls a lot of times in security.

Right? Well, like what happens is that system never stops changing, especially a system like software to find Telecom, right? Like, you know, that's a massive ecosystem and it's very difficult any human to know what's going on to ride the context and verify the security.

So what what the approach we're taking with with this and with America sort of approaches on that is where we're Um asking the the security components proactively say, hey under these sort of misconfigured conditions. Are these are if a kubernetes if a known vulnerable container gets launched inside of a cluster. Is it caught within five minutes?

Right? That's our expectation. We're probably just verifying these basic things.

And you know, what happens is the the if you don't do these things, here's what happens right? You find out you have a problem with the security doesn't work when it doesn't work. If that's not a good that's not a good time to find that out.

Right like, you know the problems. I mean like, you know, then it's a different kind of problem you're dealing with right, you know and not it's not that any solutions there. Any Security Solutions are bad Solutions, right?

We've got a lot of great products. So anyway tools out there but like we suffer from a recalibration issue because of the skill and complexity into change. Yeah, and it's even worse right like you find out the security problem when you least want to write a two o'clock in the morning or something, right?

But then also the system has become so complex that as you said, you know human can really understand all of this, right so you immediately start making assumptions and guesses and you try to resolve the problem. Whereas we take this proactive approach. you know, we can really understand the nature of any vulnerabilities that exist right the way that the system is performing in advance so that when if you know, we we don't catch it, right if there is an Ever if there is ever a problem, right?

We've got the content. We've got the complete perspective and the map of you know, what we've proactively assess and where we know we're strong versus you know, what what we're seeing under current conditions and you know having that as as proactive resource for us is yeah, it's gonna be game changing and of itself. I like I like the security topic though for us because I think it's a great example of a kneeling stuff that America is working on with what dish is working on.

So I like to describe for dish that dish Wireless is a cloud story. It's not it's not a telecom story right because as much as we're changing the way that telecom the ecosystem Works, we're really Importing that ecosystem into the cloud world, right and we're trying to upgrade and you know and push push that to the next, you know into the Next Generation. But the 5G story is actually much more and this is very little understood so far I think is actually a security story more than it is like a gigabit per second to your phone a little icon on the top right that says 5G because 5G compared to 4G and prior Generations have been has been redesigned with security much more at the Forefront right with you know, open interfaces.

So you have different applications talking to each other you need higher levels of security, right? We need, you know modern levels you need to guard against a lot of the issues that we've seen in Prior generations of Telecom where you can spoof devices, you can read, you know proprietary information that's unencrypted. You know, we've solved a number of those things with the 5G architecture, but what we have installed until we started working with verica was how do we know if it's working or not?

Right like we can rely on the standards the standards took a huge step up from a security posture point, but until we get out there and proactively start to measure this. Right and verify that we're consistently, you know following the right kind of both cloud and Telecom standards then how do we know if we're really achieving the vision or not? We don't Yeah, that's you know, like that's that's where I kind of started my journey a long time prior to Eric was just like you know, how do I know?

That this stuff works, right? You know, I don't know what what I built or what you know, the guidance I gave to be built was actually actually built Europe because you wrote the best code. It's completely infallible, right?

Of course, of course. Yeah, and yeah, right conditions never change and there's no drift and the environment, you know, like well, I think that's what makes I think that's really kind of what made you know, this the whole journey for me is as a as an as an engineer. I didn't know I know it's hard, right we especially in software, you know, I don't think we're we Engineers talk about this enough.

It's like a software Engineers. I mean don't really talk a lot about this how hard it is to do what we do right? I mean, it's like we're building things never exist before we're manifesting cool things from words, right?

That's good. It's words. It's human thought it's it's it's awesome.

But it's also super challenging right and you know, like I know but but knowing that that it's a process of failure to succeed and software also, A sort of like, you know begets this could need to verify that like, you know, okay it when I built it it was under these conditions this context the context might have changed we did he just continually verify that that that it they works the way it's supposed to before it adversary finds out right like that. It's not working. You know, that's not a good time.

But also it's like but what I what I love about this journey and people want to seeing people adopt like the security aspects of certification is is that you start to get a level of just confidence you've never seen or had before right like I know so if I know for a fact this zero day comes out on this platform like, you know, yeah, I mean that's that's a bad thing. We're gonna write after that right but it's like we regularly do we really have access to these controls for Effectiveness. Like we have a good base understanding of where we stand right like and It starts to I mean put that ends up providing and is better a trust right like a like because confidence becomes, you know, something you can you know, you know build a better trust relationship with customers, you know that you know, I it's exciting just to see that be applied at this scale it is yeah.

Yeah, we've got a lot to a lot to do a lot to learn. It's great to have a great partner to help us learn proactively continuously. But I've got a question for you.

And I mean, oh, no, what do you see? What do you see next especially in the security resiliency Realms, you know like Even if we're successful, you know, and in the next phases of really adopting this kind of like, you know proactive posture resiliency to this really complex distributed system where we're entering into Realms unknown after that, right? What do you see coming?

You know that that we should be thinking about or maybe we could explore right to push the envelope even further, you know, you know I see on this. I mean we've done each other for quite some time. And areas to really be working on is observability.

Right like you you touched on it, you know in terms of new errors of observability is sort of a telecom or the cloud right as you as you reframe it for my life the wireless Cloud. There we go is you know, but this, you know having better observability and understanding about the security and understanding how it's working how it's not working fix that, you know, it's in terms of solve for It software is the big gap in salt and salt in terms of self. It serves of security observability, right?

It was easy, but we had appliances and data centers, right but it's all software driven. That's just we just need to verify that we can see when something happens, right and the bear because Solutions verification is one component of that. We're exposing we're exposing.

Hey, well this this vulnerable was introduced but there is no longer there was no there is no no visible component. We would not have known it wasn't working until we were compromised and you know, then it's our own then it's becomes a press release issue and you know, the lawyers get involved and that is no fun for anybody. You know?

Yeah. Yeah. We'll see.

Well, maybe we'll get into the quantum routes, you know, can we observe it without changing it? I don't know but all of our problems don't worry about yeah, but you know, it's it it's a journey and you know, as you said this is An evolution. I like to think of it as an overhaul or an upgrade right?

We're not we're not. We're not aiming for something. That's that's not achievable.

But we we recognize that we don't know the end state right that it's a journey for us. It's a journey that you know particularly on some of the core challenges that we have around resiliency and security. You know, we're we're taking together with Erica and you know, we're we're eager to discover and enable the next generation of applications and and software whatever that looks like in this, you know industry fortato era, right?

So for for all companies and developers that have ideas that are and are interested in kind of discovering and and you know joining the journey we're you know, we're very open and any year to work together because what we care about is the is the adventure right changing the way that we're communicating making it easier to communicate because you know, we want to accelerate towards the future. Well, so, um, I think that probably that that kind of concludes our our session but I do want to just pick a comment that if I was an engineer, like if like this this I crave interesting engineering work. This is to me that this is what this is the maze this is some of the best work going on right now in Tech right?

I would like I would I would I would join the dish team in a heartbeat for sure. Um, so thank you Brian for for joining me this conversation. Just chatting about some the great work that she's doing so the work fair is doing with you in what continuous verification is.

So yeah. Thanks Aaron. Thank you, very opportunity.

And thanks. Thanks to the audience and texture.