Joshua Corman – Trust, Transparency, Dependence and You

This is texturing TV. So this should be a pallet chain cleanser of the sort. I'm Joshua Corman.

I'm very much from the devops. Cloud and was a key contributor to the Phoenix project and what we used to call Rugged devops and I think and now it's called SecOps and all the things and if you like esbomb, I'm responsible if you hate it, I'm still responsible. But I spent the last two years doing some emergency Federal service for the pandemic sisa the cyber security and infrastructure Security Agency.

The newest Federal agency called on me early in the pandemic to say can you help us through an emergency hiring Authority design and Implement a team to keep hospitals running to protect the vaccines Supply chains. So it's been pretty interesting. I'm gonna you're gonna wonder what this has to do with devops and Cloud native, but I think you'll see a few things that now we're entering a high consequence cybersecurity world for cyber physical systems and National Security Public Safety human life.

I used a ton of Deming a ton of gold Rat a ton of empathy and shift left and devops principles in the prosecution of this work. But also I want to give a charge to those of us creating digital infrastructure that we need to be better A lot better and a lot quicker. So I'm gonna go really really fast through way too many slides but there is a point here.

I'm curious to see who gets it. So I am the Cavalry as a group I found in nine years ago. It's a volunteer group of hackers trying to save lives through security research.

The idea was the Cavalry. Is it coming on issues affecting Public Safety human life where bits and bites me flesh and blood so I asked the hacker Community. What are you willing and able to do so I am the Cavalry was a personal pledge and we've been trying to be good faith voice of reason technical literacy helping hand policy makers and safety critical Industries ever since we've had a pretty substance of impact, you know, we try to say hacking as magic.

It's not a crime, right? There's good wizards and there's Wizards thank goodness. We agained off and Hermione and Harry Potter to fight the darkness but not only is it not bad.

I just put hacker on my bio when I testified to the Senate last week and one of the centers said you call yourself a hacker. I said proudly even amongst the helpful hackers. There's really five motivations.

They all start with the p there's protectors that want to make the world safer puzzlers that do it for challenging curiosity. Prestige that do it to be the first of the best of the famous One profit personal professional gain and then protest so I'm a Protector first and foremost and I like really hard problems. I'm a Puzzler.

We try not to just point out problems, but also Frameworks and solutions and these have become kind of the Bedrock first principles for things like the first iot cybersecurity law that hackers cause which passed in December 2021. And also things like FDA medical device security. So we try to use native language to the demographics we're speaking to and this one was called the Hippocratic Oath for connected medical devices, but it was things like all systems fail.

Here's five postures. You need towards failure. How do you avoid failure?

How do you take help avoiding failure without suing the helper? How do you capture study and learn from failure have a prompt and add to response to failure and fail safely or contain an isolate failure. So these have good engineering principles behind them many stolen from safety Engineering in the physical world, but we've been trying to create these as Primitives to have more defensible maintainable Reliant digital infrastructure.

So a couple things in speed round inflection points software was really bad until the Melissa virus and I love you virus scared the bejesus out of the shareholders of Microsoft. So the seminal Bill Gates Trustworthy computer memo now they have lost some of their shine recently, but you know, they basically said if you could pick a security bug or out of feature, you should fix a security bug because it was a material threat to their dominance as a platform to Linux and Unix and all sorts other things. So I keep asking when is when we gonna see the trustworthy critical infrastructure memo because we don't really have a pathos for software engineering right?

There's a hippocratical for doctors Do no harm people who create physical Bridges have a ring they wear Made of iron that helps remind them as they're on the drafting Board of the awesome responsibility that comes with creating Bridges. But what's the pathos for software engineering? And you know, I think the closest proxy is moving fast and break things including democracies.

I'm not a big fan of this Zeitgeist. It might even be read Hoffman. If you're not embarrassed by the first version of your product, you launched it too late.

These are the things on the hearts and minds and the tips of brains of VC based and Angel based innovators. I want you to imagine this quote for the people who make bridges. If you're not embarrassed by the rivets on the bridge you built when you opened it you waited too long.

These are critical infrastructure. We depend upon these no one sat and Perpetual fear that this building was gonna collapse upon you and Crush you stealing concrete are dependable. We are increasingly depending on software.

In modern civilization, but it's not nearly as dependable. So many many moons ago. I don't remember how many years ago.

I wrote the rugged software Manifesto as a response to the agile. Saw for Manifesto hoping it could be maybe a Hippocratic Oath in an upgrade. I don't want to just be agile.

I want to be rugged too. Right and it had cool lines in it and one of my co-authors, you know, we've been wrestling with this he wished he'd put this in his economics book, which is the true cost of insecure software, but he basically said is software and art we say we're gonna throw some paint on the page, right or is it a sign? Is it an engineering discipline where there's real consequences and his answer is it's both right?

It's art with engineering consequences. And we really have to reckon with when we're making you know hand paintings that don't matter the minimum viable product and when we're putting software into something that could kill people. So when I testify to Congress or to different things, I usually have a line like this it's about relative dependent staying gears when my mentors so I can't help but talk like him through our over-dependence on independable things.

We have created the conditions such the actions of any accident or adversary can have a profound and asymmetric impact on human life economic and National Security Now, does anybody know what this picture is because I have to go fast in this talk. Maybe know what this is. It's not Pittsburgh.

It's not Chicago, huh? No, it's it's the Cuyahoga River in Ohio. This is approximately the site of the Rock and Roll Hall of Fame.

That's a river on fire. Can anybody guess how you extinguish a burning River? So the pollution was so bad in the Cuyahoga River from industry in wailing that the river caught on fire.

You know, you think that would tip public Consciousness. Well, this wasn't the first time it caught on fire wasn't the third time it caught on fire this River caught on fire 22 times across a 70 year period before people said enough is enough. It burned down Bridges did property damage to factories on the Riverbanks.

People just tolerated it until the consequences were were too much. So when we launched on the Cavalry, we had a law professor friend in your tuition. Who said No One's Gonna listen to you and tell people to die Josh.

I said okay, I believe you but let's do the groundwork and prep work and build the trust so that when that moment happens, they'll turn to us instead of lesser people with lesser Motors and lesser ideas. So in the last two years, maybe you're not all cyber people. So there are promise you there are no lies here.

We have had cyber security attacks successfully perpetrated against the water we drink. Sounds like a river on fire the food we put on our table with folks like JBS Meatpacking. Sounds like a fire the oil and gas that fuels our cars in our homes with attacks on Colonial Pipeline and others sounds like a river on fire compromises of the municipalities around our towns in our cities.

The schools are children go to even federal agencies charged with National Security. Things are burning. And yes, even the timely access to Patient Care during the pandemic.

So when I started a congressional task force in 20 16, I think technically it was a part of this just a 2015 law. We started it just after Hollywood Presbyterian hospital shut down patient care for one week in La they had a different ambulances up the street in LA traffic even one block in LA. Traffic is probably a ride you won't survive if you're in an ambulance and they ultimately had to cancel surgeries and move critical care patients because of this outage.

This outage was a single Java to serialization flaw in a single JBoss library in a single Medical Technology out of 20,000 in the hospital and they were worn by the FBI about the Sam Sam attack in make sure you check if you have a vulnerable j-boss and they said what the hell is a J boss. So when we started this task force for congress I said, I love my privacy. 0.

We're gonna focus on patient safety and patient care availability. During the timeframe the Mirai botnet was a bunch of cheap iot cameras with a hard-coded password on patchable that took out the internet for a day right before the US presidential election in 2016. So these cheap iot devices could be harnessed into the world's largest botnet and do significant harm to Commerce and public information.

So we published our report. I'm not going to go through all of it came out Mother's Day weekend of 2017. It actually got delayed three weeks.

Connor has had it but it got delayed because five years ago on Mother's Day weekend just two weeks ago was the one I cry attack but we'll get to that what we told Congress at the time is that Health Care is in critical condition for cyber security in the top five reasons are 85% of the hospitals in the country don't have a single qualified cyber security person on staff not even one. They're defending unsupported out of Life Technologies. Like Windows XP Windows me Windows 98 even Windows Vista.

It's successor for XPS also in life, and we hope that software end of life never leads to a human end of life. But now it has and with premature and Hyper connectivity. We took devices that were never meant to be connected to anything and we connected them to everything we tied reimbursement to it so that we could receive and transmit electronic health records.

These vulnerabilities are not just privacy. They affect patient care and a typical medical technology has over a thousand CVS or common vulnerabilities and exposures and while many will not be exploitable. It only takes one so not a good situation and we publish this Mother's Day weekend in 2017.

So many of the hospitals said we can't afford to protect this stuff and I said if you can't afford to protect it can't afford to connect it. They hated that so I tried to be more gentle and I use Stan Lee with great connectivity comes great responsibility as we're publishing. This wanna cried takes out 40% of the healthcare delivery in the UK.

And if you know how Strokes work you have one three or four hours to save brain and save life if you can walking and talking again, there were zero stroke centers open in London. So if you had a stroke that weekend you didn't recover but the party line is no one was harmed. And as we're reeling from this and I'm giving a keynote in Tel Aviv to the Cyber week and all the UN partners and NATO Partners on Cyber policy about how lucky the world got with Wanna Cry not pechia during my 12 minute keynote hit the world and did significant damage 10 billion dollars damaged 1 billion alone to Merck Pharmaceutical.

So this was a cybermunition from a nation state targeted against Ukraine before this most recent round of physical and pleasantness. It escaped its blast radius because software is wicked and twisty. And it got into global Shipping, you know, 20% of global Shipping through marisk.

So there's spy novels written about it like sandworm and this dramatically affected us Healthcare as well things like nuanced software for voice to text dictation right out of the state of Massachusetts were down and since it's so entangled with surgical clearances at billing and other parts of healthcare it two little kids almost didn't get time sensitive surgery because they couldn't get out of the locked up voice to text dictation Cloud as depicted in the sandworm book. So what we know is that delays introduced degraded care and outcomes. There's a seminal New England Journal of Medicine article that says if you have a heart attack during a US Marathon, you're statistically significantly more likely to die in 30 days.

I was measurable across different years and different marathons. It's not the runners. 4 minute longer routes around the runners caused a measurable impact on mortality rates.

4 minutes is enough to lead to loss of life for heart. So I kept getting told no one will listen to us until people died. So I did with any self-respecting hacker would do we started killing people?

We started the world's first ER hacking simulations with real board certified positions that happen to grow up going to DEF CON and they said Josh, you know, what would happen if you hack a pacemaker for 96% of the patients. I'm like, well they could die. Right?

He's like, no then just get tired. They go to their doctor. They change the settings.

I'm like, oh, so are we wasting our times like? Oh, no, we could really kill people you're just doing the wrong examples. So we started showing people how to kill people so ABC Nightline covered it just like Airlines do flight simulators.

Learn how to handle certain circumstances doctors train routinely within the ER would paid actors and surgical dummies are realistic to catch those unit those zebras that are not common. Like if you have a pregnant woman, how do you give a defibrillator? It's too heartbeats.

It's a little different so you have to practice those differences. So we started hacking Things We did an insulin pump where we gave a second unauthorized dose found by a diabetic himself on his own device and it could lead to a fatality Billy Rios found that you could give a three-hour dose of a calcium channel blocker in 30 seconds. By changing the drug Library, which is fatal.

Icemaker defibrillators where they three digit universal access code in some of the most popular Pacemakers in the world allowed us to change the software. On the device and administer an electroshock every minute on the minute to see what would happen and it both caused heart attacks and resuscitated them to the point where they had to open heart surgery. So this is covered on ABC Nightline and we tried to keep ahead of the threat we started doing this in 2017.

We've done it every year since we also do tabletop simulations to see if hospitals can handle it. We've graduated to hacking the or so we hacked the Integrity of the database for the blood bank. So people got the wrong blood type which coagulated in surgical field.

Inflations we proved you can kill people without touching the patient simply denying access through ransomware of Imaging helps you fail to save lives for a stroke because a rupture is quite dangerous and a clot can be saved in say brain and save life instantly with a miracle treatment. But if you administer that clotbuster to a bleed you'll kill the patient immediately. So the mere unavailability of Imaging and the next proximal Alternative Care was too far away.

So we lost patience just by doing that and then during my emergency Federal service in the pandemic you had these Technologies the Internet Medical things that are multi-million dollar expensive radiation delivery machines that look right in the hospital and Theoretically function in the hospital but all of their computations for where to administer and how much to administer that radiation delivery is done in the cloud in the cloud was down for a long time. So the country was unable to use these bricks that just don't do anything without their brain in the cloud. So the pandemic took those seams in cracks and made them worse and I'm gonna get even further from devops for a moment.

My team studied in February of last year when the country hit 500,000 lost Americans to covid. We also hit a hundred and fifty thousand lost Americans on top of that to non-covid deaths These are called excess deaths in the CDC tracks the mercilessly across every year from a statistical model to see what's the difference between expected deaths and actual deaths by caused by region by month. and not to my surprise a lot of these were time sensitive things like heart like brain like pulmonary where minutes or hours matter.

4 minutes can kill you. And for hours can kill you. What do you think four weeks did the state of Vermont?

The largest and most protracted ransomware outage and recorded US history. So I asked the uncomfortable question in my team did people die and as we did some data science and we're proving a statistical model for how ransomware attacks can strain a hospital sufficient to lead to excess deaths immeasurably. So the Wall Street Journal on the front page noted the first lawsuit that's gonna settle for a specifically named victim a baby in Alabama who was born successfully died after birth after complicated birth.

This hospital was ransomed. They had degraded access to the technology that assists the scale and scope of their medical professionals failed to warn or reduce the census to the hospital admitted the same level of patience, but that patient ultimately Parish the modern delivery of care with the nurse to doctor and patient ratio is a feature of the technology assist you. Take away that technology.

It can have dire consequences in the doctors and nurses exchange text messages saying that they would never would have delivered. A baby that way had they had their Imaging. So it's going to settle so we have a named victim and my team did the first statistical model to prove loss of life.

So we know that the graded delayed access to care affects mortality rates, and now we have proof in the narrow and in the broad sense that you can these attacks to grade into Lake care enough to kill people. So I published with the CDC morbidity mortality weekly report the instrument we use was that when the country hits 75% ICU strain or intensive care unit strain, you would see 18,000 access deaths in two weeks. These are not elderly people with comorbidities the fastest growing demographic from these excess tests were 25 to 44 year olds young people.

Critical infrastructure each people people who make sure we have water and food supply and truck drivers and Manufacturing and doctors and nurses. And wastewater treatment Engineers, so I'm became quite concerned about this, but worse if the country got to 100% ICU strain, you would see 80,000 excess deaths in two weeks. So we're talking really really big numbers.

The reason I'm punctuating this is on October 1st. We published an analysis of the various cyberphysical strains on the Nations ability to provide medical care and ask the uncomfortable question of Ken cyber attacks lead to loss of life. So there's plenty in there and I hope you all read it, but we used a lot of Deming in gold Rat and devops principles in a massively multi-disciplinary cross-functional cross sector team to show the relationships between things like manufacturing Healthcare delivery cyber attacks supply chain disruptions Etc.

And essentially the system is so lean that usually disruption can be diverted to next year's proximal Alternative Care, but in an environment where everyone's strained there is no proximal room in the end. So these failures that used to Cascade from one city having diverts next to nearest County was now happening at the state level where the crisis standards that care to deliver bio Idaho started to trigger crisis standard as a care neighboring states. Again, those conditions were time sensitive.

There are things like heart brain pulmonary diabetes and they were asymmetrically affecting younger people and black and Hispanics. So when we studied some of the states hit hardest Imaging was down chemotherapy. He was down Radiology was down lab testing was down records were down and as such various Services stayed down and while people got partial recovery within a few weeks.

It actually took across a five-month observation period to get back to 100% And when you're not at 100% your ICU strain goes up and your excess tests go up but there were two ways people died. Some of them were pandemic constrained in some of them are not number one is when you have the Cyber incursion, you have to divert ambulances the next proximal alternative care if that radius of your next proximal Alternative Care is too far away. You may not survive statistically or empirically so that is the first area for acute specific deaths for latency sensitive conditions.

And this particular type of death will persist beyond the pandemic and what we've learned is that an attack in a rural Hospital can have a much higher consequence than an attack in a city where there might be next proximal alternative hospitals. But what happened it was easily measurable during the pandemic is the hospital strain above the safe 75% ICU level could be exacerbated which led to subsequent access to four and six weeks later. So looking at a state across the statistically significant observation period of five months during the same pandemic with the same.

Population controlling for state level effects and the type of Hospital in the size of hospital we could see that hospitals affected by Ransom and their counties achieved these excess death stress zones sooner and stayed there longer than their peers. So we can no longer live in denial that these are victimless crimes or that this is merely a HIPAA fine or a ransom. People have lost their lives, maybe people that you know.

so what are we gonna do about these things? Well, there has been some political appetite after the solar winds attacks. There's an executive order that pushed a lot of my babies like software Bill materials and software transparency and software Integrity to try to raise the bar.

I said for a long time transparency is coming like Game of Thrones What I really meant is s-bombs coming or software build materials. This is a Deming principal stolen from Toyota Supply Chain management in the 40s applied to Modern software development. It was meant to make us more efficient higher quality faster mean times identifying repair less unplanned on schedule rework.

It happens to also be increasingly the source of most of the high consequence failures in software world. If you think that this is something that the Congress Critters will ignore it's now on John Oliver right? There was a whole segment on ransomware into fact on hospitals.

How many Hospital tax there were how effect patient care and that again see that number 85% of our hospitals lack of single security person. So for your own personal safety and risk management, And I want to point out a pattern that we're not free of yet that even though the country's done with covid covid has not done with us. And this is not actually about covid killing people.

It's about ICU strain irrespective of cause is killing people. So I took a snapshot for the first two years. And what I want to point out to you is whether you're looking at case rates or just net ICU strain.

The Orange is covid ICU strain, the teal is non-covid strain, but once they combine to hit 75% you start seeing excess test two four and six weeks later and we have had several phases of the pandemic but let me color code it for you. We have a small spring surge of hospitalizations a big summer one and a huge flu one now. When you look year over year instead of just trying to be hopeful and Wishful and make this stuff go away.

We want to be done with this stuff the spring one year every year is not that interesting the summer year over year got much bigger the second year despite the fact that we had no vaccines for the first three. In plenty of vaccinations in the second three waves. So it's really not about vaccinations either.

It's just net Hospital strain. And if you stack all these different things here's the lesson I want you to take away. With each wave of the pandemic they are getting bigger year every year and more importantly we had.

10% you know a hundred percentiles of dark doctors and nurses for the first two and we're down to 90% after the next few and then we're down to 70% after the next year. How do you take more patient load with fewer medical professionals and expect that we will not see further access. So the good news is after that devastating flu season.

We just had this spring waves not so bad. The bad news is the summer wave is already started and you're starting to see hospitalization rates way back up irrespective of cause so while the pandemic is less fatal, especially to vaccinated and boosted people. It is not reducing the net strain our nation's health care and that strain is leading to access deaths of 25 to 44 year olds and others from delayed integrated access to care.

Any of these that are done by the virus are not really controllable by this room, the additional inefficiencies and loss of life perpetrated by unsupported software things like log4j. Each of these waves is done devastating things. So back to some of the solutions when log4j the largest software supply chain exposure in history hit could hospitals answer the same two questions.

I tried to after Sam Sam in 2016. Am I affected and where am I affected the good news is we're pushing and promoting software transparency and software bill of materials. And now it's an executive order now agencies have to do it and the FDA is asking for permission to mandate these things.

But unless until we can drain the swamp of all that Legacy Tech and start applying continuous integration continuous delivery fast patching mean times identify mean time to repair for unsupported unsafe vulnerable and attacked software in our digital infrastructure. We're taking elective risk. So when we talk about esbombs we They were all in a supply chain.

Most of us are in the middle that bedside infusion pump is a final good example, but 95% of its written by somebody else through third-party and open source compound parts and those compound parts are turtles on Turtles all the way down to these individual atomic parts. But you know most of that attack surface is not been scrutinized and if we take a page from Deming in Toyota Supply Chains It's that we should use fewer and better suppliers of parts. We should use the highest quality parts from those fewer better suppliers, and we should track which parts go where throughout manufacturing and a quote Biggie Smalls mocote More Problems and we have way too much code and elective attack surface and elective code and it's my sincere desire instead of having the water attacks on the water.

We drink the food we put on our table the oil and gas that fuels our cars in our homes. The schools are children attend the municipalities to run our towns and cities and yes, even the timely access to Patient Care. That we start realizing that software isn't very, you know soft and it isn't very casual software is digital infrastructure underpinning the nation's critical infrastructure and the bottom of mazos hierarchy it needs.

So if you hear about s-bomb, please look into it and take it seriously, and if you haven't I encourage you to do so because it might be a regulatory requirement for you any day soon. I know cyber security is not the theme of this conference. But a lot of the hand waiver at Best Practices these were developed before the ransomware Revolution when we care mostly about the confidentiality of information, we now care about the availability of life saving services, like water and food and timely access to care good enough security probably never was but certainly isn't anymore and with the hot conflict between Putin and Ukraine and threats critical infrastructure for those who interfere.

We are very prone. We are very pray. We've just survived thanks to Predator appetites, but they have become more Brazen and aggressive.

So we're gonna have to do some pretty important things and this may separate critical infrastructure software from non-critical casual software, but if this speaks to your heart and you think the best an upper echelon of innovation is CI/CD and devops and empathy and shifting left and all the buzzwords that we like to grow on about and make fun of we need a bigger boat and we need a better breed a digital infrastructure and then is likely to come from the Phoenix project devopsy tribes if from anywhere and let's make sure we're applying it more to resilient maintainable hospitals so that you can stay alive and your loved ones can instead of social media platforms that spread misimal information. So I'd encourage people didn't listen to someone wise about washing his hands before going from the morgue into delivering babies. They listen to him a hundred years too late.

I'm hoping you guys listen to me a little quicker. So this guy freyrite took a 95% or so mortality rate for childhood leukemia as accepted as the best we could do and now we have inverted it and now we lose about five percent of childhood leukemia, but he did a bunch of unnatural and barbaric things which ultimately led to plately transfusions and bone marrow sampling that hurt really badly and the invention of the cocktail for plural forms of chemo therapy and radiation therapy, but now we saved most of the leukemia patients instead of losing most of them. So we have had rivers on fire.

We're gonna have more before it gets better. But I believe the issue here is you're over dependent on Dependable technology and software so we can either depend upon it lasts. And/or make it more dependable in mind instinct is the tribe and the culture and the patterns that are gonna make us better are the ones that can get past our cynicism and our crankiness and our whining and Out how do we make more defensible maintainable resilient infrastructure so we can get the promise of connected technology without the Peril.

So that was Heavy in the break. Maybe people can ask me what the hell this had to do with devops, but my simple hint is my team saved a lot of lives on a lot of different things many had nothing to do with cyber. Some of them were keeping cold things and speakably cold through the cold chain of visor deliveries after operation warp speed things like that.

What we did though is we said Conway's law is killing people. And every great thing this species has ever done and every great thing we did during the crisis. Came down to high empathy massively multidisciplinary teams at intersectional wrist points with collaboration.

So I strongly encourage that maybe the solution in many your problems. Even if they're not life and death are gonna be smashing organizational boundaries to form multidisciplinary intersectional multifunction teams to do the big things that are charts are not well equipped to do. This could be what makes your company more successful.

It could be what saves the lives of those you care about. Thank you for your time. I think it's lunch time.

Did I testify to you? Yes Senate help committee last week Healthcare education labor and pensions. Yeah, they asked some pretty good questions.

And like I got the sense that they heard some of it. I've got quite a few follow-up meetings. I'll believe it when I see the questions for the record.

They have up to 10 days to give them to me, but I did not pull any punches. to play but I did not pull any punches early indications are there's a some political will to chair is Murray ranking member was not present but his alternative was Cassidy, but you know that committee has a Romney Rand Paul Bernie Collins my senator in the state of New Hampshire Maggie Hassan quite a few Physicians. So the Physicians asked medically informed questions, and there's a bipartisan bill of two of them.

To at least bolster medical device regulations for FDA called the patch act it passed the house already, but in a bipartisan way and I think they're looking for education to know if they should pass in the Senate. But I'm deeply concerned about hospitals. Not just the medical devices they take in but their ability to defend themselves and you should be too.

If you're looking for a job change, please go help your local hospital. All right. Thank you.