Modern Digital Apps at Scale – Lee Atchison

This is Textron TV. But the distinct pleasure being joined by Lee Atchison today. He is an author.

He is a cloud expert. He's with Atchison technology. He's an author written books.

He writes articles Consulting lots of things that you do, but most of it is about sharing your expertise, which is what we're here for today. So welcome late, Thank you mention. It's great to be here.

Good to have you. Well, tell us a little bit. I sort of like dabbled in telling us about yourself.

But root once you give us an accurate picture sure you didn't you didn't find a basically I've I've been in the industry for well forever seems like it's been about 30 years now, but the last 15 years in particular is probably the most important I spent the last 15 years in a combination of Amazon and New Relic and Amazon. I was there doing the early days when you know, the retail organization was building, you know, everything in a monolithic application and they were just starting to split it into service my service based architecture. And then I was there when when AWS first got it.

It's kick off at a start and involve through that whole process and and build one of the Year early services. That is elastic Beanstalk that came out of AWS. And so I've been involved in a lot of the early days and and You know and devops and Cloud native and microservice architectures and in all of those different Technologies Technologies.

I was involved a lot in the early days at Amazon and making those things happen. So then I moved to to New Relic and you know, New Relic at the time, you know, now they're a relatively large large company, but at the time they were a small startup the entire development team just fit into a into one small room and in in in the Portland office and but they were struggling with scaling and that's where I realize that a lot of my expertise at Amazon could help them and that led me eventually to write the book architecting for scale which is kind of led to my whole writing Consulting gig as we go along there. well, perfect timing for it who knew, you know, we would have this acceleration with Yeah, transformational kind of applications digital going to digital experiences and let's move everything to the cloud in a compressed period over you know, the pandemic and all of that.

Well you I it's like you planned it or something, but well, I don't think our planned it but but it certainly seems like it. Yeah, I spend several years in New Relic, you know last several years traveling across the world. I mean, I would do three three and a half weeks a month traveling to see customers and doing different things and that ended and in the pandemic started and it was like it was almost perfect timing but that really is the the kick off the lead in to my next stage of life, which is basically what I'm doing now.

Yeah, it's timing is everything right sometimes exactly good. Just yeah fortunate when that happens. You know what I think a really great topic for us to talk about is You know where there's so much kind of around modernization of applications whether you're doing yeah occurred apps or modernization of digital experiences and things like that.

And so you have these tracks of what's happening in the business world and acceleration of business and wanting to leverage software make that more part of their strategy. You also have of course move the cloud move to architectures like Cloud native and new technology. Yeah, and sometimes that's that can be a royal crash in the highway or it can be, you know get things in the fast lane and it really works well.

I think a great conversation for us is like what are those things that you've seen that help people while they're trying to make changes like putting in devops to be able to do software faster, but we're also going to do some Cloud native or strategies like that. What are your thoughts around that? sure, so, you know for for a company that really is struggling with modernizing.

I think one of the first things they need to really wrap their mind around is devops, right, you know devops is probably the oldest of the modern Technologies if you will, but it's really critical just the most Central. Yeah. It's it's a central aspect everything that goes on.

And you know, it's it's almost taken for granted. Now when you talk you to mainstream media, they you know, everyone of course must be doing devops yet. There are still a lot of companies.

I still struggle with that, you know Concepts especially Concepts like continuous delivery. It's difficult for a lot of established companies, you know, the What I try and get across the people when I talk to them is the the concept that you know, the more often you release the less risk you're taking and that seems counterintuitive to someone working. Especially if you're working in an established Corporation established Enterprise with established processes that whole concept of of moving fast means less risk is is foreign to them.

And so, you know, it's you know, the it really is I think the central aspect to what you have to get over that hurdle in order to truly Encompass continuous delivery and in order to accomplish devops. and You the other thing that's kind of related to that is is what you hear is. I can't really small often my test Suites take this long to run or I have to spend all this time going through a QA process and you know, all these layers of bureaucracy necessary before I'm even allowed to release something.

But well, you know, it's sometimes I go to those companies and I say well just get rid of all of that. Yeah, just don't test at all. Now.

There might be a little extreme and certainly a lot of companies. If you don't go from one end to the other very good at the door, right? Yeah.

Yeah, but don't get me out the door. That's true. But the what really what the idea is is that the the fast moving process itself provides you a buffer for testing, right?

You know, you you reduce the amount of testing so you can move faster and because you're moving faster you can respond to problems quicker and as a result of that you don't need to test as much upfront. So it's it's a cycle that builds on itself, but you have to get on the cycle first and it's it's a it's a very very hard thing for large inner prices to do so used to checklists and processes and all these layers necessary of approvals to get things done. Mac goes completely against the grain of devops.

So that's some of this every one of those steps or checks or a lot of them happen because some problem occurred one time and so we add more to the process right to prevent bad things, you know, exactly under bad things though should never happen but probably aren't going to happen. Anyway, that's right. That's right Decades of risk aversion.

Hmm has led to the point where you are no longer managing risk, you're avoiding risk and that's a death mail for most modern companies. It's interesting. I just went through a process kind of like this and I'm not in large Enterprise.

So not speaking at that level in any way, but when you're launching new products doing new things you haven't even greater flexibility for that sort of How much testing do you do right because a lot of times it's just you need to get things in market and get people to use it. But if you've got a process where you can react quickly and respond and fix that and or add that or what or take it off disable It Whatever feature flag at whatever you're doing. You know you can you don't have to have everything is perfect as we thought we needed to have of course.

It was never perfect. Yeah. Yeah, it's I think that's the fallacy right is it?

your Enterprises think that by going through all these steps we're making quality better and so therefore It's perfect, right, you know, they were work striving for Perfection and you're never going to get there. And in fact the the best way to get things something high quality is not strive for Perfection, but rather to experiment and take chances and control chances manage chances. Yes, but chances and things just get better as time goes on.

It's that repetition to Improvement. Right? Exactly.

You know, you don't win the basketball game, you know by shooting one basket right here lead up to Great how you get there. Well, let's so so devops is obviously a big part of it because it's about how you create software and be able to release it on you know much more increase the velocity much more frequent where where you can in your organization as you put those things in place what sort of the next Part to look at and re-examine. Sure, so, you know the next thing that you hear about is cloud native, right?

And and I think the cloud native is really two distinct things. There's a whole bunch of things that make up Cloud native, but there's really two aspects of modernization that that enterprises go through when they think about wanting to be Cloud native one. The first one is wanting to be wanting to utilize the public cloud and get the benefits of the public cloud and two is the concept of moving an application from a monolith to microservices MMM.

Really when you wrap, we pull everything away unveil the onion you pull things apart. That's all the cloud native is as those two things. Now, you know five years ago the moving to the cloud part was the hard part, right?

That's where people were focused on having a hard time doing having a hard time struggling making it happen. I think that's getting easier and easier as time goes on. It's becoming less critical of a complex step that it used to be the the biggest struggle.

I used to hear about Cloud migrations were companies. He said I can't trust the cloud. I can't it's not secure.

You know, we what do we do? You know, we can't move this workload to the cloud. We have to leave it on premise because it's got high security requirements.

So Transition, but I think people are finally making this transition to realize that the cloud. Can be and is more secure than your data center, you know, the the processes that are in place to help you provide the tooling you need to build a secure application are are part and parcel of what's available in the cloud and Cloud. Providers have a strong desire to make sure your application is secure so they build strong infrastructures and provide tools to you to make your application a secure as possible.

And they provide best practices that have worked with, you know, Decades of experience that they have available to them much more experience than you have in your company. No matter how big of a company you are and all of that experience is is driven towards best practices for using their tools to make your application safe and their environment. And if you follow those best practices, you will have a secure of an application as you can get much more secure that you can possibly build in your own Data Center.

So I think companies are starting to get there. I mean, I used to hear this all the time in certain geographies specifically in Western Europe and certain industries specifically health care and finance and I'm hearing it less and less now probably still hearing it a little bit in some Niche markets like, um, like Swiss private banks are still and very reluctant to move to the clouds still but that even that's just starting to transition, but I think more and more Or the acceptance of the cloud and what's needed to move to the cloud is is becoming more mainstream and more acceptable what isn't though as a realization of what the cost structure is for the cloud and the changes available. To you in that cost structure and it's you know, the you still see a lot of companies do the let me lift and shift to the cloud.

And once I lifted shifted to the cloud guess what the infrastructure costs more. Oh this obvious. Obviously, the cloud is more expensive.

So I move back to my own Data Center. but you know all Of the advantages of the cloud. It doesn't talk about, you know dynamic resourcing it doesn't talk about Changing Financial models for how you can account for your infrastructure from Capital expenditures that you having in your own data center to operating costs.

Yeah, a cost of goods sold sort of operating costs that you have in in the cloud and and the ease at which you can adjust your infrastructure costs to match what your real needs are. And when you take all all of those things into account, the cloud is almost always cheaper than the non-premise data center, but it's hard to convince people of that. I just think about when you're building data centers you always have a horizon, right?

How how far ahead are we planning? So we have X capacity for the next three years five years exactly exactly over buying to start with for your own Data Center. And so when you're lifting all that up there, you're kind of taking that whole same approach exactly over provisioned everything a lot of ways.

I yeah, here's an idea. I want to run we're in by I also think that As you get as you start to move to more of a faster delivery pace. With with how you how you create software and deliver software.

The fact that you're on someone else's Cloud you're in that environment who knows how to run it very well. Let's see. Let's assume that they know what they're doing they do but they're also not only good at that.

They're also good at introducing new services and incrementing those services on top of that much better than you're going to be able to you're right but your own, you know, going back to job of being days, right, you know your own version of whatever service up there and maintain it yourself and keep it upgraded and you know in Pace with what's happening. So you actually can accelerate the technology maturity and Adoption of that at the same time with your app. Oh absolutely agree with you.

I think it's you know, when I and Enterprise is in the mock of trying to get their application working in the cloud and justifying the expenses and making that happen. It's hard for them to think about that part of it. You know, you you really have to start with the here's the benefit you're gonna see right now and oh, by the way once you see those benefits, you know, Now, let's start looking towards the future of Leverage buys and leverage technology builds and the value of shared infrastructure in general from the standpoint of all of these value-added services that are being built that they can never be built for a single customer.

But but are being built in such a way that you can take advantage of them easily and and efficiently so, you know shared infrastructure have huge advantages. It's it's share shared infrastructure is not a dirty word. As a lot of people on security aspects used to think but shared infrastructure truly is valuable and and is a benefit to you and your application.

Awesome. Well, let's talk a little bit about you mentioned the move for monolith to microservices easy to say but that's a big that's a big shift too. Like boy.

That's a huge one. It's not an easy transition, you know, if someone's looking today to build a new application from scratch, you know, I almost invariably tell them yes use a microservice architectural pattern start day one. Don't try and retrofit it later.

Do it build it in for the beginning but moving existing applications the benefits of moving a lot of these large monologues that have gone so large in complex that they can't easily be be grown and developed on anymore. The benefit of moving them is huge. But so is the cost of moving them and so I work with lots of clients who are in that migration, they want to move to microservices from a monolith.

I mean, that's the situation Amazon was in when I move. Amazon and as a multi-year project, they went through to move from their what they called obidos, which was their Amazon retail stack which was a single monolith of everything that runs with an Amazon fits within this monolith to their distributed. They called it the time group of but I'm not sure what it's called nowadays.

They're distributed environment and they were able to move from a model where they could release code once every couple of weeks if they were lucky and struggling to get releases out the door to a model where they're releasing code thousands of times a day or even thousands of times an hour and some respects and opinion how you measure it and it's just so much easier to innovate and develop and grow in a model of the application. But moving a monolith to microservices is a is not an easy transition and Success is Not Guaranteed one of the things that whenever I start with a client that's doing this is I I talk about what I call the valley of despair. That is when you if you look at it like an XY quadrant and you yes time and energy effort goes into a migration and it's on the horizontal axis on the vertical axis is value out of that effort and you you like to see nice curves like this, right?

That's you put an effort you get value out. That's not what happens at all in the microservice migration. What happens is it goes down for a while and then it shoots way up.

So you go for a long period of time during a migration where you where you're putting in tons and tons of effort for not only no gain, but for negative game you are making things worse in your application as you put energy in to do this migration, and then suddenly you start seeing the benefits and things get better, you know the light at the end of the tunnel everything's great. Wonderful. And and that's the that's where you want to get to.

But going through that Valley of Despair is very very hard. And I've seen company after company stopped their migration right at the low point because it's like we can't do this anymore. This is too expensive.

It's not working. We don't want it to happen. I've even seen companies say declare victory at the bottom where they say.

Well now now at least we know how to do microservices we've learned how to do this. We'll finish this another time in the future. But right now we need to stop we got more important things to do and they stop at just the wrong time.

It's a really hard thing to do. So one of the first conversations I have with the company when I brought in as a consultant to talk about this migration process is to go through this Valley and to get a commitment from high in the management chain to know what that Valley means and the impact of stopping early. And and you have to have that commitment.

Otherwise don't even start one of the challenges. I can't be this is an experience that I had. Moving from monolith into the cloud and moving some of it to Cloud native was the people who built it were no longer there that they had retired or you know, this is yeah for 15 plus years right move down to other jobs.

It wasn't large team but you know ten so people that were kind of core to working on it. So there was a lot of hesitants touch big parts of it because people were free afraid to break it. So we're not only slow to release it.

There are parts. We just didn't want to update because nobody would do it but they lose their job. So what we ended up doing was we did this analysis said well and the business here's the services we want to start app offering here's what we'd like to experiment within Market.

net application at the time. net and and it worked out to be pretty good because we didn't take on too much and we we got enough that we could start to go experiment in Market create some new Services, you know the UI for it and that seemed to work pretty good for that particular applications. And if you can't be you know that fortunate to say well the whole thing's got to change right?

That's a hard hard left. Yeah, it is it is and actually the technique you talk about there is the technique that I ultimately recommend when they start down this process. Unfortunately, when you're when you have when you don't have something as clearly defined as what you're talking about it in that example, you have to decide what part of the model do you want to pull out first and invariably what happens is you pick something you pull it out that creates warts and the side of the of the model you pull these individual things out and you end up with separate services that are not in the easier to support than they were in the model of that now have network connections that they didn't have before we're too warts on the side of a monolith that isn't really any smaller than us before and and that's the part of this Valley that I'm talking about.

That's where you're making it worse before it gets you're making it worse before you make it better. Yeah. So it's it's a it's the right approach to like but just realize that that you know, sometimes if you're in one of these situations where you're developing a whole new product line or a whole new connection that you still need the monolith, but it's a separate entity if you will.

That's a great opportunity to build that into services and then make the interconnects back to the monolith and you end up with fewer warts fewer. Problems in the monolith and your new businesses working in a new model and that's great. And that's fine too you end up with some organizational challenges because you you know supporting a monolith.

The development process for building a model of is very different than the development process for Building Services. So now your organization is split into two and has different processes systems culture and and keeping that working and sustained over the long term can be difficult as well, too. Sometimes what happens is companies do that and they build services.

But ultimately what ends up happening is they treat the services like many models and then they put them back into their main processes. And now they don't have one model if they have 10 monoliths. Just some of them are smaller than others, you know a little better but not you wasn't exactly yeah.

And so it's it's a you know, it's I really hesitate. Working with a company that wants to increment into microservices. It's like that's fine from a technology standpoint increment in But from a mindset standpoint, you have to be focused on the goal.

You have to be focused on the transition that's needed to make that happen or it's not necessarily worthwhile to even start. It's kind of like if you're gonna shift from designing Passenger cars to race cars, you kind of have to get into the mold of a race car designer. That's right.

That's right. You passenger car like you thought about you know, 55 to 75 mile an hour traffic. Everything you've never been in a race car.

I mean, yeah that could that that might be important to you. What's been a lot of fun. Enjoy talking with you Lee.

I like love to hear a little bit more about your book tell folks about architecting for scale. Sure. So I I did a little bit when I first started that I the book came out of my experiences that New Relic where I discovered that they were going through the Pains of growing from a startup to a large company with a large highly scaled application and they were going through all the availability and scalability pains of making that happen.

I learned a lot of techniques and Amazon for how to how to avoid that. Not only not really talking about the you know, coding aspects. The book isn't about Coding for scale or coding for availability.

It's more about the systems and processes and cultural aspects of what it takes to build an organization that supports a modern application. You know, how do you organize your company to support services? How do you organize your company to to provide a service here within your application?

How do you deal with sla's? What does that mean? How do you those sorts of aspects?

And so I took the things I learned from Amazon and applied when I move to New Relic and I put it into into a book. Well, when I did that New Relic put me sent me on the road to talk to our customers about all these Concepts and and it's amazing the amount of feedback. I got from customers who who agreed with all the things I was doing and helping and then they would introduce their own stories and their own lessons and say oh this is what we learned and all those sorts of things.

And so that led to the second edition of the book which is diversion. You see behind me. Now O'Reilly the book did well enough that they wanted to do a second edition.

So I built that the second edition is oh about 50% longer but about 80% new content. And so it's got a lot of the learnings from taking the first version of the book on the road talking to customers and applying those changes back into the book as well. So this book came out about a year ago and I I hope you end up buying a copy and then taking a look at it.

It's available from Amazon and all the normal places and as part of the O'Reilly Safari program as well. Well having read the book I would highly recommend it. So so thank you.

It's an excellent read and I mean there is a ton of good information in there. Not just sort theoretical explanation and some real practical stuff though. I think it's super helpful.

So well, thank you. Thank you Lee working folks either check out more and you've written more than that book have other things that you're doing as well as your Consulting practice. Where can they connect with you, sir?

fm modern digital business. fm. Great.

We have an episode coming out soon. I was we do for having me on your podcast. Of course, September 12, September 12 be there.

All right Lee. Thanks for joining us again. Look forward to coming back soon.

Thanks Mitch that.