IT Automation at Enterprise Scale – Richard Henshall, Red Hat

This is Textron tv. Hey guys, thanks for the throne. We're here with Richard Hensel, who is head of product management for Red Hat Ansible, and we're talking about the state of automation across the enterprise.

Richard, welcome to the show. Hey, thanks. How many again, Mike?

I feel like we're kinda have a little bit too much of a good thing when it comes to automation. I feel like a lot of organizations have embraced automation here or there and everywhere. The end result is something that feels like islands of automation that are getting difficult to manage because each one of those islands requires some level of expertise to kind of master what's your sense of what's going on out there and how do we get to the next level.

Uh, that's a, that's a good way of, of looking at it. I think we've got, you know, well first off, usually say what is automation, right? As a first question for a lot of people, and you know, like everything in the IT world, we don't necessarily have a consistent definition of that.

Some people mean, you know, automation, the build process. Some people need automation, the more robotic process, business process, you know, but I'm, I'm, I focus mostly on IT automation and, and what that means. And I think your view of, hey, we've got a lots of different fragmented siloed groups doing their own thing with a different view, that's a pretty good, pretty good unilateral approach that most, uh, or view should I say, not an approach that most organizations have, uh, have got themselves into.

So what do we do to kind of move beyond that? I mean, on the one hand, folks would say that's represents progress. On the other hand, we do wind up trying to integrate these various frameworks and that gets harder and harder as we go along.

So, um, what do we do here? 'cause I think people are kind of stuck. Yeah, I think we've gotta think about scope, right?

And we've gotta think about intent and we've gotta very importantly think about the problem we're trying to fix and the value that we're trying to achieve. And, you know, a lot of the time is there's, there's the difference between enthusiasm and sort of activity is that everybody's enthusiastic about something. Hey, somebody said, Hey, we can do this thing.

It makes our lives better, right? That's a really good motivation. Or you get a very top down message of like, as an organization, we're gonna transform.

We need to automate. And then nobody really qualifies exactly what that is. And so we have to look at this from a couple of different angles.

And one of them is the technology side. You know, you have to have, I've always thought about this when introducing technologies is you have to have something that solves that purpose for the individuals. You know, you have to make it worth their while.

It has to fix their issues, their challenges, their difficulties, anything that's more difficult for them than easy for them. That doesn't necessarily get those people motivated, right? Because there's not, this isn't just about finding those couple of people that are brilliant.

This is about getting everybody along on that, on that journey. And when you flip it around the other way, you need to look from the top down of it's not just about providing some tech and suddenly everybody will come. It's not a fill of dreams moment.

There has to be a purpose and a a, a focus right? To what you want to do as an organiz. What are you trying to achieve?

And I've seen successful organizations where they've married that bottom up and top down approach. Maybe we should do it that way, um, where they've married those two things together, but they also see a lot where they just leave a chasm in the middle. You know, they can, they can incentivize and motivate the, the workers, the people doing the actual work to get going and doing something, but they're not coordinated, right?

They haven't actually given them that common goal to go for. So they start going in different directions and that's where you get that fragmented silo. Um, or you don't really have any follow through, right?

If, um, if you're not being held accountable for whether you're achieving something, do you achieve it? No. You achieve what you want to do.

And, you know, making sure that leadership are driving that vision and that objective and set and motivated and guiding people the right way. That's the bit, getting those two things to join together. That's the important part that we often see is missing in the way people approach this.

And as part of that fragmentation, one of the things I think we're seeing is that there's been a lot of writing of custom scripts that are somewhat poorly documented and, uh, the folks who write them move on and nobody knows how they work. So they go out and rewrite them and start them over again. Is there a, a better way to think about this?

'cause I think if I look at a DevOps workflow, um, there's custom scripts everywhere and not everybody knows exactly how they work. Yeah. And that, and that's where things like collaboration come in.

But also that what's, what's the motivation? What's the incentive? I, I talk a lot to people about this.

Are you incentivizing the people through their goals, through the behaviors you're expecting of them through the, the actions you as leaders, as managers are actually, uh, showing to those teams or you're saying, Hey, automate some stuff. 'cause that you gave that DevOps workflow with lots of custom scripts. They automated it, you asked them to.

That's great. I also want 'em to make it part of the sort of institutional knowledge of the organization in a way that other people in the organization can learn from it. That they can see it, they can, they can understand it, right?

It's not some special language that one person knows. It's something that everybody can understand and see. And that, that coordination aspect on top of collaboration, again, is the other.

That's the value. That's the value add of making sure that the organization gets what it needs rather than just the individual. Because you've gotta marry, you've gotta marry those two outcomes together.

How do we get the knowledge into the automation framework in the first place? Sometimes I talk to folks and their best people are so busy holding everything together that they don't have time to figure out how to automate it. So they just keep getting in this infinite loop.

Yeah. Um, the age old problem, I'm too busy to take a moment to pause and think about whether something can be done better. And what was the, there's the cartoon, isn't it?

Hey, I've got this fantastic thing. Look at this round wheel. And you've got the people trying to post the square wheel, up the hill saying, sorry, I'm too busy to take a look at what your problem is.

Um, this is where, this is prioritization, this is resources. I mean, I think, you know, finding those interested people, the ones with the right attitude towards change, giving them, you know, carving them out, giving them the space, right, to be that champion, that, that fulcrum point where you can push those them in the right direction, start showing the value, start helping people get um, easy wins. Quick wins, start carving away that activity and to that, to that point.

Exactly. I was, I was with a customer this week and we were talking about this exact topic and he said he was, he got the initial group of people on board off they went loving it, fantastic. Really bought in.

They were, they were all for it. And he couldn't get those next periphery of teams to get that, you know, to extend the process that you got beyond individual team optimization. And he went in and he said, look, remove toil, 'cause they all had that same excuse, too busy.

He said, remove toil, this will help you remove toil. And he said it was that simple. That's what started to get them to realize this was the point that they had to do this isn't, you have to do this activity to give yourself the space to also make that improvement.

'cause nobody wants to sit there and just do the same repetitive task day after day without value. They want to get, most people want to get onto those sort of newer, interesting items. Um, so you have to give, and again, giving them that space, giving 'em that right incentive, really important part.

Everywhere we go these days, people are walking around saying AI is the answer regardless of the question. But in the context of it, automation, are we on the cusp of something here? We've seen some generative AI stuff that you guys have been working on, and I feel like maybe we're moving towards democratization of automation.

Where are we? Yeah, I, I think that's a good way of looking at it. I mean, ai, yes, it could be the answer to everything, right?

That would be nice. Um, maybe in a few years in the moment. It's a really good tool.

It's a really, um, it's a really interesting new way of looking at it. Last time we spoke was about, uh, Ansible, Lightspeed and what we're doing with large language models to help with the writing of automation. That is, I think, a, a key thing is if you think about automation, I've got a job that takes me 10 days.

I automate that takes me 10 minutes, right? Think about what we can do with large language models and AI to help with automation. I have a playbook.

If I know what I'm doing, I can go and try and write something. I have to go look at the modules or look at the documentation to figure out what I have to do. I have to understand my process.

Maybe I don't understand my process, right? Maybe I don't understand how that technology works. I go through the large language model and ask it to help you with that.

I've taken a job that might take three or four hours to write that automation now only takes me 10, 15, 20 minutes. A very similar pattern there in between how automation works and how AI can be applied to this sort of worldview. And so for me, that is the, it's a, it's a tool to help with that acceleration.

It's a tool that helps. And I think that democratization is a good one. Giving giving people access to something more easily that will help them learn faster and gain that experience.

It's a really powerful tool for that, right? But it still needs somebody behind it driving it with an intent, with a, with a want, right? I still, you know, still want to have to do it, but it is a nice idea to say, Hey, my mother could even use AI to write automation, but she's not gonna know what to write, so therefore it's not gonna help her.

That the AI knows what to do. You still have to have, um, that need and want to do something, Need to understand the fundamental concepts. One of the things we are starting to see is, um, Ansible is showing up in a lot of different places.

It's in the cloud. It's being used by networking folks to automate network processes. And I wouldn't quite say it was pervasive yet, but it's definitely headed in that direction.

But where does it go from here? What should we be thinking about? And how do I kind of bring all that together in an interesting way?

Uh, Yeah, this is the age old pro. The question that I sit and, and ponder a lot, you know, what are we, what are we looking at in the future? I think we've gotta start thinking about, you know, the more we do that democratization, and that's ultimately what we're trying to get to with Ansible.

How do more people have access to more capability to do automation? Give them a focal point. What comes next?

Well, then you start thinking about, you know, I, I talked to Ansible Fest at Red Hat Summit about automating the operational activities, right? You know, the diagnosis and the remediation of what we do when we've now got more complex systems. But now I want, I'm starting to delegate that automation to other people to run.

I'm also letting more people write that automation against standards and best practice that I'm enforcing through my AI model. What about the policies that I need to make sure that that can run? We all know about policy as codes.

How do we incorporate that capability into ansibles? Ansible can leverage what you want to do as a policy, as code framework. What about compliance, right?

How do we do those additional steps? So I think there's a, a growing of the totality of what automation can be applied to. And that's where, you know, when you say you see Ansible in all these different places, that's ansible's sort of like strength is, it can be applied to lots of different steps and often in conjunction with other componentry.

So you can still have some specialized tooling for certain areas and then you, you wrap that with Ansible or combine that with Ansible to augment the capability and then bring all that back to this sort of like, more consistent layer that everybody has that initial access to. And that's what we try to get to with the automation platform. How do I say, look, this is where you can go to get automation to happen.

It doesn't mean that something else won't be used in conjunction. The cloud is automation, you know, the consistent platforms. We have things like OpenShift, their automation in there as well, but then it's the user interface to that automation that's also needed.

Somewhere along the line did we get confused between toil and value? And I think a lot of folks are running around and they think that their value to the organization is the toil they perform. But perhaps maybe if we take a giant step back and rethink what our value should be, we'll have a different attitude towards the toil.

Um, I, yeah, I, the reason why I smile at that question, I was, um, I've gone through like my own personal development, the last sort of like, you know, couple of years about what I want to be as a leader to my team, to the people I work with. And I thought, you know, my, historically, my leadership style was if, if I can, if you can haul a bunch of rocks, 200 yards, I can haul a bunch of rocks, two, two, uh, rocks, 250 yards, you know, lead by example. That's sort of like, you know, do more, you know, be faster.

And actually that's not the right thing. And that's that toil aspect. You know, I, I can do, I can fix 20 problems a day as opposed to 10.

Well what if you can do it in such a way that you can fix a thousand, right? And you said, well, they're fixing a thousand in management world. That's, you know, how do you grow a team?

How do you lead, how do you enable that team to also be the best they can be? But then this is then going back to, to automation and the value. Well, by taking away that toil, right, we're helping the organization, the wider groups we all work with to be the best they can be.

That's the value. Is it reducing security risks by the number of manual logons that occur? Is it the number of virtual machines you can produce in a month?

'cause that helps with business agility. Is it, you know, some, is it the number of the meantime to resolution that you can reduce because you've got more consistent systems with more consistent deployments with, you know, the ability to actually answer some observed event that can, that can make a change and, and resolve the problem with the system faster. But at all the point you're making your business better, right?

It's not about just the person, it's about what you're actually there to do. And I think that that shift from toil to value, um, is a good way to look at it. Ultimately, what do you think the level of scale that we're gonna see in terms of being able to manage it?

Because, you know, it wasn't too long ago where people ran around, they track metrics like, you know, virtual machines to administrators and that's kind of obsolete. So how do we think about how many IT people does it take to manage in an environment and, you know, are there measurements for that or is it just so open-ended now that it's whatever we can bear? Um, I think there's, there's different buckets or camps that around that, you know, especially if you're moving to, you know, larger scale microservice architectures, but we start thinking about edge, you know, and tens of thousands of small low value IOT devices that are used as sensors to manage things.

I mean, what if we get to that point where, you know, all cars, you know, cars are complicated enough as it is now when they, when they keep progressing, how many different devices and sensors exist on those things? So I think there's different sort of segments of what they are, those measurements. And I think, you know, I look at the, the complexity we have to manage.

I look at the number we can manage and I look at the time it takes, right? For us to do those things. And I think those become some really good measures for what it takes in IT automation.

If I've got massive large scale systems, five G's gonna make increasingly make things get bigger and more complex. And as the devices get smaller, but then the complexity of our systems hasn't reduced. It's got, it's got larger.

So how do I successfully manage complex systems with a small number of efficient users and resources, right? Because we're able to combine all that data down. I talked to it people, there seems to be some tension in the air and it's probably been there for a long time, but maybe just getting, uh, a little more intense.

Everybody wants to add new features and capabilities because, you know, they're digital companies and they're forgetting about the platform needs to be stable. And the it people are telling everybody, we need to ensure stability 'cause all these things will break the thing. So what's your best advice to folks to kind of strike a balance between those two extremes?

Uh, remembering what it is that, you know, what, you know, the, the pride people take in their work. I mean, I remember being back in that day, I just like to be able to do one thing really well, right? And that's a typical engineer's perspective.

Precision exact measurement. You know, it being good. You know, if you are, if you are in incentive and you are in the way that you look after your work, is this thing never breaks.

The second that somebody wants to change it, it's got a chance of breaking it, right? Because that's never not happened before. And so therefore, but keeping it running is important, but I think then you flip that either way around you go, well, no, our value is in making changes so that people can get more stuff out of it.

And I think, you know, empathy between those two groups, because they're different, they're fundamentally different sort of mindsets and approaches. You know, the engineer and the race car driver, right? You know, the, the engineer builds this precision machine.

The race car driver tries to absolutely thrash the hell out of that machine to get the most performance out of it. It's a similar position, but that there's a relationship there between those two that the engineer is about giving as much flexibility as possible and as much performance as possible to that race car driver. And if we thought a little bit more about that in our relationship between the people that want to keep the lights on and keep things working and the people that we're trying to make changes to try and improve what the business value is, which ultimately pays all of us to keep doing those activities, you know, that would probably help.

And maybe thinking a bit more like the race car driver and the engineer, um, you know, and entertainment is our value might be the way that we could, uh, help those teams understand the difference. All right, folks, I cannot help but be reminded of an old Star Trek episode where basically Scotty, the engineer says, I'm giving it all I got, and the captain says I need more. It's pretty much the same old conversation.

Hey Richard, thanks for being on the show. No, thanks very much, Mike. All right, back to you guys in the studio.