Feature Flags and Microservices: Oil and Water or Vinaigrette – DevOps Unbound EP 31

Hi everyone, welcome to another episode of devops Unbound them upside down devops Unbound is a every other week video show where we spend about 40 or 45 minutes talking about relevant topics and devops. Devops some bound is sponsored by our good friends at tricentes. It has been since day one and we're very thankful to chase dentists for their sponsorship and help with the show more than just, you know, saying their sponsor or writing a check they sent this is intimately involved with us and helping us pick our topics finding guests and just keeping the show on track because it's not an easy thing.

So many thanks to linear normal and the rest of the great dentist thing for helping us this week shows a great show. It's all about feature Flags. We're gonna jump into feature flags and the second but first, I want to introduce you to this great panel.

starting itself is a Bitcoin guest here on devops and bound. We love having them on, you know, he says great great insights and things to say, it's our friend Brian Dawson, Brian welcome. Thank you for having me again you Brian why don't you give people a little bit of your background if you don't mind.

Yeah a bit about my background as I I started my my you know, Journey as a developer actually creating console games. It was what we now know as PlayStation which which I'll talk a bit about later, but that really triggered that was my first experience with what you would call kind of the the rudimentary days of sort of feature Flags fragments Etc. I eventually went on to the company that founded subversion and there got an opportunity to go out to a number of clients and help them Implement continuous integration continuous delivery and and more modern more modern practices.

And then and then today I work with a bevy of great Partners in the blockchain space at a company called Ripple. Actually, thank you Brian. Next up is our first timer on devops on them, but she's she's works with a company.

That's not new to us here a tech strong one. com. Twenty thirteen twenty fourteen and that's Cloud bees and this is Hope Lynch from cloud beads.

Welcome home. Thank you. I as you said, I'm a cloud beasts long career in technology.

I often tell people if there's a job you've done in technology from pulling wire working on servers coding pretty much everything. I've touched it one time Heather Smith some time at Cisco spent some time at Red Hat. I'm most recent role before having to Cloud bees in a cloud bees.

I am and product marketing. But before that which is why I'm so interested in this conversation. I was a product manager for an iot platform.

So microservices feature flagging was a pretty critical part of how we were getting our jobs then absolutely, and of course the whole devops of things if you will, you know bringing devops on a ways of looking at things like feature Flags to iot devices is is such a big It's a it's a new frontier. We're gonna be exploring. I'm actually the middle of writing something up about devops world coming up later this month and you know the theme there's devops remix.

and and we're also doing another show the month after called devops experience and devops of things and this whole devops everywhere and they do these two ideas run in together where we're seeing devops go where no devops has gone before right and into iot into the edge and feature flags, of course place a big role in that testing plays a big role in that it's gonna Really Gonna Change a lot of Patterns that we've developed. Anyway, thanks. Hope for joining.

And let me introduce you now to our third panel member today. He's also been on. Devops on down before you may recognize I had some bearded mustaches our friend Adam Kalsey.

Hey, Adam welcome. Thanks. Thanks for having me.

So a little bit about myself. I started as a developer in my career way back in the Dark Ages before cookies existed in in browsers, and I currently lead some of our product management organization at tricentes. I've been building products exclusively for Developers for about 15 years now.

And and Alan you missed an opportunity to coin a term. You should have called it thing Ops. You know what?

It's not too late. com everybody. I just registered.

Okay, that's gonna be really popular with iot crowd. I'll just oh, yeah, but you know, I mean in a case of what's old is new again if Google has their way we might be be doing away with cookies, right? And yeah, so, you know, we'll see what happens with cookies.

So let me and then bringing up. Certainly last but not certainly not least is my co-host for devops and baron as well as one of my partners here at Tech strong group one time friend Mitch Ashley. Hey Mitch.

Welcome. Good to be here Alan and a great panel. I'm I'm not gonna tell you when I started developing and what wasn't around back then that's a whole another day, but I'm excited today because it's Flag Day on and devops and bound Flag Day talking about features like so let's get right into it.

All right there what there were Punch Cards involved though? Right Mitch at least one deck of them. Yes.

I remember they anyway guys were here to talk about feature Flags today and it's gonna be a great conversation, you know our audience. I I pride myself. That our Tech strong audience is rather sophisticated from a devops point of view.

com. You know in technology in general but there might be people who are watching out here who say I think I know it a feature flag is what does that have to do with cookies right there? They're not 100% sure.

So who wants to tackle? You know what hope I'm afraid we're gonna call on you to lead it off if you don't mind. Why don't you kind of Baseline for the audience?

What what do we talk about here? With feature Flags, right? I will say it in this way.

So if you are deploying software and you don't want to make a big bang deployment or you don't want to let's say test everything in production on everyone at the same time. If you use feature Flags, you can Target your deployment. You can have a very small test group.

So if you want to test with let's say an affinity group that knows they want to be your beta customers every time you can roll out to them only see how it performs and if it's going well roll it out more widely, maybe also in a Target fashion, but if not, then you can have an easier way just to turn those features off and and make your improvements. So feature Flags give you a lot of control on who can see what as your end users and and you can also make their experience better by sometimes targeted Improvement that's targeted enhancement. It's just And I would have if you've ever been forced to commit by your lead Dev before your code was ready you feature flag something.

That's fair enough. So, let me let me let me play Devil's Advocate or chill in the audience. So what's it there between feature flag a name AB testing?

I'm gonna jump in on that one here with you. I think one of the big differences is you don't have to deploy code with feature Flags. It's already out there just enabling and disabling what's there?

So you might still want to play code for future features or whatever, but that's one of the big advantages is just turning things off and on without deploying any code. Yeah, I agree that certainly it you know. We look at or maybe it's me when I look at feature Flags.

I look at it from sort of what is the end user experience? Right and then user experience can have an A or a b or a c or you know multitude of different experiences depending what flags are turned on or off? But I think another way of looking at this is what's the view from the other side of the dashboard.

What's the view from the development side or the upside or the testing side even and and how do we interpret? You know what? We see with feature flags.

And then how does that? influence What we're doing? Adam you wanna yeah, maybe take this down that rabbit hole.

Yeah, one of the things that that happens a lot of times is your implementing feature Flags, you're you push something out and and you run your tests and find out that. Oh, I I only tested one of the past if the feature flags and it starts to get even Messier when you get dozens of flags hundreds of flags thousands of flags, and how do you how do you test all of those and and ensure that you're actually testing all the different configurations and all the different possibilities and and it's a mess especially as you start having flags that are should be retired. They've been there too long.

They're not longer used, you know, do do I even toggle this flag on to run my test because this thing's been around too long now and and nobody's flipped it on or off. So, I mean, you need need good feature Flag Management, you need a good understanding of what all the flags are what they do. What should be in in different scenarios and what aren't without a Out of that.

It's impossible to set your flags correctly whether you're talking about production usage or testing or anything else. And Adam if I if I could and I'm glad you brought up QA and also proper management, right? Because pop properly created configured and managed when we talk about the value to QA.

You know, there's there's a number of things one thing for instance. I've run into a lot as agile and devops and and you know two weeks Sprints really start to take hold this that a QA often has a hard done keeping up. Right?

So one of the things using QA is an example, you know that you're able to do is you're able to deploy the features that you have the ability to test and then you know, you can you can toggle Test toggle off for production test then toggle in production which speaks to an overall value of feature Flags. There's a similar benefit to the people that are responsible for deployment. You don't necessarily have to wait for a feature to be fully tested, right?

You can deploy the package and then the team can turn on features as they build they're ready to be rolled out to production either to a whole group or as hope mentioned progressively to other groups. So I like to say there's there's them feature Flags enable continuous everything for everybody in the sdlc. Yeah, right building a big bang deployment.

Sorry. Yeah, I I've long said that that you can't have a continuous delivery unless you have the three-legged stool of continuous testing feature flagging and observability. You have to know that what you put out there is working you have to be able to control what people see and you have to be able to test it all and and you have to do that all in both production and lower environments, you know, we used to use I tested in production as a pejorative and now it's it's a way of life, but the thing is is you can't test only in production and you have to test everything you have to test to make sure your feature flags are working.

Did you actually talk off the thing you thought going off? Right and I think going back to though the point around a b testing. Um, a B often is for some organizations a trial bling.

We're gonna change something we're gonna change something relatively small who does it resonate with that point. Often you don't really need the QA team involved. They're not going to do integration testing or end-to-end testing on something like that.

But then when you start to dive deeper and you say we're choosing B, everyone rally around that and we're gonna swarm and build we're gonna build B and deliver that to a wider range of customers that gives a lot of those QA teams enough Runway to understand what's coming and and hopefully prepare for it a little more so they're not left behind as much as they sometimes are. Yeah, there's there's two different ways that that happens. I mean, you've got either A/B Testing as I'm trying to decide which way I'm going to do and in which case your QA probably should have tested both of them and make sure they both work but also, you know, I increasingly see feature flagging and a/b testing and is a as a substitute for some testing.

Hey, let's let's spin this out to two or three users and find out if anybody screams. Let's set our users be our QA, you know, that that may be appropriate in some things. It's probably not appropriate if you're testing, you know nuclear missile alerts and in Hawaii and you oh, yeah, that's not tell everybody that you're that's what I remember that yeah.

um, so let us now jump into you know, I feel like we've laid a good foundation here on featureplex, you know, the the title of today's show is feature flags and microservices oil and water. You know, we we've seen feature Flags obviously become sort of mainstream devops. I think we haven't said it if and if we haven't I'll say it now it's because it's it directly relates to one of the fundamental.

Devops principles which is feedback. Loops. Right.

I can't think of a better example of feedback loops than feature Flags actually now. but the world's changed since we first you know read the Phoenix project that almost 10 years ago now right and Cloud made of microservices has burst on the schools scene. We are seeing these monolithic applications being deconstructed and and we deployed as a stream of microservices that coordinate and work together There are some people who say yeah microservices doesn't work.

So good with feature Flex. Well, if you believe they do, let's hear why or you believe they do not let's hear why not? What's the problem if you believe they do let's hear why.

I'll throw it open to any of you to kick it off and and I'm interested in everyone. I think it's important Allen that that we kind of subdivide the space. And as you said earlier, I want to kind of underscore we we've talked a lot about user featuring feature Flags or feature flagging user facing features, right?

But let's also keep in mind that that we can feature flag back end and we can feature back at Future flag back and it almost any level of granularity. So I'd say on one side of that argument when we say microservices doesn't work with feature Flags. Well, not necessarily if I run a micro service I can feature flag out components or functionality within that microservices as long as I ensure.

I'm preserving my interface contract with the people that are consuming my service right? But then now when you start to talk about what I'll call system level feature Flags, right which are usually those could still be back in. I'm going to grossly say that those are your user featuring feature Flags.

Now what happens with microservices, of course is you can have five 10 15 50 different micro Services many of those run and developed by different teams that all roll up to represent a user-flower user-facing feature. Right? So now we get into we've broken into these individual teams to enable autonomy and velocity but way to set how do we establish that there's a feature flag that we want to set to to turn on or off the system feature across 50 micro Services, right?

So now you do kind of get attention because to feature flag out that user facing feature across those 50 micro Services. Now you break some of that autonomy Orton Atomic benefit of microservices. So I lay all that out to kind of set the stage and say why I know we have Solutions encounters, there is there is a some real validity to the perception that feature eggs and microservices or oil and water Yeah, and I I believe that.

You know pretend there are two of me. I'm an a/b test right now. So, I believe that microservices can work with.

Feature Flags, if you are using it to validate changes in microservices, so if you have so many that it is hard for the team to manage and I and I've been with teams that have had challenges just managing that distribution if you set it up so internally, um, it can validate changes the individual microservice and your production environment fantastic the con for me. is one of the things that microservices was supposed to get us away from was all of the deep coupling that that we've seen um and the additional complexity. So this is taking us back sort of in the direction of sorry service oriented architecture.

They work for some may not work. Yeah, I I questioned the entire promise that that anyone thinks that these things shouldn't be working used together. I I think you'd be hard-pressed to find a developer who is implemented microservices at any sort of scale who thinks that that feature flagging or some way of controlling deployments, you know is Brian pointed out you've got the the contract changed.

How do you ensure that the clients are in sync and that that's part of that that decoupling of the the tight coupling if if I have to deploy all of the clients or all the dependent Services every time I make a change it's it's a mess and I'm gonna sound like an advertisement for future Flag Management Companies here, but if you don't have a good way of managing your future Flags, if you don't some homegrown system where you know, it worked for for 10 or 12 or 100 flags. Once you start getting into these complex webs of There's 16 Downstream dependencies and all the flags need to be toggled at the same time. And in the right order for that to happen.

That's that's gonna be a mess and then you know again sounding like an ad for the the company I work for, you know, testing those and and having good contract tests and knowing that that the different systems are able to to use the right API contracts and that they're getting the ones that they thought they got when you do change those those flags is is crucial other young why is you're going to have Downstream problems and and unexplained stability and very very hard to debug that things. You know, I remember there were maybe a handful of Articles making that argument right? That's been a few years back, maybe three or five or something like that.

And to me it's it's run amok run uncontrolled feature Flags could bring chaos to any environment just like, you know duplicative microservices and you know subdividing things too far going too far with it, you know, a big Advantage can be our worst. Enemy. I think that the thing and I'm not making the premise that there aren't made for microservices.

Of course, you can use them in microservices or not. Right? But I think what it does is add another layer of complexity to it and what you were saying Adam about you have to manage that across the team not just within a team right you could have people implementing duplicate feature Flags, right?

Just imagine the folks in in an Ops environment whether it's Deb or SRE or an Ops person, you know with hopefully and observability or platform trying to put all this together, right? What versions of What code were we running when what was serverless if that was maybe in the mix and what were the future flags that were set at? Time did anybody log it or have some way to track all that?

So it does add another layer of complexity. So I think you've got to be applied good engineering but also kind of product engineering product management across the team to bring a little bit of discipline to it. So it just doesn't become Insanity.

Right? And I and I think you know also there's a bit of kind of human interaction. I think you know, you've probably heard me mention we've done a few of these saying at the end of the day humans are endpoints of code right there where it's created.

They're where it lands and what does happen as you coordinate feature Flags across these multiple services at the end of the day, you need to enable teams to coordinate right you think about the scenario where maybe I have a feature flag where I've I've you know, we're trying to take a piece out of the system. I flag something out that within my certain microservices turns off with some function effectively to put it generically now I call into that interface contract is the same but I'm getting a null value for one of my variables one of Yeah, and and all of a sudden my service dies. I don't know why.

So again this is where you have to have systems and processes to coordinate those flags. You have to have people working together and and then I'll get to what we haven't talked about. I think for the audience here's where interfaces start to become important.

If I recall Martin Fowler's initial blog about about what is now become feature flags and ultimately feature management. I believe he did talk to having to manage that it code level at scale. Right and I think it's implied and most of our audience probably gets it but let's call out that when we talk about Enterprise feature management and Enterprise feature management across micro Services UI and ux to check control and toggle those flags is critical right, but it's not inherent in the definition.

But then again when we go to Enterprise feature management one thing that you should think about here is not only a dashboard to turn them on and off. But a dashboard that provides a shared View and leads to a shared understanding of what's on what's off. What's flagged Etc.

Well here that goes I'm sorry good Adam. It's not just useful in this. So it's crucial in order to if I'm going to change an API contract and I'm going to to break things down the road if I can run two versions of that that API so that somebody can can then run against either one and provide a deprecation in order for them to effectively roll this out and manage this across the Enterprise you have to do it through through feature flagging the the Upstream teams need to be able to not do these big bang things and whether it's a feature flag through literally implemented in code or whether they're treating their deployments as a feature flag, you know.

Hope do you have a lot of people using feature Flags to kind of manage deprecation of things. Have you seen that very much at cloudbees? Um at cloudbees, I'm a little more removed from the day-to-day, but I think there are a few in previous organizations, I think.

it was something that was always a goal but was hard for the teams that I was working with to do. Well, right it was more of A manual task for some of them and but I think partly because feature flags for them at that time were a lot newer and they were more focused on the implementation. How do we clean up all these flags that we see out here, but it did definitely how um, it did help with with branching.

Right? So once you have the flag turned off, we don't have have all these Long Live future branches that part was more automated, but but not so much with Other declaration. Well, right hold it Enterprise feature management that that idea of not just what one of the great things if you're if you're using an Enterprise feature system and it's across the entire Enterprise is I can go look and see does anyone still have this feature implemented.

Is it is this code path now dead and so yeah where where I might be able to look in my microservice and go. Oh nobody's hitting the microservice. It's it's safe to turn off this version is it though is this a microservice that somebody hits once every six months and yeah, I turned it off but it breaks.

But if I can go look at a source of Truth and say hey, everybody has moved their feature flags on to the new version now, it's safe to actually do that deprecation. That's cute. Yeah.

Yeah and and looking at which flags are off wish which flags Have basically been removed. You know, that generally is saying we have accepted this. This is our path to production.

Now that has been useful for a lot of teams because it's already somewhat confusing for some depending on how they've Implement their feature Flag Management that I think is always a a source of conversation in most organizations. But if they have their continuous integration continuous delivery paths, mostly work through and they have integrated feature flagging into that and every team has understood this is part of our way of working. I think it speeds everyone up.

It's a great practice. I know I know Alan needs to shift our subject, but I want to jump in real quick to underscore because Alan asked earlier how do feature Flags affect the experience for people that are upstream and I think you you and adamant on a key thing ultimately. On some extra work to coordinate it could simplify your code.

It can make your code less brittle it can reduce branching and can make your code less complex. It can free you from having to to manage deprecated code to the same level. Absolutely, but you know, I think one of the things we've skirted around, but we really haven't hit on.

And I think it directly relates to the whole microservices think is the issue of scale. Right when we talk about feature Flags. You know, this is not just is the background red or blue is my button rounded corner or Square corner, you know, like interface kinds of things like that, but the behind the code they're in the UI.

Ux. I mean the sheer scale of feature Flags. at some level feature flags as it relates to microservices has almost been a victim of its own success.

In that once we start using them and we and we see the good. Right, we got that feedback. We see what we can do with more more, you know button down the hatches man.

We just keep adding more and more feature Flags. I I've seen you know, we've done Series in the past videos with some of the feature flag companies and you know the amount the sheer number of feature flags being deployed is literally in the billions at this point, right? I think that's very regular basis.

We've met the enemy and they are us right. Yeah. Okay.

So so when you take that kind of scale and then it bumps into microservices Which is trying to you know, certain scaling microservices is yet another issue. We probably do another devops and down, right? It really kind of is oil and watery a little bit, right?

It really does. You know how and what these answers? Well, you got to use a feature flag program for this because the scale is just you know, yes, you have to use a feature flag program.

That's a that's a no-brainer. But what is that feature to flag program have to have that a really allows you to. To work at that scale in microservices kind of architecture environments and still be able to make the testers happy and the developers happy and the security folks happy.

As well as the end users. It right it has to have knowledge. It has to have knowledge of everything that's out there and be able to expose that.

It once you have tens of thousands of flags, where are they being used who's using them? How often do they being used? What code pads are active which ones are not which one should we be retiring and and that's where building your own seems kind of nuts.

I mean look, you can build your own you but you know, you can build your own credit card processing system too. But let's set your core business, you know, go go use something from someone else, you know, whether it's it's Cloud bees or one of their fine competitors, you know, it's It's scale with microservices knowing what things are being used. And where is just crazy crazy important.

I would add that Adam and it's also what are they for? What is this? What is this feature flag for and do we all use it the same way?

There's also a school of thought of treat feature Flags having a life cycle. Right because they may be introduced for some internal testing. I think you give that at that example Adam, but it could be for beta features that are being used by certain users and Market.

We could, you know, flip the flag and be in unreleased day the next day. It could be for duplicate. You could actually have duplicate code right with variations of it using feature Flags to turn on or manage which ones are turned on and what scenario but I think you want to avoid the problem of you know, Some future flags are meant to stay around but not all feature Flags, right?

They do need to have an end of life if they were for a specific purpose like that. So but but that's that sort of life cycle management treat it as a proper product or part of the product. So I think that's part of where you're talking about Adam.

It's knowing what's there but also why is it there? What's it doing? And how are we going to use it today tomorrow and then maybe stop using it if we do.

It's not primary Mitch. I think you call out a key thing when you talk about change soap. So Alan before we get to kind of what does the solution need, you know, hit a couple of things.

Look you you oftentimes feature flag, you should grows from bottom up. I think now we're starting to see it come in from sort of top level engineering practices down or product. Um, I think in either of those similar devops transition you get a point where you need to have a thoughtful deliverable rollout, right?

So you need to have change management you need to have communication. You need to have alignment around the philosophy of why we use it. What do we flag?

And then to your point Mitch a life cycle a couple of things. I'd like to call out. Hope beforehand it right over to you is gonna be when we talk about what ideally do you need to see in a feet in an Enterprise feature management solution?

Well, we need, you know, we need language support right reading language what we need stubs ideally. That allow us to easily integrate those feature Flags across possibly the multiple languages or across our Tech stack, right? Ideally, you would have those stubs be able to grab some of that intelligence that could then be rendered that that Adam would call out right?

You need a a rich, you know easy to use user interface that hopefully can either extract or enably you to tag some additional context around those feature Flags so on and so on but hope I'll stop there because I realized you were you were jumping in That's okay. That's okay. Um, one of the other things that I think is really important that is sometimes overlooked when talking about feature flagging and especially if it's a homegrown solution is what is your governance model?

How are you addressing security? How are you addressing role-based access? How are you auditing to understand?

What changes were made? That's something that's relatively simpler to do if you have a small team. But if you're talking about dozens and dozens of microservices thousands of future Flags across a large Enterprise without having some mechanisms in place to manage those things you can quickly you lose control of what's happening and and how your processes are.

Yeah, I remember the first time that we started talking about feature flagging and somebody was saying well, we should look at some of these vendors that are out there doing this. I said you that's insane. Why do I need to Outsource my if statement and then we ended up with a few dozen of these across a few dozen different microservices that I went.

Oh, that's why I need to it's not Outsourcing my if statement, it's Outsourcing all of the decision making and understanding that that's behind that if statement. Right or providing infrastructure for the decision-making around that interesting that if statement, that's a good window Outsource my if statement, I like it. but guys, let me you know, we we're more almost done here.

There's a couple of things I want to make sure we we hit for our audience and You know when we talk about scale and feature Flags is a scale issue today, right? If you're using feature both Flags, you're probably using them at some scale best practices this, you know, a spreadsheet isn't the option Adam to your point about home growth. I can't imagine people using.

feature flags and microservices and these things and trying to track this stuff like in a spreadsheet you've got to be using and there are there is some fine feature flag. Management Solutions out there Cloud bees has one. Of course, is Hope mentioned but there are others too.

I'm not here to endorse one over the other. I'm just saying use one. Right, you can't be serious about using feature Flags without a feature Flag Management program.

you can't be serious about using feature Flags in a microservices or architect or architecture based application without having some feature Flag Management product that really integrates in your your development team your QA as I mentioned before security folks all of that now. That all being said, what is that what you know, have we had feature Flags matured to the point? Where we see sort of a best practices.

Right when and if so, what are some of these best practices Beyond using a real? program to manage them Anybody I I feel that one. future best practice let's say there are a lot of organizations who are seeing future Flags aligned to Features.

Absolutely because it's called future Flags, right but understanding how that is tied to your business and what your business wants and your business outcomes is also really important because Focusing at too low of a level may not let you tie back to what your overall organization is actually trying to accomplish and if there's a major change being able to able to narrow down which work is tied to those flags that affects that business outcome could be pretty critical in the future versus someone going through even with an application and having to pick through and make those associations. And and I'll add to that I think are extension is a comments earlier. I think one is aligned around a what and why similar to Next Step kind of down from the business objectives across these multiple, you know, two Pizza teams microservices teams are we used to describe them above that?

There should be a we view feature Flags is this we use them in this case. You can even categorize or type them. So you have an internal lexicon to talk about it.

And then again what a system or a solution won't necessarily take care of you alone is gonna be how do we communicate what we're doing across those which will then get to my my second best practice that I would recommend and that's to do regular review cross-functional cross team reviews of the feature flags that have been established evaluating, you know, one how they impact other systems to are they still needed. um And so two of those prep those two practices can help better align on how do we establish and maintain them? And then how do we how do we track and deprecate those so that feature Flags don't end up creating more technical debt than reducing Tech.

I'm I'm gonna disagree a little bit that I don't think feature flags are resolved problem. I don't think that this is I don't think we know enough about the problem domain to have a lot of good best teachers and what is considered a best practice today, maybe a terrible practice once we start learning enough about it, and you know, we've got We've got feature Flags used in so many different scenarios. There's there's phased rollouts.

There's a that's part of A/B Testing. It's part of you know, all sorts of different things and treating them all in one way or trying to to establish a practice for feature Flags is probably going to narrow our thinking down enough that this is such a a relatively new thing that we don't really understand the behaviors and the and the governance models and the the human behaviors on the back end of this well enough to start saying these are the best ways to do things and we should we should let teams explore and and find their own best ways right now that trying to to set any sort of a best practice maybe it's kind of restrictive in the in the growth of this and understanding of how this stuff works. There.

Yeah, I I worry I get the point but I worry that it could that that there's a you need to strike a balance between freedom and exploration and and to to Allen's point from earlier. Jess having people create feature Flags littering the code with feature Flex, right? And and I do so yeah, I think that's an important point out Adam, but I'd say that but I call out to people there has to be a balance right just letting people run can create a ball of yarn that's hard to hard to unravel.

Yeah, and I'm gonna Concur and brings to mind. In some organizations where people say we want to experiment we want to innovate right experimentation and innovation. Our processes and practices to find a way to feed back what you've learned how you've improved into the organization.

It's it's not a free-for-all and even with I think best practices. Marketing back to Agile you look at something you say. Okay.

Here's the best practice in the industry today. Let's try it. We see these items don't work for us.

We're not going to treat this as if it is, you know, Something we can't deviate from but everyone aligning around a core and at least trying those things helps you find out to me faster. What doesn't work in a way that you can feed that back out into the organization. Yeah, I just I just worry that we're going to end up like we have with agile where people go well agile means scrum when we're going to implement scrum and and we're we're abusing developers with yeah ceremonies and I agree not gonna get past that we're ready.

That's I don't want to repeat that same pattern with feature Flags. I I Bristol at anything called the best practice Yeah partly for that reason. Yeah.

No, I I you know, we've seen it with devops and ads out right no matter how much you disclaim. Look. These are General guidelines bring them in and Implement them in a way that's fit for your organization yet time and time again for agile and that people will come give me give me the canned food spam version of this please so I can go deploy a thousand times a week.

You know, that's just not gonna yeah, so I get you on that. Yeah, I'm gonna add one thing and being wrong. I've I've been wrong at least once before so so it's okay.

Yeah. Some Al cap this off Allen this way is that I don't think any of us want the Dogma right? I think that's what we want to avoid.

Actually. I have a different thought. I think we need patterns not best practices.

Just think about developers think of Singleton. Is this Etc. I think that's that's what we need for feature Flags because there are great patterns for how people use this.

It isn't just kill switches and functional features switches Etc. There's more to how you use it. I think that would Advance the state of the art the best.

I don't think there's a book on that yet and maybe someone here wants to write it. Let's do it. Be careful what you asked for.

Oh. Hey guys. This has been a fascinating conversation.

I I think it's obviously on topic that we can talk a lot more about. But we don't have time today. So hope Adam Bryant.

Thanks for joining us on this episode of devops on bound. Thank you for watching. We hope you found this interesting and you and learning a little bit about future Flags.

Um, thanks to chase sent us responsoring. That's it's always, you know, they're great partner to have on these and and I think you can see why listening to our conversations I invite you to watch the next devops. I'm down which I think is a live Round Table Edition coming up and Or depending when you watch this.

It might be after because I forget not everyone watches it the day it comes out, but you can catch. Future episodes of devops on Daniel also find them on Tech strung dot TV as well until then though. This is Alan Shimel for Tech strong as well as well.

With Mitchell actually and Adam Kelsey Brian Dawson and hope Lynch. Thanks for watching devops Unbound.