Donald Lutz – Platform Engineering and the Internal Developer Platform Framework – DevOps Experience 2024

So today I'll be talking about platform engineering and the idea of a platform engineering framework. And the past three years, platform engineering has become, uh, a field which has grown as people have moved to Cloud native and also use Kubernetes. So I'm gonna kind of go over where this came from and how we got to platform engineering.

So the traditional deployment model was really manual. You know, you use scripts, you have different tools and processes. It was difficult to standard or scale deployments.

And, you know, this was kind of even, this was even pre DevOps, but even in the DevOps era, this was kind of the standard way it was done. There were just more standard tools that people use like Terraform Jenkins, but conceptually it was still this model. You know, the challenges were that it was, you know, each deployment environment was unique.

It was hard to reproduce and maintain consistency as the applications grew, manual deployment process became more time consuming and ever prone and manual deployment required more personnel and resource and increasing cost. So even though equ as DevOps grew, even though you had better and better tools, that still resulted in more, you know, personnel and more, uh, processes that you had to deal with. So with this, you got to be origin of platform engineering because we needed to standardize scalable environment, just cloud computing.

You needed extraction and automation into the DevOps process, and the rise of containerization and orchestration kept on expanding. And also the popularity of serverless computing. So as all these things came together, you know, people started saying, well, how are we gonna, you know, deal with this?

You know, we need some kind of idea of a platform, something where we can put all this on and allow it to work and make sure everybody has a way to do this and make it work. And then there was sort of the question, why do we need platform engineering? You know, is is it valuable?

Are we just creating more abstraction? We don't need more complexity, or is the reason why we should do it? And it really made people realize you needed a dedicated team focused on the innovation of the application of services, that how they deployed and operated production.

You wanted to de development teams focused on their, the business features that need to be released on the applications that the end users needed. And you needed to line out. Teams deployed and operated the application across the organization and it really became apparent to development teams lacked the skills to self-manage and do this.

They didn't have enough time to both build the applications and services and deploy on the, on the platform. It was a very complicated problem. And that's how we sorted into this idea of platform engineering.

And the slide, it's a very complicated definition, but fundamentally it's a discipline designing and focus on building tool chains in workflows that allow for self-service capabilities for software, uh, engineering organization. It bridges the gap between software development infrastructure by providing a robust platform that does use infrastructure code and GI ups, but it, there's other tools get involved with this like cross blade. It isn't just straight Terraform.

There's quite a few other tools that get brought into this, uh, methodology could even include things like Helm. So these are the, these are the core platform engineering principle. You know, every platform meets a purpose.

So if you're ready, SAP, your platform has to help you to deploy. You know, SAP, if you're running Azure, Azure, how do you deploy, you know, a KS in Azure, you know, but you don't wanna repeat work. So without a platform, you end up, like I said, writing scripts that may be offshoots or they may repeat it all the time, you know, and the platform is the product.

So it's not just a set of scripts or a set of softwares, it is the product that you're deploying for the organization. So these are sort of the detailed platform auto engineering principles. The first one is auto, auto automation and infrastructure as code.

You know, you know, it allows for automation and reproducible what the platform should look like. And it's easy to create new instances. You can create them on demand and you try to remove all the manual options and try to get them to zero.

You wanna focus on efficiency. It needs to handle all the challenges you have when you deploy things manually. You know, if don't recreate tools like CICD, whether it's Jenkins, you use all the tools and you read them into the platform.

So everything works as it's supposed to, and it needs to be self-service that the developers can go there. Since they have the platform, they can deploy their apps, they can build their landing zones, all the components of the platform without having to involve anybody else. It doesn't mean the platform doesn't evolve, it's just like any other software project.

It keeps on going, enhancing it is a continual evolution as it keeps going. You know, and these are sort of the detailed methods you need, you know, and it's sort of, it's making real big changes from, you know, Hey, what does DevOps do? But what platform engineers do and how do we get there?

You know? And here this is, this is another one, one of the key differences being platform engineer a DevOps, and an SRE. You know, a platform engineer focuses on the operations to make developers more efficient.

DevOps does, you know, automate software infrastructure, but it isn't the platform, you know, and SREs deal with maximum application reliability. The scope is, you know, you know, the platform is the whole scope. It's everything where DevOps is just streamlined development and deployed.

You know, it isn't necessary applying the tools you need to use. And SRE ensures reliability and performance through monitoring in incident response. You know, and platform view has everything from source stone, CICD, uh, Kubernetes Tools, yaml, you name it.

Where DevOps is really just the build monitor, test, automate and containerize applications and SREs deal with, you know, SLO management, automated incident reports, and even chaos engineering. And if you build a platform, right, all of the, these boxes get embedded into the platform itself. You know, so when you start looking at a platform engineering team, you have your platform engineers, your SREs, your cloud architects, your security engineers, and other rules.

The interesting thing about a platform engineer, they're also a software engineer because the platform is built with code. So the whole platform is designed and built. There are ones you can use like, uh, backstage, uh, that's from Spotify.

Uh, there's Port, there's a few other ones, but they involve either integrating or writing code. So the platform gets built out and you know, that's sort of the level of roles you need. You know, they're not, you know, the other roles can encompass anything from auditing to whatever.

It's, whatever makes the platform run the correct way, you know, but this is A-C-N-C-F model that I created. You know, it's so the platform engineering maturity, how mature is it? You know, you know, it's sort of the investment adoption interfaces, operations and measurement, you know, and it kind of moves across as a provisional operational, scalable and optimizing.

And the whole idea is you wanna get this platform to be very mature. So everything happens. So you really do have the platform when the product, so finally you can measure it, test it, know where it's going, how does it work, why you should use it, you know, and these are the real, this is sort of the CNCF definition of how you get there.

What kind of platforms do you wanna use? What does it look like, how does it go? What else could you do?

You know, and you're trying to get away from any custom processes, any ad hoc things, really getting to stuff being standardized, built and tied into the tools that exist. So the next phase of this evolution is the whole idea of an internal developer platform. You know, and this is where everyone's trying to get to an internal developer platform, is what platform engineers are trying to build or represent.

You know, it's a self-service platform that allows developers to interact with the platform. Them after understanding any operational things and infrastructure things. It's their one-stop hub for everything.

You know, it needs self service capabilities, prebuilt templates and blueprints and collaboration and documentation. Historically, the documentation would be all over the place. It would be in different source control systems.

It could be in something like SharePoint. The whole idea is anything you need that relates to the deployment of the software is actually in the IDP. You don't have to look for it, you don't have to mess around with it.

It makes it fairly simple to use. And these are kind of my representation of if you built an IDP, this is kind of how it would work if you had it on AWS plat, you have your source control, you have your workflow, you have your platform tools and Terraform CDK cloud formation. You have your CCI pipeline, your registry, the orchestrator that makes everything run your CD pipeline.

Then you have a compute data, networking and services. And then for Observable, you have CloudWatch, and then you have your Secrets Manager and HashiCorp's Vault. And this is sort of your IDP say, if you're deploying an AWS.

So this is the same platform in Azure Cloud. There's a little difference. You have a IDE with Visual Studio Code Cloud.

I can't say that. Visual Studio Code. You might have your service catalog in backstage, uh, your source codes and GitHub with your workloads, your re your, for your platform Cisco, you have Terraform with bicep.

Your CI pipeline is DevOps. You have Azure Container Registry, the platform Orchestrator, which is Symantec, your CD pipeline, the Azure DevOps. You have a KS, Azure, sql, Azure DNS, Azure Service Bus.

Uh, the Azure Monitor in secrets with Azure Key Vault, uh, the HashiCorp Vault. And then if you have it in GCP Cloud, it's sort of similar, you know, have your service catalog and backstage version get told. And GitLab GitLab application source code, code source code in GitLab your workloads.

You have Terraform, you have various automations. Uh, you'll have your CI pipeline. You have the artifact registry.

You use Symantec Platform Orchestrator. You'll have GitLab, you'll have, uh, Kubernetes engine, that's, uh, GKE. Uh, you'll have cloud risk ll cloud DNS and psa.

Uh, and then you'll have, uh, cloud monitoring in cloud longing. And then you'll have cloud key management services. So these are all the components that allow you to build a platform engineering framework.

So all of this rolls up to, if you get this all built out, you create a really nice platform engineer and structure. You have the right principles yet involved. You have built your platform.

It allows all the development teams to able deploy without having to be involved with the platform. They're separated from that. And that allows everything, uh, to proceed forward.

So if you get this done, you know, you, you're on your way to building a great platform engineering framework.