The Sandbox Solution: Closing the AI Testing Gap

Hey everybody, we're back at Amsterdam, and we're here at the KubeCon + CloudNativeCon Europe Conference, and I'm talking to my friend Valdemar here about a new version of LocalStack that's out. And LocalStack is basically a local replica of an AWS environment for developers. Valdemar, welcome to the show.

Thanks for having me. Good to be here. So what's new in the latest release and the latest update, and what can I do today that I couldn't do two days ago?

Yeah. So we had some exciting changes that we pushed out yesterday. So we basically unified our community offering and our paid offering into a single unified experience.

So previously we had an open source repo, that was on GitHub, and we built an image there, and then there was a separate experience for our paid customers and the paid offering. And we decided we want to unify this into a unified version. We also provide a free tier with pretty generous, for non-commercial use.

And you actually get more functionality and more features also than we got previously in the free version. So that's kind of new. We are generally always evolving and implementing new features in the emulator of course.

There's been some exciting developments. We have AWS as our flagship product, but we also recently launched a Snowflake product about a year ago. And we are about to launch also Azure in the next couple of months.

That's currently on a private preview. How do you keep those things in sync? Because the AWS cloud environments are constantly changing, so how does that kind of stay even with what's happening on the local desktop of the developer?

That's, yeah, a great point. So we build a lot of automation and internal tooling to keep up with the pace of change at the cloud providers. So that starts from, for example, scraping all the API specifications.

Those are public, and then we can see whenever there's a change in the APIs, we can start implementing the change on our side. And we built some pretty sophisticated, just testing validation mechanisms that make sure that we maximize the parity, as we call it, the fidelity of the emulation of what we're building. But yeah, I'm not going to lie, it's a catch-up game and it keeps us busy, for sure.

Yeah. So who wakes up in the morning and decides that this is their problem? " Right.

Who takes ownership of making sure that whatever gets built locally can actually run on the runtime? Yeah. So this is really sort of what our kind of bread and butter is.

We have an internal prioritization mechanism. Of course, any larger customers that we work with, we know well in advance which kind of services, which kind of workloads they're running, so we can more proactively make sure that everything works. And then when it comes to more the broader community adoption, we just have a backlog of feature requests that we just work through in general.

But yeah, this is really this high fidelity environment that we offer is really sort of the value proposition of what we bring to the table as a company. " Yeah. " Exactly.

So we're working more and more with these DevOps or platform engineering teams that are basically rolling out developer experience for the entire organization. And typically it's something like, you have a make script or something that stands up your local dev environment, and they are the ones that are kind of standardizing it across the org, right? And we work with these platform engineering teams to make sure that they have a consistent experience across all the devs in the organization.

There's also some insights and telemetry that they receive, if they use our paid offering to see which kind of services are being used and how many instances are basically running. But yeah, it's all about having a central entity, either a cloud center of excellence or a platform engineering team typically that we work with. We, of course, live in the age of AI now, and you can't walk down the aisle here without somebody leaping out to tell you about their great new AI thing.

How do you envision AI playing out in this world? Because, I see AWS is playing around with AI and- Yeah ... will you create this complimentary set of agents, or how does that work in your mind?

Yeah. So the whole AI acceleration is actually something that plays very much in our favor. We did not plan for that when we first built LocalStack, but now, the whole industry is talking about sandboxes and how you want to make sure to have a risk-free environment that these agents can operate in.

And LocalStack is the ultimate sandbox environment in that sense. So what we started doing is we built, for example, an MCP server integration that makes it very easy for these agents to talk to LocalStack, do a lot of the testing. And it just accelerates the feedback cycles also for agents.

So there's a couple of comparisons that we have on our website. It takes maybe 15 minutes to spin up an RDS database cluster on AWS, minutes, and it's five seconds on LocalStack, right? So those are the kind of side-by-side comparisons, and those have compounding effects the more you test and the more agents you run in parallel.

The other thing we hear a lot about these days is digital sovereignty. And, it's no secret that a lot of developers work in one country and they work for companies in another country. Mm-hmm.

Is this mechanism that you have kind of giving me a way to hire developers anywhere in the world and give them local instances so that I'm compliant, but when the actual thing that I want to run, I might want to run somewhere else? 100%. Yeah.

This is something we see a lot that, if you have a global workforce and you might have different local regional jurisdictions or different legal systems. So yes, we're kind of democratizing that in a sense, by giving you more on-premise or local execution environments before you hit the real production. And this whole topic of sovereign cloud has come up a lot at this event also.

I think in Europe specifically, it's a big topic, presumably also in other parts of the world. So it will be interesting to see where that goes in terms of, is Europe going to create their own cloud at some point, or is it more like a region of AWS or some of the big cloud providers? But it certainly, again, is very much in the same narrative that we also play.

It's decentralized a bit more, shift left, test more at the edges or at the end consumer. So interesting trend also that plays in our favor in a sense. " Yeah.

I think we definitely have a set of best practices how to ideally set up LocalStack and get the best value out of the tool. For us, a lot of it is just educating the market for some of the more advanced features that we're offering. Because the foundation of what we do is these emulators that give you this local cloud environment, but we're also building a lot of what we call DevX features, things that sit on top of the emulator and give you additional developer value for debugging your stack or doing certain experiments.

We have a feature that we call Chaos Engineering, for example, where you can inject errors or latencies into these APIs and test for resiliency. So it's mostly about educating the market of what's already possible with the solution today that goes beyond the pure emulation part. Are you seeing a shift towards platform engineering as kind of a larger trend?

And what does that platform engineering team look like compared to, say, a traditional DevOps team or even a loosely affiliated set of developers? Yes. So surely, the term IDP, internal developer platform, is something we hear a lot from customers.

There is some off-the-shelf IDPs that are being used, but also increasingly teams building bespoke solutions within the organization, within the enterprise. And almost every large org we work with has some kind of a DevX team or a platform engineering team that is following, if it's a regulated environment, for example, banking, insurance, they might need to follow certain internal rules or laws. So yeah, it's very much sort of a core offering, and then you have these satellite integrations that are usually making it more seamless to integrate it into a certain organization.

Yeah. How much development is still happening on a local laptop versus when there was a push for a while there to kind of push all the developers into these cloud services. Mm-hmm.

But it seems the developers kind of like holding onto their machines. Exactly, yeah. Because if you have your IDE set up there, you can set break points, and you'll have deep inspection of your stack.

I think maybe also with agents now, this idea of these cloud development environments is coming back a little bit more because when you shut down your laptop, then the agent dies, right? You want to make sure that you have some consistently running instance in the cloud. The nice thing about LocalStack is it's a very portable environment.

You can bring this on your local machine, in a CI/CD pipeline, it can be in a cloud development environment. So it doesn't really restrict us in terms of where it runs. It's more we're alongside wherever your code runs or your agent runs really.

Or is the developer just going to wind up with two machines, one for them and one for the AI agent that runs a couple of tasks and then ships it over to them? Yeah. And I think we've seen those with things like Cloud Code, sorry.

Was it... Yeah, Cloud- OpenCl I think. OpenCl, exactly.

Yeah. To kind of using some of the Mac Mini that's maybe in your home environment and just spinning those up and having this agent run there. So I think it's kind of a bit of a resurrection for compute that is just up and running all the time, and you have things running that are reacting to events or doing work on your behalf.

Yeah. At least we're all talking about the fact that in the age of AI, the pace of application development should accelerate considerably. Mm-hmm.

" My personal view is it's probably the latter. So we need to fundamentally rethink software engineering best practices in general. So what we're already seeing today is a certain review bottleneck, right?

So a lot of code is being generated by these agents, and it's still predominantly humans who need to now review that code, have a conversation with the agent, something's not working and so on. So I think just everything that's been built over the last decades around human-centric engineering, I think will have to adapt to a certain degree. And I think the other interesting area is code generation is kind of growing at an exponential rate effectively.

There's just so much code that's being generated, also sloppy code in some cases. But the actual code that makes it into production is not growing at the same pace, let's say. So there's a big sort of gap, which we call the testing and validation gap.

We need to give these agents an environment to test and rinse and repeat in terms of the logic and have also strong quality gates in place. So I think over time it's going to be less maybe about the actual code logic. It's more about making sure that you have guardrails and quality gates that really define the outcome that you want to achieve with the application effectively.

So how do you envision this playing out? Each developer's likely to have multiple AI agents that they're using to build. Mm-hmm.

There's probably going to be autonomous AI agents on the back end somewhere on the CI/CD that's managing certain tasks on behalf of a bunch of developers. Mm-hmm. How will all that get orchestrated?

Not to mention all the agents that might be sitting up in the cloud. How are we going to kind of manage that so these agents coordinate? Coordinate, yeah.

So there is already some level of, for example, this new paradigm of spec-driven development, which is something that's now emerging. So there's already a certain specialization happening that you have agents that are a bit more like a PM, like a product manager, and then there's another agent that's a bit more like a software architect, and there's another agent that's more like a senior or junior developer. So we're starting to see some specializations already in the planning and execution phase of how software is being built by agents.

The other interesting trend is kind of adversarial situations where you have one agent maybe generating code, another reviewing, and they kind of critique each other essentially. Or the most advanced use case will be like you have a fully multi-agentic use case where everybody's trying to achieve the same goal, and then you have a weighting function and the best solution just wins in the end. So I think those are the kind of scenarios we're probably going to see more in the future, I guess.

Will these agents kind of argue with each other then at some point, just like humans do about certain things, and how will we decide to resolve that? And what's going to be the language that they talk to each other, right? Is it going to be English?

Yeah. Yeah. I mean, definitely interesting times.

I think we're going to see much more in 2026, 2027. So it seems like the industry is in almost like a-A pivotal moment right now with figuring out this next wave of developments. And as you said, a lot of the booths here are about AI agents, so interesting times for sure.

So let's pretend that you have been made king of all things IT for a day. What's that one thing you would fix by decree if you could? King of IT.

So I would focus on efficient utilization of my resources, so making sure that there's no waste and no idle resources that are not being used. So that's the first part. And when it comes to application development, it would be focus on the guardrails.

Really, in order to accelerate things, you still want to keep the quality in place, so testing and test quality and coverage become even more critical than in the past. So yeah, those would be the two points, resource efficiency and guardrails. And we talk about things like FinOps out there, but it seems to me like developers aren't inherently wasteful, but they don't have a lot of visibility into how much infrastructure is being consumed and what it costs.

So- Mm ... is that something you can provide using your platform? Because I don't think they understand how much it might cost to run something that they're building on their local platform.

Yeah, 100%. So I think Werner Vogels from AWS mentioned that every engineering decision is a buying decision, right? Because as a cloud developer, if you choose to go with one service or the other, you implement it in your code, but it has a direct cost implication in terms of how much your organization is paying for it.

So yes, 100%. The one part is having a rigorous process, especially for dev and staging environments, to keep those effective and efficient and not creating unnecessary costs with our solution, for example, but also if you test in the cloud directly. And then, of course, in production, there's all kind of FinOps optimizations that you can apply for auto-scaling and other techniques.

But for the part that we focus on, it's really anything that's pre-production, and there's a lot that you can optimize and automate by going more onto your machines, more local, and local testing. To your second point, there's always not been a whole lot of love between the security people and the application development folks. Are there things that we can do on the local machine to reduce that friction so that I can maybe, I don't know, have a set of best practices that are running locally, so at least everybody has the same experience?

100%. Just from an onboarding perspective of a new engineer with a solution like ours, there's no need for cloud credentials or giving them access to the kingdoms in a sense. It's just getting started, up and running real quick.

The model that LocalStack provides and offers locally is even that you can test your applications and develop them even without IM or security built in. So we have a way to enforce IM locally in LocalStack, but you can also turn it off, switch it off. If you're just interested in the functional piece and you just want to develop your app, you don't care about the security piece, and then you can maybe do that at a later point in time.

That's also feasible and possible with LocalStack. But yeah, to your point, I think security teams will be happy because they don't have to distribute credentials to everybody and do key rotation and whatnot. Don't have to take care of cleanup scripts that are running every night and clean up unused resources.

So it's just a fundamentally different way of thinking about resources and development. Yeah. All right, folks.

You heard it here. Well, it turns out laptops and the cloud can live together after all in harmony. Hey, thanks for coming by.

Thanks so much for having me. All right. We'll be back in a bit.