CircleCI Shifts AI Coding Feedback Into the Inner Loop

Hey guys, thanks for the throw. We're here with Rob Zuber, who's the CTO for CircleCI, and we're talking about the rise of inner loop engineering in the age of AI. And the way we think about building applications is changing.

No surprise, but maybe not in ways that we all fully appreciate just yet. Rob, welcome to the show. Thanks for having me, Mike.

Great to be here. So from your perspective, what's going on? We've had this notion of inner loop and outer loop for a while now, but inner loop kind of referred to what was happening on the developer's desktop, and then there was the software engineers and the DevOps folks who were on the outer loop.

Is that changing, and what does that look like today? Yeah, I think the concepts are there. But the desktop and what I'm doing on my desktop even, if you think about that for a second, half of what I do on my desktop is ask a remote system to generate some code for me that comes back to my desktop.

So the boundaries are a little fuzzier. And I think it was really about the fast cycle and the slow cycle, if you will. The direct feedback to me, I'm running some local tests, I'm working on a problem, and now I'm going to put that into maybe a PR, into some CI pipeline, something like that, and with an expectation that now I'm going at a slower pace around a bigger track, if you will.

And the acceleration that we're witnessing really has been on that inner loop part, right? I can get code generated faster, I can fix some issues faster, et cetera, but then we keep talking about that kind of grinding once it gets out of that world. And so I think what we're witnessing now is a desire to pull more of that capability, maybe the best way to describe it is into the inner loop, right?

How can I get all of the validation, the checks, everything else, the confidence that I was trying to build further in the process, how can I get all that done myself as the developer in conjunction with my agent or whatever tooling I'm using to build, to create, so that I can get through the rest of the cycle also faster, right? And that might be doing review. There's a lot of agentic review out there.

Why would I put up a PR that is a surface for a human to talk to another human? Why would I have my agent put up a PR so another agent can come along and comment on it? That's just extra overhead.

I could get that commenting and sort of correction to happen locally, right? So we're trying to, I think we, and I mean, we're doing this from a product perspective, but the industry is trying to move more and more of that stuff into the inner loop so that it's all happening at that clip. So the key things that we have to do, the validation, the checking, the conforming to standards, we have all the tools to do it locally in that desktop environment so that we're just executing with really tight feedback loops.

If you really think about what we were trying to do in DevOps all this time was reduce time to feedback, right? And therefore improve delivery velocity. And now we're able to compress that even further if you look at the problem the right way.

Does that make sense? It does. The way I kind of think about it is that there was a lot happening on that outer loop that we're now shifting to the inner loop, and we need to do that, it would seem, because from what I can tell, people are overwhelmed by the amount of code that's moving through their pipelines and into their CI/CD systems.

So just because we're writing more code doesn't mean we're producing more applications faster, unless we find some other way to think about this and maybe shift the weight of this thing. Yeah, I think that's exactly right. And part of that is being confident in what we sort of said, we're creating a lot more, it doesn't necessarily mean we're delivering more value.

So where does that break? We see, it's in our numbers, we put out some reports, more branch builds, right? People generating more code and using, in our case, CI as the means to check whether it's good without always the same commensurate increase in main builds.

Which means I'm using the branch build as a way to check if the agent did the right thing, and that feels late, right? I shouldn't necessarily be surprised by the time I get to CI that the thing that I-- We see people failing their linters in CI, right? That's something you could definitely check earlier.

And so part of that is getting the agent to know how to do those things, right? Which is something that we've invested a bit in. But I think overall, everyone's trying to do that.

And then part of that is having the right tooling to support that in, again, people that are just listening, I've used a lot of air quotes, right, for local, because local is never just local now. It's CLIs and connections to LLMs and connections to other tools and whatever. But orchestrating all of that through your agent that you might be running locally, you might be running in a cloud agent, but tying it all into that cycle of creation, right?

Getting all the feedback and correction there, because if you're just piling all this stuff up at a human gate down the line, you're still throttled by your human gates, and you haven't really gained anything from all the acceleration we created. How do we get that done exactly? Because it would seem like there would have to be some changes made on the inner loop side of this equation.

So how do they manifest themselves in your case? Yeah. So, the way we think about it is providing CI-like feedback because a lot of people tie CI to a very specific process, and we don't care as much about the process as we do about the value of that process, which is validating that your stuff is good.

So CI-level quality feedback into the agentic loop. So if you look at, I'll use Cloud Code as an example because we use it a lot If you look at how the agent is structured, it has capabilities in the harness of the agent, things like pre-commit hooks and stop hooks, and on certain events, you can ask it to do specific work that is not at the choice of the LLM or the agent, but is baked into the framework. Right?

" In our case, we have a thing called Chunk Sidecars, but call us and we will do a very fast assessment of the quality of that and give feedback directly to the agent. So the human's not even involved in that. The human can trust that the agent has to go through this step, gets the feedback, and says, "Oh, it looks like my implementation is not correct.

" And because those particular cycles are fast, designed to be optimized, designed to be a little cheaper, honestly, they're saving you the downstream effort and work, and they're giving you the feedback a lot earlier. We also find as a side effect that they reduce token consumption because we're giving better direction to the agent earlier to get to the place where it's trying to get to. And then construct, is CI and CD becoming more tightly integrated or more loosely coupled in that motion?

I think, how would I describe that? I think it's I want to say both. On the inner loop side, right?

It's almost like I'm using it in two different form factors, if that makes sense. One, I'm doing these quick light checks, and then two, I should be more confident that by the time I go back to my traditional gatekeeper, there's no way it's getting to production if it's not good model. I already have a higher level of confidence.

So that's still downstream because all of this stuff is going to be merging, right? So the interesting thing about a lot of the tools out there today in terms of the development is it's very single player. I'm over here writing all of this stuff.

You're over there writing all of that stuff. Merge conflicts used to be bad, and now we're really seeing if you produce a lot more and you can make huge swaths of changes across a code base, there's more risk. So how all this stuff merged together downstream becomes-- That's sort of like where the traditional CI piece fits in, but the inner loop piece, I guess is more tightly coupled.

But it's not specifically dependent on any given approach. I think it's just really important to make sure that whatever you're doing, you're getting that feedback early and giving guidance to your agent. I think the thing that really shifts about it, and I don't know how to think about this in terms of coupling, is that it goes from just a binary answer, right?

"Oh, my thing failed, therefore, I'm going to go do some different work," to very specific feedback. " Right? So it's not, I don't know how well this reference lays, it's no longer a game of go fish, just a guessing game of maybe I'll try something else, but very targeted, you should probably look at this thing because this is what's not working.

Is this evolving therefore into maybe a system of agents that we're not only orchestrating and managing, but they are kind of communicating amongst themselves to resolve some issues, and it's going to be more of this whole kind of agentic engineering workflow that we haven't quite figured out exactly how it will manifest, but that's where we're headed? I do think there is space for multiple agents for some definition of agent. I also think some determinism remains.

It's sort of what we cling to in a way as humans. Maybe cling to is wrong, but is it going to give us the confidence, right? One of the things that I think changes is take or leave what your feelings about the quality of human PR review.

I think it was always a little questionable, certainly not even quality across all reviews, across all humans, right? We know that had some variance, some variability in that process. But nevertheless, we would, for example, look to ensure that the tests that have been introduced look like they provide good confidence for the code that was introduced, right?

" But we can look at the tests and say, "This looks like it would identify if the issue I'm looking for is present. " Right? If an agent generates a bunch of tests, often the agent is sort of outcome driven and the outcome is green, right?

Therefore, I write tests that I know will pass as opposed to I have ensured that I have really good coverage of the intent of this change and know that the intent is correct. Right? And so where we're investing on this kind of deterministic side, although interestingly, this is where I think the blend of agents also comes in, is how do we separately assess the code change and assess the quality of the testing changes to ensure that, yes, we feel good about this change flowing through because we feel good about the tests that have been created, right?

And I do think, to your point of just many different agents interacting, any human stepping into this is not operating at the velocity of the agents, right? And so we have to be very particular about where we apply human knowledge and capability and understanding, right? Whether that's flagging, okay, there's a 2,000-line PR.

It sounds ridiculous historically, but now it's like a par for the course every day. These 10 lines look suspect, right? So I want you, Mike, to have a look at this, but just worry about these 10 lines.

We've looked at everything else, the agents, whatever. We feel super confident in all this, but this feels like it might introduce some risk, so we want human review there. Because if someone says to me, "Hey, before we push this out, it might be some risk in here.

" By the time I get to the 10 that are interesting or important, I'm just ... scrolling. I don't know.

This all seems fine. You know what I mean? But if you could tell me, look here and look here, and here's the intent of the change, just make sure this makes sense to you.

Cool. Then I can focus in on that, right? And so I think, yes, we're going to leverage multiple agents.

Yes, we're going to continue to leverage some determinism to feel confidence, and then I think we'll be very targeted in where we try to apply human interaction or human intervention by being able to identify anything that seems particularly risky in the change. Otherwise, it all just, again, backs up against a poor human who's just reading reams of code that no one else has looked at. And so ultimately, it sounds like we're trying to find the balance between probabilistic and deterministic, and where those gaps are is where the humans are going to step in and kind of provide some additional insight and analytics and understanding.

But they don't need to necessarily read the entire code base because, well, they might not even understand the entire code base. Well, yeah. Humans are also non-deterministic, right?

It's so funny that we're so puzzled by non-determinism because it's us, right? Artificial intelligence is modeled after our brains in some ways. Of course, it's not exactly human.

But we should be pretty comfortable with that because the behavior of our peers, right, in an engineering organization has always been non-deterministic, right? We think we're very clear in our communications, and then we see the outcome and we're like, "Huh, I thought we talked about this. Something was lost in translation," whatever that is.

So we should be used to that. And I think that there's kind of two parts to that. Non-determinism is great when harnessed in a system, right?

I can build a toolset or a system or a machine that leverages the power of non-deterministic tools, but ultimately gets me not necessarily a deterministic outcome, but a high-confidence outcome, right? I don't care about how some of the things inside are done, but I care that I have confidence in the ultimate intent in what I'm delivering. And then from a human factor, yes, we'll flag things that seem concerning to humans or should be concerning.

We can't build confidence that this piece is acceptable, so can you come look at it? But I think what we have to do, and the way that we've always done continuous improvement is as humans look at that and say, "How could we improve the machine so that next time we will be able, the machine will be able to assess the risk, and it won't require human intervention," right? That's how we'll get faster and faster at this delivery is say, "Cool, it's good at these bits, but it's not good at these bits.

What are those bits? What's common about them? " And I think the human job as engineers is now building the tooling more than it is building the software, right?

That's where we're going to be investing a lot of our engineering effort. Do you think that we're holding the AI agents to too high a standard compared to humans? To too high of a standard.

It's a great question. I don't know what too high would look like. I do think we bring a level of skepticism that's not necessarily tied to the output, right?

There's a human bias that's like, that can't be good. I'm not saying everyone is doing this, but I see it, right? We can't trust that because it was generated by a machine.

And I think if you did the double-blind review or however you would think about it, right, and just put, for some reason, I'm picturing blind taste tests. But if you put a bunch of AI-generated code next to a bunch of human-generated code in front of some other experts, what would the difference be? I don't really know.

And in some cases, I think, over time, I believe we will see emergent properties of AI-generated code that doesn't suit our taste as humans, but is optimized for agents to then go make the next change, and that's what actually ends up mattering, right? Because everything we've done in software, I have the strongest opinions in the world that I've now tossed in the bin about what good software looks like, because I care about software that a human can evolve, right? So the thing that matters is the next change, not this change.

Can I structure the system in a way that it's flexible as our business changes and our needs change and the market changes? And what I desire in terms of being able to make the next change or believing that humans can make the next change is a little different, I think, and I don't think we know exactly what that is from what agents need. They leave a trail of documentation that no human would leave, and not because we're lazy, but because we don't find it particularly valuable or insightful.

It's often out of date relative to the actual implementation, et cetera. But it makes the next agent work way faster, right? So I think we have, I don't know if it's necessarily the bar, but there's divergent paths in terms of what is good, quote-unquote, or high quality, where the measure of quality is our ability to safely make the next change and continue to deliver at high velocity or even improve our velocity.

And that, I think we're still discovering. A lot of folks I talk to are obsessing about their pipelines all of a sudden. And they're trying to figure out, well, do I need more pipelines?

Do I need separate pipelines for AI agents, or do I just need to clean up my pipelines and make them more efficient, and I don't need more pipelines? Yeah. Generating more code produces a lot of downstream effects.

And absolutely, we're seeing the growth. We can see it directly in our numbers and consumption on our platform as folks build and deliver more software. And for the most part, we're excited about people's ability to do that.

We want to help them, right? I think- One of the things that we hear a lot is that AI is an amplifier. And I would say another way that I think about that is that it's like seeing your process on time-lapse footage, right?

Things that took a long time to notice now happen really, really quickly. And so the cost, the overhead, the brokenness, the manual steps, they're all under the microscope because it's really easy to see them and reason about them on these really short timescales. And so people are noticing, hey, maybe two hours to run a build or pipeline or whatever your terminology of the day is, is not good enough, right?

And you have some basic queuing theory problems in there, which is just like, my arrival rate of change is much higher, and now I'm backing up instead of clearing it eventually, right? If you have a long enough pipeline, or if you don't have enough parallelism or capacity in your pipeline, asterisk simulator, whatever. But if you don't have the ability to get that stuff through and you're waiting and then it-- I was talking to someone the other day who was saying, their builds clear overnight before everyone comes in the next day and gets back to work.

You stack up enough of them, they won't clear overnight, right? So yes, you can spend more money and get more parallelism, all those things, but then your bills are going up, and so people are saying, "Now it's become a real thing. " And if I can't get feedback, people are plugging the pipeline directly into their agent, sort of as we were talking about before.

If I have to wait hours to get that feedback, what's my agent doing? It doesn't feel like I'm getting any of that acceleration, right? So I don't think people necessarily need anything super different.

They might-- Well, I'll take that back in a second. They want to be very good at optimizing and making sure their stuff runs really effectively and it's reliable, so flaky tests feel much more expensive because it might be a percentage, but the full actual number, not the percentage, is going up because the total number is going up, and therefore it's just feeling more and more expensive now. And then I would say the idea that I would run a pipeline, have a human review as a PR, run a pipeline, but then it's cool when it goes into production.

We're going to sit here and look at it, and we're going to go click some buttons and make sure it actually does what we want it, because we don't feel super confident in this part of the testing, and we forgot to add observability or tracing on this part, right? You don't have time for that anymore. And so I think people are getting much more locked in on, I need really high confidence that when it goes out, either it's good or I'm going to know really quickly that it's not.

I've managed the blast radius. So, we hear a lot more about observability, about feature flags, quality of the pipeline in the first place. All of those things are, again, when you amplify it, the downside risk is getting higher, right?

Like if my oh, what is the name of the DORA metric? Change failure rate, right? And measured as a percentage, if you took even an elite quartile change failure rate, I can't name them off the top of my head, a little embarrassing, but if you took that and said, "That's fine.

" And then you 10X or 100X your change rate, right? Then you 10X or 100X the raw number of incidents that you're causing for your customers, and that elite change failure rate is now a disastrous change. So, the percentage hasn't moved, but again, if I push 100 times more changes, then the raw count of issues that I've created for my customers has gone up 100X, and now people are sad, right?

And so, I think that all of this tooling is under the microscope, and people are, now the things that you just papered over and said, "Oh, yeah, we do that part manually, it's fine," is just not acceptable anymore. Or it's putting you so far behind your competitors and behind the pack that you got to fix it. Not too long ago, everybody was focused on, quote unquote, token maxing.

We were trying to get people to use AI. So, some people even had leaderboards for who consumed the most amount of tokens. Have we come full circle on that now?

Because everywhere I turn to now, people are focused on tokenomics, and they're starting to realize that the cost of this stuff is real. Yeah. I feel like it lasted about a week.

It probably depends. How long your token maxing sort of depends, one, on the culture of your company, whether you put up that leaderboard and sort of held it up in the first place. And two, on how much spare cash you had on hand.

Like the jokes about people at Meta, or take the foundation shops that still don't even care because it's baked into their cog somewhere. They don't actually think about the numbers. Fine, right?

Get after it. But a well-operated, smaller, profitable organization that's just trying to grow but works against a well-structured budget, all of a sudden was like, "Wait a second. There is no cost control on this.

How did this happen," right? And a lot of the tools, honestly, are so new. It's one of the side effects, I guess, of growing so quickly, like when we talk about the foundation model providers, et cetera, that their tooling is pretty meager when it comes to account management and cost control and everything else.

So yes, we're seeing that very quickly and people make the references to early days of cloud and whatever. It was not hard to be like, "Oh, look, I can Click this button and get another machine. How convenient for me, right?

" Right? And the challenge in organizations there, which is the same challenge we're seeing now, is that we put cost management into the hands of individuals who generally have no insight. They're not accountable for the cost.

Ultimately, someone's going to come yell at them, but they have no real insight. They have no visibility into what they're spending or what they're using. " And so you have this 30-day window.

Pick the simplest example. I remember this happening, this is probably, I'm dating myself here, but in the early days of mobile telephony, people would fly to another country and bring their phone along with them and call all their friends and tell them how cool it was, and then get the bill when they got home and go, "Wait a second. " And so you don't know until there's this really lagging indicator.

And so people are investing absolutely in managing the tooling. There's tools, capabilities within the sort of core products that we all know. There's also a lot of work going into how do I optimize the tokens that I even send to the model, right?

A lot of times people are just sending enormous amounts of context that isn't even really helping. They don't know what's in their context window. It's too easy to use without that visibility.

And so now I don't think we'll see a massive clampdown, right? We will see educational discussions like, "Hey, it doesn't look like you're really using this tool particularly efficiently. " Those sorts of things.

But there's a huge group of people who are just doing a great job at more effectively building the product that matters to their business. And for those folks, we want them to keep going. We want them to accelerate.

That's our perspective. But we want to provide tooling, like we're providing tooling internally for ourselves to minimize the waste, to alert people if they are on a bad trend, those sorts of things that are pretty basic and you would expect. But I think when this first started, kind of to your original point, there was a lot of skepticism and a lot of slow adoption, and organizations felt like they were getting outpaced by those who were being successful.

And so the only thing that mattered was adoption. And we went through the window where we want to know how many lines of code you created with AI this week. I'm like, no one ever cared.

In fact, people were really sad when I created a lot of lines of code manually. So why are we pushing for... And then it's like, forget token maxing now, I'm like line maxing?

What's even in those lines of code other than a bunch of crap that nobody needs, right? So we always do this to ourselves. We get a little off on the metrics, and we start measuring something that it turns out is going to create unintended consequences.

It's a tale as old as time. But I do think the tools are arriving. They didn't exist because this grew so quickly, right?

I would say many platforms still, you have two or three accounts with the same provider because they haven't even managed to stitch the pieces together. We'll get there. It'll get solved pretty quickly, but yeah, the token maxing was a...

As soon as someone created that word, we knew we were all in trouble, right? Well, folks, you heard it here. Hey, DevOps, software engineering, it is changing in the age of AI, but oddly enough, the more things change, the more they stay the same.

Hey, Rob, thanks for being on the show. Thanks so much for having me, Mike. All right.

And back to you guys in the studio.