Observe President Qi Jin on How AI and the “Vibe Loop” Are Transforming Software Engineering

Hey guys, thanks for the throw. We're here with Chi Jin, who's president of engineering and product for Observe, and we're talking about how software development is changing in the AI era. You welcome the She.

Thank you. Thank you for having me. I think we all agree that it's changing, but I feel like it's kind of like a traffic accident in the sense that no witnesses seem to be seeing the same thing.

And I get a call entirely different reports as to what's happening. And it will range everywhere from developers are saying It's nice. I can have something to explain something to me, and it writes some code, but I still gotta figure out how to vet it to CEOs who think that, you know, the entire economics of software development is fundamentally changing in ways that will allow them to cut the SAP by, I don't know, 80%, depending on who you talk to.

Somewhere between those two extremes is probably some sense of reality. But what's going on here, my friend? What are you seeing?

Uh, well, uh, thank you, Mike. I, I think, I mean, uh, am day, you know, if you look at Avidity, right? Uh, let's say you are the head of engineering.

I, I think with all the technologies your basic needs doesn't change, right? So what, what is the basic needs in the engineering organization? Um, or, you know, a leader of the, uh, engineering organization, you, you kind of look at it saying, okay, one, your software needs to solve customer, uh, issues and then needs thing need to be reliable, stay up.

So sort of really deliver a customer outcome. So that's sort of a more like reli based, right? And then the second thing is just like you, typically the productivity, like your engineer typically spends 30% time, uh, on calls and charge incidents or all these things.

Uh, so that's sort of more productivity. The first thing is, you know, you're gonna running, uh, you know, C-P-U-A-W-S and God forbid, and GPUs. Now, uh, that's typically took, um, you know, in a, in typical SaaS, took somewhere between 15 to 30% your total bill.

So that's like how my infrastructure should be efficient. Lastly, it would be like, uh, if you can do all three, well, you'll be like, okay, well, is there any other thing I can look at it, my customer base, how they use my product? It's more like a product behavior, customer analysis.

So with all these where observability, uh, need solve, uh, didn't change before and after ai. Now, uh, if, if we just put that aside to say, uh, with the ai, what changes? So, uh, I, I think if you look at then the, in the space, who is the player, right?

One is, uh, the engineers in the engineer side. I think the coding, uh, the creation side, you now have a strong assistant and take out all the body code, and they can write a code much faster. It can debugging what, uh, these, all this cogen to is great.

The only, the thing that is missing from a cogent tool is it, it it only goes to dev environment as soon as you go to production, none of the cursor, you know, winder and, and can actually tell you what's going on in the production. So that's, I think that's gonna happen next is the, the cogent tool. What takes you all the way through to debug in the project production incidents, reliability, all this.

Uh, so that's sort of from the, the, the, the coding tools. The other side, which is, uh, the availability tool, ability tool lets you essentially solve this problem. I efficient, can I reduce instance, can I reduce toil?

The problem is there, rich is only always on the production. They cannot go to the dev side, say, Hey, listen, you know, you are, um, you write a wrong code on this way. You could do, you could do that way.

I think they were actually going to the cogent space to say, okay, if I had this log problem, how can I, um, uh, how can I, um, you know, uh, doing better code to, to do production. I think eventually this, the coding side and the observability side is gonna, uh, integrate and become one process in my view, uh, which is sort of end-to-end developer focused flow, uh, we call a vibe loop, which is from sort of a, uh, coding all the way to production as one true versus a separate markets. So, Hmm.

And how will we bridge that divide? Because today you'll hear people talk about the coding tool that they used, created a bunch of code, but it wasn't really aware of the runtime environment, so the code didn't really run. Um, so how do I get to the point now where that ei coding tool was smart enough to create something that is optimized for my production environment, which is often in a snowflake in its own right?

Yeah, yeah, totally. Um, that's a, that's a, uh, you know, I, I give you a couple example, right? What, what happened in the production oftenly, you have an incident, you come in, you look at metrics, uh, the metrics may all look screen, but you know, the customer didn't run.

You are like, Hey, I wish I instrument the customer feels so I can see which customer have a problem. Uh, so, uh, so how do you do that? Now you have to go find a code who actually, uh, emitted the metrics at that customer field, and then when you add it, you take another site to show off.

And so that's actually a very laboratory process. Uh, so that's number one. The, the second thing is, in the past people say, oh, I, I, I will, I should be able to add this instrumentation.

But the problem is the volume's so high that they couldn't afford to do that, right? Um, so I think to solve that problem, uh, my opinion is, you know, every developer want the signal, uh, on the log metric and traces, but none of them actually want to write it and put into the code. This is the last thing you want to do is adding log statement.

I think, uh, uh, giving the, the advancement of the tool, uh, you know, the, the code generation tool, understand your context. They could add a lot of the instrumentation automatically either using auto instrumentation or using understanding code logic. So you're adding these things, it's almost like how you generate business insight.

And that thing with the continued deployment will flow to the production, and you can see this thing very quickly. So that's, that's a, that's a big thing, uh, to do, which is let, uh, let the more insight generated from the code generation to the wrong time. The riverside is very interesting.

If you look at all the code generation, they have a context. They'll say what application you want to generate. You're like, Hey, you know, I, I wanna generate a website, you know, I wanna, I wanna do this and that.

The thing is, when you run in the wrong time, it actually had a lot of insight too, like, what's the services, which cluster you are running? What's traffic? If these contexts can send to the coding tool, it can say, oh, you know, you run a lot of limits.

I I actually need to increase your resources. I actually need to put putting a protection. So the other side is the, the, all the context from the wrong time become part of the knowledge, but you can ingest in the coding time, which actually will make your quality much better.

Hopefully, I, I expand it. Yeah. Will that ultimately lead to more efficient code?

'cause I think one of the dirty secrets of it is that a lot of the code currently right, is inefficient and we wind up spending a lot more money on infrastructure to run it. However, also, people will say that the code that the AI coding tools are currently generating is even messier. So totally, how do we kinda, you know, close that loop.

Uh, I really, uh, like how you, how you frame it, right? You know, when you close the loop, the first loop you close is based on the, uh, instrumentation best practices. You do wanna send enough right signal to the wrong time.

So you sort essentially recall your, uh, I, I'll call that, uh, business intelligence on your code, right? So when you write this thing down, uh, couple thing would happen. One is, uh, how do you deal with, with basically incidents and errors?

So to close the loop, uh, in the past what happened is you, you have a arrow, you have a log, you have traces, you, the people would take a copy of that thing, you know, copy that log, try to go search in your code base to search these things. Uh, nowadays to close the loop, you can have MCP server, uh, essentially, you know, m mc server is just a protocol for, for AI to understand each other, right? So, uh, mc server of your availability system, and then the, the coding tool can directly create that.

Uh, for example, ask very simple question, you know, which server has the most logs, arrows? And then you are like, oh, I have a 4 0 4, you know, in my case, right? And it will say, I switched the 4 0 4, there are three places in your code have this thing.

And by the way, the the last one was someone changing it, do you wanna roll it back? And you, and so, so that's actually one sets which is reduce, uh, reliability. The other thing, uh, just, uh, on efficiency, I feel like you really spot on.

Then people said, listen, you could actually write a very efficient code. The issue is you don't want like a micro optimize, because every line of code, you can optimize the thing, but the the point is like, you wanna optimize the thing, get caught a million times, not a one time. So how do you know a code get caught a million times?

Well, you only know that in the wrong time. So if you look at the wrong time and say, oh, you know, this thing got caught a million times, this take a huge latency. They take a lot of efficient, uh, uh, issues.

You can take this signal and go back to the code and ask a cogen saying, Hey, listen, the memory usage is not not good. I mean, can you think about shutting it? Can you think or reduce it and whatnot?

Uh, I think then you can write it efficiently, code, uh, uh, you know, get going. Uh, ultimately the efficiency, the quality come from the creation side. You know, I think visibility at the end of the day are still like, if you can impact on the most left side of the process, which is creation side, it will be the most efficient way to do it.

We also somehow magically expect that we're gonna be building more applications than ever. But when I look at the DevOps workflows that are out there, they're brittle. And I don't think I can keep just throwing more pipelines at that.

So how will I manage all this additional code and application development projects that I assume are gonna be happening more in parallel, but how do I kind of get that to be workable in a way that doesn't overwhelm me or just break everything as I have it? Yeah, yeah. Uh, love it.

Obviously, everyone have a, a lot of different perspective. How do you do, uh, the best practices, right? And I, uh, if you mind, I, you know, I'll give you one example.

So, uh, so I walked in the Google search, right? And Google infrastructure for eight years. And I was actually always surprised, at least inside Google, um, actually the dashboard and, and the metrics wouldn't really a big thing, but we, we have some stuff, but you, you would, you would imagine if you do search, you would have hundreds of thousand dashboard, a hundred thousand pipeline.

The truth is not, we actually, I have several dashboard and it's all I could, and I'm kind of, uh, reflecting on is like, why? So there's a couple things. The first thing is obviously being a single company, uh, they, they have the best infrastructure.

There's a best practice. It was enforced. So in that way, you, you do have lots and lots of code, but, uh, in our case, the the monitor dashboard, the, the metrics, the middleware, uh, the, the, the pipeline, they were actually all standardized.

So in some way, you, if you standardize and if you really, really hone in on that standardization, then it will be better. So, uh, in country to everyone was thinking, you know, look, we have a lot of, uh, cogen tools is not gonna be mess. I do think, um, there's a chance for this cogen tool to take the best practices and take the sort of cookie cutter standard way.

And, and then in that way, uh, you actually have less profit. You might have a more and more volume applications, but because they all like does this thing very, very standard way, it actually, uh, will be, uh, more efficient. And so that, that will be sort of a, and I have an example, right?

Um, let me give you one example. I used to do knowledge graph. It runs like a 50% Google search.

When we did, uh, its backend server, we used a, uh, internal Google tool called Block It. It basically generate all the dashboard and, uh, you know, it generate the debugging flow, it generate deployment flow. Um, and none of our guy actually need to worry about this thing.

It's all standard. And I log in, I got all that things, I only need to do a few customization. So that the, the thought I had is, you know, if you can get that standardization into, uh, the, the, the vibe coding tool or into the visibility, then you actually had a better way to solve this problem.

The hope of the, and I see the same works, like even in the scale, uh, the key thing, standardization, uh, and, uh, and the best practice in the, in the get-go in, in the code, not actually to say, oh, I already a mess. How do I curate it? Right?

Mm-hmm. So, And does that lead to better observability and monitoring because, um, able to identify things that are deviating from standard AK anomalies and those are the things that are worth investigating? Yeah.

Yeah. You, you're, you're spot on. You know, for example, you know, we have, uh, we have, uh, these standard called hotel, right?

We do metrics, logs, traces. And if you look at some of the issue with Tel is they have a thing called a cementing convention, which basically gives the fields the different, uh, you know, what the database called what the request really called. Uh, it's good, everyone understands good, but, but the thing is, if, if you wanna every developer to follow that thing, there's a hundred thousand name to track and everyone create different names.

Now, the issue is, if you call this a, you call this b in, in, in, in, in the runtime side, you don't even know which is which. But if you standardize in, it's almost like observability is built in. You're not even aware.

All you're gonna do is you run a code interpretation is already building. Uh, you go there, you just ask four questions like, what do I have? Um, uh, are they running?

Uh, uh, if anything fail, can you tell me why? And then can you fix it? And, and then developer should spend more their time on the creation, not on these sets of things.

And then the best way to do that is try to make the whole thing standardized, you know, generated. And, and then with AI sort of manage that, uh, versus sort of a, uh, you know, training more and more people try to do, uh, observability. So What does the future of the software engineering team look like?

And I'm asking the question because will each member have their own small army of AI agents, or will there be kinda one AI agent that is trained to handle a task on behalf of everybody, or some combination thereof In my way? You know, I, I think, I think ai, uh, in some way, we already had, all of us have a small server army working for us, right? We, you know, we will do charge GBT, which is a question answer bot or do Gong, which is a sales bot, right?

We'll do, um, you know, in the, in the, in the document search or another bot, I think what happens in the coding space, uh, you will have some coding agent really good at the, at the, at the UI there, there's some coding agent, really good at understand your specific domain, like a payments, you know, you, you have some coding, really, you look at the debugging, um, you would just leverage them, you know, sort of calling them to, to your, uh, to on your call. And that's actually not that different from, uh, if you, if you, if you step back, look at what's going on today, right? So today, every one of us would run hundreds of thousands of machines on the AWS.

Now that's basically, we summon all the machines that with different jobs, and they're coordinated all this stuff, compared to 10 years ago, every one of us have a desktop. We just run it, and then we just, we just gonna scale to these things. I think it's gonna happen that way too.

Like today, we only running one code agent. I think in the future we just gonna have lots, lots of power. Everyone will still collaborate with other, other developer.

We could do a hundred, a thousand tons of productivity. Um, I think, uh, I think that's how you would work with, with coding agent. Obviously you still with your coworkers, we all have a different domain expertise.

That's fine. But that, that's how I see it. Hmm.

Every time there's a new innovation, we seem to see the same patterns. We use it to do what we're currently doing, slightly faster, but you can't help but wonder if maybe we should just reinvent what it is we're doing in the first place. So as you kind of think about software engineering in the future, what's it gonna look like?

Um, I totally agree with you. I mean, uh, if you, uh, this happened with every, uh, every innovation, right? When, you know, I'm just thinking out when mobile happened, people said, it's a smaller desktop, let me carry around and let me click.

Right? And towards then it's become news feeds. It's become Uber.

And, and I think, uh, it's a pattern that everyone, when a new technology come, everyone is gonna sync the airflow and fit to it. Um, I, I, I don't have a match wand, but I would say for, uh, I think, I think software engineer will, um, in some ways a liberation. 'cause if you think about it, uh, software engineering, uh, today, it's much more like a manufacturing process.

Means you spend most time coding, you spend most of time fixing and debugging, uh, you don't really spend that much time with creation. You, you do have a little bit creation. I think, I think eventually because the manufacturing process become so fast, software engineers almost become like a creator.

Uh, like if you think about how creator created videos, if you think about people who manufacturing and produce shoes or whatnot, they, their main thing now is the design and not fit together. The taste. I think the software engineering will be like that.

They can wielding the AI coding much better, but majority, however, they're not on, on the TBI typing forever. They're just gonna waiting the, the agent to, to, to perform doing this thing they wanted to do. So that, I think software engineering was much more close to a creation process versus sort of a manufacturing process like today.

Ultimately, then, what's your best advice for your fellow software engineers who are looking at these issues and trying to sort it out themselves? Because I think everybody's having the same thought process. Maybe not at the exact same time, but eventually they're coming around to the same issues.

Exactly. Um, okay, so, you know, I'm part of software engineering too. I've been in the block vault for quite a long time, right?

So, you know, initially it was, uh, you know, on-prem software, then there's internet, right? And there's, uh, uh, then there's sort of a cloud and there's a mobile, right? There's AI in every, uh, aspect.

I, I sort of went through this process of like, okay, that wouldn't work, right? And then, and if you, if you, if you try to look at it from your perspective, right? You, you, you sitting at the place today, you look at the, you're always like, that thing wouldn't work.

And my recommendation is like, you, you first change your proposition is just say, if you really believe AI will work, you say, well, the AI will work. How does that work? I don't really know.

How would you, how would you then make sure you're part of AI with you should be staying in it? Like I would, uh, I would refresh the technology, I will use ai, I would get into the company with more close that space, and then, and think about like, I want AI work. Now, once that happen, I think you will be part of that evolution.

You'll learn all the things along the wave versus to say, okay, is there safe spots that AI wouldn't touch it? I, I think it's very hard every time when you see this technology innovation, uh, it will start with small, it will start with boutique use cases, but eventually it will be prevalent. And you're gonna know these things, you know, it's gonna change, uh, everyone's life, right?

It's like a cloud data, like a mobile data, like internet data. I think just sort of embrace it and try to say, you know, I'm part of the, you know, AI movement, how do I learn, how do I adapt? Uh, and then you'll find, you know, the, the, the, the path is even Brighter.

So, all right. Well, folks, you heard it here. Even in the age of ai, there's still no substitute for hands-on experience.

Hey, gee, welcome. Thanks for being on the show. Thank you very much.

Cheers. Point and back to you guys. And Steve.