The Future of AI in Coding with Codeium’s Varun Mohan

This is Textron tv. Hey guys, thanks for the throw. We are here with Varun Mohan, who's CEO for podium, and we're talking about, well, just what will be the impact AI will have on application development because, well, it starts with the coding, but there's all this stuff called DevOps, and it's all one big software development lifecycle.

Varun, welcome to show. Hey Michael, thanks for having me. Let's start with the people, because you know, everybody's kind of walking around going, well, is my cheese moving here or not?

No one's quite sure, but what is your sense of what is the impact AI will have on developers? Are we gonna need more of them, less of them, the same amount, just augmented, what's, where are we on this adventure? Probably the way I would look at this, or the way we think about it is it's kind of similar to higher and higher programming languages existing, right?

In the beginning we had assembly, then after that people started writing CC plus plus Java, and then Python in the early two thousands. And every time a new programming language comes out, that makes it simpler to write more software. There are just more developers.

And I think the fundamental reason for that is software is technology creation, and there is, there's no limit to how much technology and how much value can be created from software. So the more leverage you can actually provide developers, I actually think the end result is we will have significantly more developers in the world providing significantly more value. What will be the level of expertise required then?

Because a lot of those developers are not gonna be well schooled in to your point c programming languages. So, um, do I just need to be able to kind of think logically and I can become a developer with the help of ai? Or will professional developers just build more software faster?

Yeah, I think it's gonna be a little bit of both. I think you're still gonna have people that go into the weeds actually understand, you know, multi hundred programming, all these other sort of low level details that people, frankly speaking, most Python developers don't think about from a day-to-day basis. Right?

Uh, how, how do you write thread safe programs and all these other things? Um, but I think what is more going to be the case is there are going to be more people that are now considered more tech adjacent, that are, feel more empowered to build software, right? And the fundamental reason is because AI makes it so that the, the level of abstraction that the developer has, potentially people will write at a higher level programming language.

Um, once again. Um, so I think we will, without sandbagging on your question, I think you'll still have people that can get into the weeds write CC plus plus. I don't think that software is disappearing.

If you look at most large enterprises, a lot of companies even have COBOL to this day. Uh, right. So I don't think existing programming languages will completely disappear, but I think in the future you're gonna have people writing significantly higher programming languages and that think about more of the problem, right?

Rather than the exact implementation details, which I think is what most developers like thinking about in the first place anyways. How will these workflows evolve? I mean, in my mind I'm kind of starting to think about it in terms of there may be a human developer that has a bunch of AI helpers around it that are optimized for specific tasks, and I'm kind of supervising and orchestrating the workflow across these things.

Uh, I mean, how do you see it evolving? I actually, I actually think that that is one, one way it could pan out, but I actually think it's, it, it would, I suspect that the way it's gonna pan out is maybe a little bit differently. So what you're proposing, and I think what a lot of people are are sort of seeing is you have a bunch of these agents that asynchronously operate and kind of do tasks for you, right?

And you're able to kind of go out and maybe split up the work that you have to do into 10 pieces and they all just go out and, and, and do 10 different things. Um, I think the, the thing of that about that is I don't think that's a very rich idea in that Upwork already exists in some ways. Like you could already kind of go out and pay someone, uh, to, to go and do a, a task for you.

What I think is the more rich idea is I want to be able to get a single task done 10 times faster. And I think that's what's going to be feasible with ai. I think what's gonna be capable is you're able to solve a very hard problem and AI is able to help you navigate the code, debug the code, write the code, test the code, review the code, and deploy the code significantly faster.

And that's where true leverage comes about. Leverage doesn't happen by making it so that you can do many things asynchronously fundamentally, because if you're a business, right, and you want to get a task done, what matters most is how quickly you can get the task done, right? Like let's say, I think the idea is going to be AI is gonna enable a project that previously took a year to get done, the critical path of that to actually take three months.

Um, so it's a little bit of a different framing, if that makes sense. Mm-Hmm. How will our backend DevOps workflows evolve?

Because as you described it, there should be a lot more code and projects moving in parallel through our pipelines that are already kind of stretched thin. So what's your sense of how might AI be applied to kinda facilitate the whole deployment process? I think you bring up like a really good point, which is everyone's saying that code is gonna get, get generated 10 times faster, but we don't have a world in which we can even consume code 10 times faster, right?

Like, think about if you had 10 times the number of applications, you're probably gonna have 10 times the number of bugs, all else being equal, you're probably also going to need to review 10 times code, 10 times faster. So I think what is actually important is the entire software development lifecycle needs to be targeted. There needs to be a way to somehow optimize every stage of the way in which the development process works.

That actually even comes down to what you said. If you're a backend developer, there need to be tools that make it significantly easier to deploy new apps. Otherwise, if I'm given 10 new apps, when previously I was given one, I'm going to have trouble deploying all 10 of them, right?

Uh, and I think the way this is going to work is AI needs to deeply understand everything that is running inside your organization. Do you think the business will be able to absorb that amount of software? I mean, for years we always heard that the business complained that the IT folks were too slow.

But I can't help but wonder if we're soon gonna have the exact opposite. I think if you look at most large companies and, and we work with some of the largest, I guess, uh, sort of public companies right now, uh, if you were to talk to their CIOs, they feel software starved. There's so many projects that they want to be able to accomplish that they're just not accomplishing.

And that's not because of laziness of existing developers that it's fundamentally very hard, right? Every time you have an existing system, there's some amount of tech debt, some amount of maintenance costs attached to that, right? If it was written in a pre, in a, in a old version, you actually spend a disproportionate amount of time migrating software from one version to the next.

And that's just the cost of doing business, right? If you don't migrate your software to the next version, it can be extended. So I think these are the kinds of things that ai, these undifferentiated pieces of work that I think AI is going to help accelerate tremendously, right?

And I think that will enable companies to, to feel like they can take more scary technical bets, right? Which is that I'm going to build a much more complex app and deploy it. Well, what is your sense?

Will we rip and replace a lot of our legacy apps with something that is AI based? Or are we gonna go back in and modernize the legacy apps with some sort of AI capability? So I think, you know, maybe to give a concrete example, I think the IRS tried to migrate, uh, a very large code base.

They spent many billions of dollars of cobalt to dava, um, and they actually failed. They were not able to do this. And that's fundamentally because languages like COBOL are actually hard to wholesale translate to, to Java.

You can't just take existing COBOL functions and just convert them into Java functions, right? And I think what would instead be prudent is taking the pieces that need to be accelerated, converting those pieces to Java, and then using meth mechanisms like j and i, right ways of basically piling different pieces of the system to languages that you want to extend rather than wholesale replacing everything. I think people are going to take a pragmatic approach of maybe replacing smaller and smaller pieces rather than going and taking the whole thing and replacing it because the, the cost of doing that is, is tremendous.

Do you think that the pace at which we build apps will also become more iterative? And I'm asking this question because historically it's always been, here's my requirements, document developer goes off, creates this thing, comes back that people who created the requirements document changed their mind or didn't describe something and then they went back and it took forever. Can we get to the point where maybe this whole process becomes a lot more iterative where we're building code and generating apps, you know, within hours and days of when we launch the project Project?

I think the fundamental limitation there is not actually the, the ai, it's that the hardest part about building technology is actually understanding the requirements, making sure that you build it in such a way where it makes your business or the company or even the project long term successful, rather than taking a very short term look at it. And I think that takes discussion with other engineers, right? Takes discussion with other engineers, understanding trade-offs.

Um, and I think that's actually the very complex part that I don't see disappearing from the entire software development lifecycle. I do agree with what you're saying of the time from first understanding what the requirements are to then building a a first version of an app is going to reduce tremendously. And I think it's going to be because these products are going to provide so much leverage, right?

As I said, it's not gonna be like a bunch of async agents going out and doing something. It'll truly be that you are able to orchestrate a single, a single process that goes out and does something very complex. Will we continue to have the same level of lack of a better phrase, uh, attachment to our applications?

Or is it gonna become easier to update and replace them that, I don't wanna say they'll be disposable, but they'll be less, um, difficult to replace or upgrade? Yeah, I think, I think right now, um, what AI will enable us to do is probably be less afraid of replacing an application, right? There's so many cases where companies decide to extend an existing system when it probably should have a rewrite.

And the fundamental reason is because when it was created in the first place, that requirement was not actually a requirement that the company even knew of. But I do think in the same vein that AI is going to make it easier to write software, it'll also make it significantly easier to build more complex pieces of software. And I think all that happens is the complexity just scales, right?

So in a given year, if you build an application, the application that you built over an year is not as simple, is not as simple as the app you could have built in your pre ai. So I think the cost to rebuild a system is not gonna be instantaneous in that regard. It'll still, it'll still take some time to rebuild a system simply because systems are also more complex because AI enables you to build more complex systems.

If you were just starting out, let's say you're in high school, would you still be thinking about entering software engineering as a profession? Or do you think that this is gonna be more of a capability that everybody has because I'm invoking it through ai? I think, I think my perspective on this is still software is actually one of the fields that I would, I would be the most excited about joining, right?

Uh, so I think when I think about ai, AI is an accelerant to probably generating value. You know, I would be worried about probably being a, a farmer in some capacity if AI was going to touch farming, because effectively there's a limit to how much food people can consume, but there's no limit to the amount of technology that the world can consume. Um, and I think the way I would sort of be thinking about it is that companies are still going to exist, right?

I think businesses still need to exist that provide value. And I think, I don't actually believe in a world in which there will be single singular people building single person companies that are built using ai because I think that effectively means that the companies are not accurate, are not using AI properly also. And I think companies are smart, given new trends, they actually take them on.

If you actually look at the generative AI trend, actually large companies are, are a hundred percent using generative AI products. They're not actually sitting in the sidelines and being like, this is, this is technology that's too new and it's not gonna provide value. And then on the other side of this coin, it seems like to me that a lot of organizations are still struggling to figure out how to operationalize ai.

They don't have the data in place. So what is your sense between, you know, where we are today and when this AI nirvana actually starts to emerge or we see it? 'cause it feels like there's a gap.

There's a gap. So I think there's two gaps right now. So gap number one is what you said.

These models are not actually with retrieval, they're not retrieving everything of the internal state inside a company. Think about the number of places where data is actually stored. It's not just the code base, right?

It's stored in Jira, it's stored in Confluence, it's stored in documentation, it's stored in the actual deployment environment that is not committed to code base, right? You can open up a Kubernetes cluster and there's a bunch of uh, uh, there's a bunch of useful information there in database schemas that might not even be in the code base, right? And I think because of that, actually it is, it is infeasible for the models today to provide the correct answer a hundred percent of the time because they just don't know all the pieces of information necessary, right?

And where everything is, but that feels like it could be solvable with time. The second piece, second missing piece is these models aren't actually tremendously smart right now, right? If you look at a model like GBT four is, it is probably scoring and the competitive programming competitions in the bottom fifth percentile of competitive programmers.

And I think what the implication there then is these models are imperfect. And I think actually it is very cool to see that in a space where the models are imperfect, they're still providing a tremendous amount of value. And I think the, the, the level of value or the level of reasoning that the models have with time is only going to increase the ability to understand and also grok all the data inside the company is only gonna increase.

So I think the value that these systems are gonna provide is gonna increase. And fundamentally, there's a limitation to how much value that these systems are able to provide today for the two reasons that I just said. Do you think people fully appreciate the fact that a lot of these models are probabilistic and I think some of the use cases that people wanna apply them to are deterministic where it's gotta be right a hundred percent of the time.

So, um, do we need to get smarter about when and where to apply this stuff? So two things, right? Um, so right now, if I were to bring our product in code, um, we provide auto complete chat and context awareness.

And by the way, when we provide auto complete capabilities, podium writes over 45% of the software that a developer commits into the code base. But that's being said, whenever we generate suggestions, you developers don't accept the suggestions. Like 90% of the time they're not accepting suggestions a lot of the time.

But what happens is developers have learned to understand, hey, here are the key operating domains where the suggestions are going to be good and can quickly validate that a suggestion makes sense, right? And the suggestion could have encapsulated like minutes of work for every suggestion that they actually end up accepting, right? So people are working around the fact that it is probabilistic, it is non deterministic.

And the same is true for chat. We apply ideas like context awareness, so we can actually use all the data within a code base to inform the results of chat. The end result of now developers onboard onto code bases with code from three to six months.

It now takes three to six weeks, but still the system hallucinates sometimes. And it's kind of like how in the end of 2022, Chad, GBT came out, people noticed that the system hallucinates, but despite that they were getting a tremendous amount of value from it. Now to your set, to your point of are there cases where we do want determinism?

And I think we're already seeing the early innings of how you can make semi deterministic systems with large language models and the ways they're pairing a deterministic system with a large language model. And I'll give you an example here right now when developers write software, they're CICD pipelines and go out and look at the software, find if there are security vulnerabilities, and Ls can, can do a great job of taking the security vulnerabilities that these deterministic systems find and providing remediations and these remediations are with high probability good remediations simply because the problem has already been root cause at that point. So I think the way we will think about things is the models will be more capable, which means that we will trust them in more complex environments.

But there's a fundamental limitation right now, right? That you will not trust a model and just automatically deploy code from a model without reviewing it. Just imagine what happens if it just deletes a hard, hard drive, right?

The, the implications of that are very negative right now, and you're totally right that that is gonna restrict the set of applications that these systems are capable of being used for completely autonomously. All right folks, you heard it here. It doesn't sound like humans are going away anytime soon, but there's definitely gonna be, uh, a need for a verification process as stays easier to build and we all start to move faster in general.

Hey Varun, thanks for being on the show. Thanks a lot. All right, and back to you guys in the studio.