GenAI in Development and DevOps with Amplitude’s Shadi Rostami

This is Textron tv. Hey guys, thanks for the throw. We're here with Shady Ami, who, senior Vice President for engineering at Amplitude.

And we're talking about, well, the state of AI tools that are used in software engineering. Shady, welcome to the show. Hello, Mike.

It's very nice to meet you today. I'm excited to be here. You have been dabbling with these tools, as is all many folks in this space.

What's your sense of the maturity of them right now? Because there's a lot of promises being made, but, um, some folks are saying, well, it's nice to have a tool that can write some code, but you can't just kind of trust it all the time. I mean, where are we on this adventure?

Yeah, I think J AI represent an exciting opportunity for efficiency, but you know, it's not a one size fits all. I can give you a good success story. Actually, last week we had a hackathon at Amplitude and one of our designers basically built a, a mobile app, and it was exciting for him.

And I actually chatted with him and say how it works. He said, yes, I could use Genive AI to help me build a front end, but I needed real engineer to help me format my data and how the data flows. But it enables, uh, him as a designer to move fast.

So if there's not a engineer on the par right now, he can help and go and do things faster and iterate and actually have a kind of a, a, a coding body to help him go much faster. So that enables a lot of things, but assuming that Genive AI is gonna replace all of the engineers and all the codes are gonna be written, I think we are far away from that. Uh, you know, it's, I really like the term co-pilot.

Uh, you know, as GitHub co-pilot is called, it is a co-pilot. It's helping people to, to see what benefit they can get. I feel like it's a little uneven right now in that there is a co-pilot for the developers, but not as many co-pilots for the backend DevOps software engineers that are managing those pipelines.

And will there be a certain amount of imbalance in the system as the developers become more productive, but the pipelines haven't quite evolved in the age of ai? Yeah, and I think, you know, all of this, there's a lot of opportunities, as you said on the pipeline and the backend stuff on performance optimization. There's a lot of opportunity and I know that there will be tools that will come that, you know, using ai, but I think it's just gonna be step by step and it's not there.

Actually we are in the process of evaluating, uh, a tool that is gonna help companies basically, you know, test and create automated testing and so on and so forth. So all of these steps are gonna come and, you know, it's gonna be step by step and, you know, probably the easier one comes first and then the more challenging one is gonna follow. One of the issues people are trying to wrap their heads around is it doesn't seem like the gen AI platforms will provide the same answer the same way all the time.

It's not deterministic. Um, so when I'm building software, how important is that, um, to level of consistency because, um, historically we've kinda always wanted developers to be in the same space or at the very least the same kind of methods, but um, does that really matter? At the end of the day, No.

I think as long as you have a developer that does not just cut and pasting what it gets from the generative AI and actually understands that, and you know, you, you are fine. You know, different developers also come with different ways of coding something. Not everybody comes exactly the same way.

So you can have un deterministic, you know, uh, responses from ai, but as long as the developer understand, and again, use it as a co-pilot, understand what is happening here, get the benefits, and, and then have his or her own basically spin on that. And I think, I don't think that's gonna be a major issue. There's also been a lot of talk about the vulnerabilities that might show up, and yet I scratch my head and I go, well, um, it's not clear to me whether the machines are generating any more vulnerabilities than the humans did, because, well, there's only so much expertise to go around.

So what's your sense of the quality of the security of the code being generated? Yeah, I think that's a very important point, and you had a good point. You know, if you go back 10 years ago, I dunno, five years ago, uh, people would still go to stack overflow and cut and paste codes.

So if there was a vulnerability in one of those, it would be in vulnerability in a lot of code places. And, you know, it's been like that. So a human being also creates a lot more vulner, a lot of vulnerabilities too.

Gen ai, you know, creates vulnerability the same way because it's following human being. You know, it's, it's trained on, on the code that we have written and all the code that exists. But I think it's very important that you have a strategy.

Think about as a company, how do you think about security? What are the important thing that, you know, checks and balances that you want to have in place? And you know, you know, make sure that you do that because I know it can be very enticing to go fast and to just put everything that you get immediately into production.

But having those checks and balances can help companies basically solve and, and depend on which state they are and what's important can help them solve some of these challenges. You know, we always had code reviews. We had engineers review each other's code so that, you know, these problem don't happen.

You know, make sure that you have similar processes. If even if the code is generated, a lot of it by generative ai. What ultimately do you think the impact on productivity is gonna be?

I mean, I can clearly see that AI is reducing the toil for developers, but it's not like developers are sitting on the end of a workbench just writing code all the time. So, you know, they need a minute to think and solve problems. So how much more applications are we actually gonna see that are being built faster?

Or is it just gonna be more like we're gonna reduce the level of toil and reduce the burnout? Yeah, I, I actually believe, you know, it's gonna help us do the simpler, mundane stuff. You know, one of the most exciting thing and some of the most challenging thing that developers do is debugging and problem solving.

You know, sometimes I always say that you're debugging your heart problems, especially the one that are, you know, doesn't happen all the time. It depends on some timing of some issues. It, you know, you are like a detective, you're looking for clues and trying to put them together to figure out.

It's, it's a, it is a very problem solving process and I really loved it when I was developer. Uh, so those things, I think, again, you can have a generative AI that can give ideas or can be a copilot or a buddy, but it cannot be replacing that. So I really hope, I really think, and I hope that generative AI can help some of the mundane stuff.

One of my engineer was saying that the naming of the, uh, variable that gives me is a lot better than what I would name them. So that's valuable. I'm gonna use the whatever naming I get, some of these, you know, some of the simpler stuff we can do so that our engineers, our developer can focus on more, you know, challenging, exciting problem that requires a lot of human being creativity.

So what should we be measuring, if anything at all, because we have all these metrics that are out there and, but it's not clear to me that they all represent, uh, something that is the actual outcome we're looking for. And yet there is no metric for joy. So what exactly am I kind tracking these?

This? Yeah, actually, it's interesting that you brought that up. One of the thing that, you know, we have been always thinking about is developer experience.

How do we measure developer experience? If you go back, you know, there were Dora metrics, uh, you know, how many, uh, how frequent can you ship? And so on and so forth.

But the consensus is that actually the best way to measure is through surveys and qualitative analysis of how developers think about it. Actually, that's what you do at Amplitude. We, we run surveys to say how developers are thinking about their experience in general.

And I think to also measure how AI is helping, I really believe, uh, that, uh, measuring and getting quantitative, qualitative, uh, response from the developer and measuring how their, you know, life is getting better. That gives you a sense of how much, uh, you know, AI is helping. And, uh, you know, that's what I think that's the first measure I would be looking at.

See if these tools that I'm deploying is actually helping my company and helping us. What does the future of a DevOps team kinda look like? Because it seems like it's gonna be a mix of humans and agents that are performing specific tasks on behalf of the agents, and we're gonna have to construct something that feels like a workflow that spans agents and humans.

How does that kind of evolve from your perspective? Oh, it's a good question. Uh, I, you know, of course it's, it's, you know, we have to wait and see and I'm sure it's gonna be evolving a lot based on how people react.

We really, human being are very complex. We don't know how how they're gonna react. We have to go and, and and see that.

But I really think it's gonna be a, it's gonna be become a very collaborative thing. As you mentioned. There will be agents that are gonna do some mundane tasks, is gonna, you know, provide hints and human beings who can take those hints and those, uh, ideas to make sure that they go to the end.

Uh, so I think it's gonna be very collaborative. Again, more and more can be over time, depending on how well these agents learn and adapt can be delegated, but they really still need a human being in charge to basically help and maneuver and, and lead. Somebody has to orchestrate these various agents.

Um, as we kind of get into AI more, is this as much a, a cultural challenge as it is a technical challenge? 'cause it seems like developers are, they're people and you know, probably a third of them are terrified that somebody's about to take their job. And another third are probably, uh, overly optimistic about what AI can do, and the other third sit in the middle somewhere.

How do you kinda introduce AI and, and set the right levels of expectation into a team? Yeah, you you have a very good point. It's a, it's a very cultural thing.

You know, as you said, you know, people are either too optimistic or too pessimistic. And I really believe the answer is neither end of the spectrum. It's gonna be something in the middle.

And the way it's, and I think developers would really, uh, uh, you know, embrace it and adopt it, is that if they feel like that this tool is helping me do my job better, it's not about replacing my work. Uh, it is, uh, it is, it's about, you know, in increasing my productivity. You know, for example, you know, uh, if, you know, we use a tool called gle and it's, it, it's not really a lot of ai, but it helps you to search your documents and figure out, you know, what are the, you know, information there and help people onboard it, help people, you know, get onboarded and use, uh, you know, and, uh, receive material much better.

It's, you know, it's very valuable to use it so people use it copilot. If it's helping me as, as I mentioned, name my valuable better, so I don't have or put a better comment that's valuable, I can use it and it's gonna be step by step and again, or, or you know, I example of a designer that I said, it's just gonna help me iterate faster and do a proof of concept quickly so I can explore and create my idea faster. Uh, it's, it is gonna be, you know, people would be adopting that, but if they feel like it's gonna be that, okay, you know, you don't have to do this job and this tool is gonna take over or it's gonna take over ex your exciting job, people are gonna have rea reaction to it.

But so far I have not seen, you know, there's still, you know, you know, the people don't think, you know, still the tool that we are using that is replacing them, but people are okay to take some of these hub. Is it your sense that, uh, we might be able to soon build more applications with the same number of people, or maybe even slightly less, but you hear a lot of folks talking about how the rate at which we're gonna build and deploy software in the next two years will far exceed or anything we've done in the last decade. So is the whole dynamic of the pace at which software is built and deployed about the change?

Yeah, I think what would happen is that we build better things because the speed changes because we get the benefit because some of the simpler stuff is delegated, then we can have, uh, more iteration, we can improve things because, you know, building software is about, you know, you don't have all the answer at the beginning. You build the software, you put it out there, you learn from the, uh, user behavior and how people are using it and you iterate. And, and so, you know, AI can help improve that cycle dramatically, and I think that can help us to build better softwares.

Uh, uh, if you are, if people are, uh, willing to leverage AI and leverage data and, and, and use that, Do you think organizations will swap out tools and platforms based on their AI capability, or are they all gonna start to just assume that well, everything's gonna have a copilot, so you know, we'll just stick with what we got and wait for however long it takes for that co-pilot to show up? Yeah, I, I dunno, I think people, uh, will, you know, it actually, you know, to be honest with you, it depends on how many of these co-pilot do they see and they feel like that there's gonna be another one, you know, and, and I think everybody's still, you know, uh, shaking their head and trying to make up their minds and, you know, ha have a perspective, but as we go and we get more of these and, and they see the value, people would, would embrace them more. But, you know, I dunno how long it's gonna take and I dunno how people are gonna be reacting.

So ultimately, what's your best advice to software engineering teams that are looking at ai? I mean, what have you seen so far that kinda works in, in, you know, in hindsight you wish you knew when you first went down the path? You know, to me, AI is an, is a great, you know, new technology that is similar to how we dealt with previous technologies.

See how it is helping you and how it is enabling you to do your job better. So don't get scared, don't, don't feel like it's gonna, uh, you know, take over job. Let's see this tool, how it can help me do my job better and really leverage that in that.

As I said, some of the things are very small, but that can be valuable. And, and then, then if you can leverage those small things, then it can help you spend your time on more exciting problems and more exciting opportunities and get more creative. And I really, I really hope that happens to a lot of companies and that help companies build better products.

All right guys, well, you heard it here. Hey, um, there's no substitute for hands-on experience and don't let fear rule your soul. And nothing to fear, but fear itself as they say.

But go experiment with ai and the next thing you know, we're all gonna wake up one morning and go, wow, all this awesome stuff that we were freaking out about, we're now taking for granted. So there you go. Shady, welcome and thank you for being on the show.

Thank you. All right. And back to you guys in the studio.

All.