The State of AI Development – Techstrong AI Podcast EP61

Hello and welcome to the Techstrong AI podcast. I'm Amanda Ani, and with me today I have Anita Koska, and she is the head of growth at Vellum. How are you doing today?

I'm doing awesome. Thanks for inviting me at your podcast. Like I've seen some of the previous presentations and like the talks that you guys covered, it all seems so fun.

Oh, thank you. I learn a lot. I'm happy to have you on the show today.

So can you share a little bit about veem? What services do you provide? Yeah, so veem is an EA development platform.

We support the whole lifecycle for companies who want to build with ai. So think about if, um, you know, product teams would like to prototype with ai, if they want to identify whether using AI is actually useful for a problem to be solved for the user, the user prototyping tool to test whether this is possible. Once they figure out what works, then they work with engineering, again, using our product to operationalize the whole thing.

And then they use evaluations to stress test it, to prepare it for production. And then, um, you know, the whole cycle also covers putting it in production. And it's, it's an interesting loop, um, that happens between like prototyping and having something in production.

Because interestingly enough with ai, it never ends when you put it in production because AI models are so non-deterministic. Um, they have high variance in their outputs. And so controlling for those outputs is like the biggest task to be done from like PMs and engineers.

And so once they have it in production, then we provide the right tools for them to observe and capture real world, uh, interactions. And then those teams basically feed that data into the product and then evaluate and test and like it all goes in this continuous loop. You, you'll forever work on your AI product.

Absolutely. And I imagine, um, as they're using these tools too, they realize that their needs have changed or they need, uh, other solutions and so constantly adapting the technology. Absolutely.

I feel like, uh, working with AI these days, it's so dynamic with like different models just showing up every day. Just like the last two weeks I think we had around four new AI models from open ai, from Nvidia, and I think like Nvidia launched the best open source model. So it's, it's definitely a continuous, uh, work that all these teams are doing.

Not only, you know, to, to switch a model or to test a new model, but then like the requirements are changing constantly. And, um, working with ai, with ai, it requires for that like active analytics and observations and like if you're building more complex agenda workflows, observability and evaluations, it even becomes even more important because an agent can take so many different directions because it's autonomous and like you need to be able to capture all of those, analyze why an agent made a certain decision, and then be able to steer it better in the next one. So definitely a lot of work to be done.

And um, yeah, I agree with you. Teams are learning every day using these tools. Um, they're just like testing new things, finding out new possibilities for the product, and it's honestly amazing to see it.

Bellum recently released a state of AI development report, I believe, is that correct? Yep. Yeah, exactly.

And can you share some of the key findings of that report? Yeah, yeah, yeah. So, um, the, there were a few quite interesting, um, results from that report.

None of them were surprising, I would say, at least for us because we're in this field just like working with companies, helping them do this stuff. So we see it every day, but like half of the developers that we, uh, interviewed for this report, they are all building an agenda workflow in 2025. So it's not at all something that we want companies to be building.

They are already building and, and like all of these are enterprise applications, but like we can debate all day what is an AI agent? Like what does it actually entails and all that. That's the the first major, uh, result that, you know, it just proved that we are in this agent era where everyone is building agenda workflows.

Everyone wants to automate their job, everyone wants this LMS to actually do the job for them. Uh, however, in a contrast to that, the results actually showed that only 25% of those same developers have actually something in production. So something challenging is happening, like is it hallucinations, is it misalignment between the product, the legal team?

Like there's so many high risk jobs to be done, especially with bigger companies, especially for companies who work in high re highly regulated spaces, right? So there's so much coordination that needs to be done. So many things need to to to be right for your team to be able to publish this in production.

And it's quite interesting, like everyone is building something like less than 25% are actually putting this in production. So there is a lot to be done in evaluations, in observability, uh, just like testing these models across the spectrum of like, are they calling the right tools? Are they interacting with other models in the right way?

Is my output correct? Um, do I need to check for context relevance? Like we've seen a lot of these interesting con concepts that developers were using last year and the year before that where we said, okay, well LLMs are quite useful, they're smart, but they don't know our data.

They don't have the context of our data. And so two years ago everyone started to build rag based chat bot that's retrieval, augmented generation chat bot, and they all were trying to use this new tack, this new vector databases to embed all of their business data in a fast and like scalable way to be able to provide this data to an LM model at inference time, right? So once someone is interacting with my chatbot, I actually want my LLM to be able to look at my DA data and then answer with some context relevancy.

And so like, if you think about it, there's so many things and so many loops and so many things being done in parallel. And the LLM by itself is non-deterministic. And so it made sense that only 25% of developers have actually something in production.

And when we asked that were what are the biggest challenges? And like for obvious reasons, they all said like, hallucinations, but like that's the number one thing that you think about. But what I would say something that we see lately is it's a misalignment between the product and engineering team when they work on this AI product.

So communication is the issue. It's mostly aligning on the things that they care about and the things that they wanna do. And I can expand on that.

Um, so like, let's say for example, we worked with Redfin last year. They were building their own chatbot to serve their customers when they're looking for houses to buy or to rent. And so because they work in this highly regulated space, they needed to be able to put clear guardrails and constraints to the model on how the model should actually recommend you places to live.

It shouldn't discriminate based on any of the protected demographic, um, you know, characteristics protected by the, the latest law that was just passed. And so doing that, you can imagine how hard it is, like how can I manage this thing that I don't know how it actually comes up with this answers, right? And so interestingly enough, their product team is the one that cares about how this output is gonna be created in production engineers.

They don't care about that for obvious reasons. Like that's how products are built. Product cares about like the user journey and how the users are interacting with the product.

And engineers are here to just like do the thing and build the functionalities around it. But for so long, because the field of AI felt like very technical, every company just relied on their engineering teams. And so they be became the biggest bottleneck because we expect the engineers to, um, prompt this model.

So we expect them to write the prompts for this models, we expect them to orchestrate their backend workflows to be able to productionize a solution. We expect engineers to evaluate and test, basically do unit tests, right? To test, um, the, the workload end to end to be able to find some like regressions or like ways on how it can improve the output we expect for them to actually plan for this whole thing.

Like what should be the outcome? How should I code this thing? Which concept should I utilize versus which ones I should avoid?

Memory long-term memory short, like there's so many things that need to be done. And then in production, what do I do with my product Now, like for for real world data, your product might be just performing so much differently than with the test data as you are building it. And so because companies rely heavily on engineers, um, they became the biggest BO bottleneck.

And with that product managers and product teams are now taking the lead and that collaboration between product and engineers need to happen sooner rather than later. So that goals are aligned, necessities are satisfied, product cares about like the output developers should operationalize those things should add, add like error logic management guardrails, like everything that, you know, a good engineer does, but product needs to own this process. And there is a certain third team within this whole group, those are the subject matter experts in the case with Redfin, I know that they work with their legal team who was actually creating a lot of the test cases in order for the product and engineering team to test their workloads on.

So they were basically writing like question answer pairs that the product team was passing throughout the workflow that they built and that they were marking it, whether it's true or not. And they were going back to their prompt to then optimize it. And like as I'm seeing this, you can imagine like how long that process is, like that whole back and forth and imagine if only one person is doing it or one team.

Yeah, absolutely. It's interesting because this is a conversation we were just talking about this morning and that being that the engineers need to think more like product managers because they're creating a product for their users. Yep.

And it goes vice versa. Like these days honestly, we're seeing that product managers are becoming more technical and in the age of ai, the product manager role will become so much more technical. It's probably gonna be like product engineering because we still require so much more adjustments made on these workflows by the product people.

And there is no other way than taking the product people to actually prompt these models to be able to think about, you know, what's the workflow that we wanna create or what's the experience that we wanna create for the user and what are the goals that we're optimizing for? And because this whole process never ends, the product team is always involved All. Yeah.

I feel like also just across the board I'm hearing more and more roles being blended. Yeah. So many different types of roles.

Absolutely. People expecting, uh, to be wearing multiple hats. Yeah.

Yeah, absolutely. And like we can also talk about like junior developers, like these days I feel like company the com, like the companies that we're working with, they're not even thinking of hiring junior developers. It's mostly blending within other functions within the company who are, uh, you know, adopting these more technical roles or technical skills when it comes to AI development.

Yeah, it's like PMs. They need to understand how to prompt these models. They need to understand which is the best model on the market right now.

They need to understand how they can align this model to be able to work in production. Then we have legal, if you're building a marketing, uh, AI powered platform, then the whole team of marketing needs to be able to be part of this whole process. If a healthcare company is building a solution for let's say clinicians, and we have one customer that does absolutely the same thing, um, they're building voice agents for clinicians to use to triage, um, like patients and then a answer questions for patients, like through the phone, everything is automated.

The AI voice agent, it works 24 7, it doesn't eat, it doesn't get tired. It just works and it works very reliably, right? Um, and so with Relay Health, what is really interesting is that they have a whole team or of care navigators who are now becoming more technical and they call them content engineers because they work with all these prompts to then adopt the AI's response or the AI's communication that you're gonna get on the phone.

And they're becoming engineers because they understand how to do that. They're working with technology, they're aligning this technology to work for them so they can automate most of their job and be, you know, more available for more patients. So the patients, they don't fall through the cracks of the healthcare system, which we know is broken.

Yeah. So from your experience working with a variety of companies mm-hmm. What tips do you have to make sure they start off on the right foot and they ensure success when it comes to implementing a new technology solution?

Yeah, that's a great question and it's something that I recently talked about, uh, on a recent AI summit in New York. Honestly, like how we're thinking about that is this whole cycle of things that you need to do that I just like briefly mentioned. Um, one for obvious reasons, as I mentioned, like you need to have collaboration to happen between, uh, all these functions within your company, but how do you actually enable that?

And so what we usually say is like prototyping when um, you want to initiate a new solution for your company that utilizes AI to do whatever you want that solution to do, it must start with the product team. But how do you give the product team the right tools to be able to simulate that whole conversation that needs to happen with an ai, right? And so, so far they've been heavily reliant on the engineering team to put up like a quick demo of some sorts or like to connect some of these models so they can play with it.

Or recently I've heard that a very popular, uh, sales, um, AI powered SDR company. They actually have their product managers in cursor in their IDs coding with the developers to test for the models. But like that's not the ideal experience.

Not every PM will actually understand, you know, how to code. And so one, the first step that a company needs to get right is to give the right tools for the product teams to be able to test different models, to test different simulations. Like how can I integrate AI within this system versus this, how can I integrate AI with my database versus an external tool?

And just providing that visual builder that, um, you know, PMs can use to quickly product prototype things. Once they have that working, then PMs would come to board meetings showing demos. So now the conversation is switched from, I think this is gonna work to, here's the demo, it actually works.

And I didn't even relied on my engineering team, I didn't waste any resources because I bought the right tools, right? And so like at that point, decision making becomes easier. Now, like the team can say, okay, well this absolutely works.

We might need to do some tweaking. What's the next step? Well now let's pull the engineers to tell us like what are the best practices?

How do we organize this whole thing? What do we need to do? Like they're absolutely the experts here.

Once they do that, then you know, we have a first version. Once you have that first version that you feel, you know, happy about that you want to test with some examples, then you pull some subject matter experts, you pull your legal team, okay, well there are some laws that we need to abide. Uh, we work in this highly regulated space.

We care about security, we care about trust. How do we align this according to that? And then they start to write like all these test cases, if a user asks about this, make sure you answer with this.

If a user asks another sensitive question, make sure that the model is aligned to say something that actually should be said in production. And so like they create all like a thousand, if not like 10,000, 50,000, how many examples they might need to be able to stress test this system before they actually put in a production. 'cause like either a solution is very high.

If, if their solution is high risk, then they need to put more time in this stage to to test it internally. Once they do that, the whole thing gets passed towards the engineering team. The engineering team is taking this whole thing thinking that, okay, this is ready for production.

Everyone has tested it, we have a green light. It's gonna operationalize it, it's gonna prepare it for production. Um, and like what we usually like to advise companies is that, um, your AI features should be separately deployed from your main application because the chances are that you're gonna be making more updates to your AI features than your standard software functionalities, right?

Because we again, come back to that concept that we need to update our AI features and we need to tune them even daily, depends on, on the product. So then they create this whole release management, they put it in production, they have all the right like management policies and all of that. Once it's in production, it's the product team again.

Now people are using it. So now we're not only measuring like analytics or user behavior or things like that, we're capturing feedback. And feedback can be implicit or explicit.

Like when, you know, do you use chat GPD? I'm assuming you do like everyone does these days, right? Like chat G would ask you like, was this answer good or not?

That's collecting feedback. So as you vote on that, uh, output open AI captures that feedback and that feedback is stored in the original evaluation table that the product team is looking at. And if they say, okay, well we got some sensitive, uh, outputs in production today.

Our customers didn't like it for various reasons, let's see how we can update how this workload works. But there is a tricky and like very interesting procedure here that should be maintained is that we know that prompts are very sensitive and you don't want to change a prompt that will solve one case, but that will introduce regressions to other parts of the system. And so having the right, uh, tooling around this process as as well like it, it must be done.

Like companies should not overlook this because changing a prompt can actually introduce a lot of regressions and you need to be able to capture whether you're doing that before you publish a new update in production. Like those are the, all of the stages that basically go into this loop that must be done in order for you to have something reliable in production. Anything else, uh, you know, above or like, you know, different from this missing one piece of the puzzle, it can get you some problems, especially for high risk, um, you know, solutions.

That was great advice. So if you were going to sum all that up with one key takeaway for our audience today, what would that be? We usually say test-driven AI development.

You need to run a lot of tests, you need to evaluate early, you need to break your workloads in-house. Users should not be breaking it in production. So evaluate early, build confidence early, put it in production and continuously improve it.

Alright, thank you so much for coming on the show and sharing your insights with us today. Thank you Amanda for the invite again, And thank you to our audience. Stay tuned.

There's more.