Boris Bialek Showcases Real-World AI Success Stories from MongoDB’s New Architecture Guide

Hello and welcome to the latest edition of the techron that AI Leadership series. I'm your host, Mike Bezu today with Boris Bilich, who's global field CTO for Mongo db. And he's got a new book out about AI and well use cases and what people are doing to actually succeed with this stuff.

Boris, welcome to the show. Thank you, Mike. Thank you for having me.

All right, well lift up the book and tell people what it's called and Absolutely. Yes. So here it is, MongoDB Press Architectures for the Intelligent AI Ready Enterprise.

And the name is Funny probably, eh, intelligent and a Ready and Enterprise in one sentence. Yeah. Well there you go.

Could be oxymorons. But um, yeah. What led you to write the book in, in your mind at least?

What distinguishes from, I guess there's a lot of AI titles out out there all of a sudden, so, um, what should people take away from this? That's really, really good question. And it was simply you were sitting in, let's go back to April May timeframe.

And we started this project and it was, there was so much stuff out there and wrong information. Everybody talked about it and when you read it, it was all very marketing fluff. And I don't wanna sound negative about a lot of people, but we try to go really deep.

We are running real projects with this things, what we are doing. We are engaged in deep hybrid search projects and so on. And we wanted to actually bring down all these architecture work we have done with our clients out of real projects in a form that people can read it.

And we started out and it became at the end 500 pages. So there was a lot of talk about it, as you can see. Mm-hmm.

Is there any one client engagement that stands out to you more than any others or maybe a few others or the things that you kind of looked at and just were, you know, just giving or your history and technology expertise just amazed by? Yeah, so there, there's two. We worked with one of the leading cancer research institutes based out of Paris.

And out of that project came a lot of knowledge in the healthcare sector and to really improve personalized, uh, medications and treatment strategies. And this was something before they did this pretty much by trial and error and then they had certain patterns. And with AI and feeding hundred thousands of cases into one big system, we were able literally to move the needle for them.

That is one of the projects that really stays into mind. Another cool project we've done was with Center Reach to help improve the lives of families with autistic children. You may see these are all I choose on purpose, not the obvious ones because yes, obviously we've done the manufacturing predictive analytics and predictive maintenance optimization projects.

We've done a lot of retail, obviously hybrid search. One of the leading apparel vendors comes into mind on that one. We optimize their search experience before you can look for, let's say sneakers and certain models.

Now you can make a picture of a superstar somewhere on television, say, Hey, uh, those sneakers from brand A, do we have something comparable on your brand and you do this please in 50 languages. So these are the projects that move the needle for me where I had these aha moments where things completely change, where it's not just like, oh yeah, we have a summarization. So we've done a lot of those rack projects summarizing data and all of these things together drove our experience.

Yes. Since you wrote the book, there's been studies out from MIT and another one from Wharton and everybody's having a lot of chatter about, well what is the value of ai? What's the return on investment?

What's your take on that conversation right now? Where that stands in? What should people really be looking for for value?

I, I think to repeat here, the famous MIT study, 95% of the projects don't deliver the value as expected. And that was kind of the driver for the book for us as well. We saw that, that people started with, I have an LLM, I have a prompt, I have 50 prompt engineers.

No, I have 60 prompt engineers. My prompt engineers nicer than your prompt engine. It was ridiculous.

And then we started, what about data? How do we get your data out of your data silos that you have a real time experience for whatever your consumer is? People talked about bringing data together for summaries.

And then I said, summarizing what? Well we use the LLM and prompting said, these are your datas. You are a legal company.

How can you put this into an LLM external? Or we haven't thought about let's maybe, yeah, let's rethink this. And we see a lot of these things happening.

So our line is really the key part is the retrieval of the data, of your data, of the client's data is the most critical part. And this sounds a little bit self-serving for a company like MongoDB who employs me, but it is so brutal blatant over the last six months that if you don't have a good handle on your data, you have good retrieval capabilities, excellent embedding capabilities, it doesn't matter which LLM you use, it's just like it's not getting out the data when you need it in the form when, when and where. And this is right now the big part, this is these probabil probabilistic software.

Everybody talks about it. Everybody gets that part, but they forget at the end. It's your knowledge and your data and your information.

What makes a needle move for your company? And this is why I believe a lot of projects fail. They forgot that really basic part.

To your point, there seems to be a subtle evolution going on where we're kind of moving beyond prompt engineering and thinking about more about context engineering. 'cause if I don't give the LLM access to the right data at the right time, I'm just gonna get some flaky answers. So, uh, is that an art in itself?

And how do you see context engineering being embraced and evolving? Yeah, the, this is interesting part. So context engineering and gen systems are very closely aligned.

And when you look at agent systems, it's all about the context in which the system can perceive data, perceive information, how can it make a decision point and how can it act? And the key for this one is actually the memory. When we talk about agentic memory, the agentic memory gives context engineering, pun intended, the context.

And to do that one, we are back to the starting point. I need that context now. I need it in millisecond.

I cannot go to 10 databases each one 40 milliseconds, that's 400 milliseconds. At that point, my consumer is already off the system because we are just talking about the data access. We're not even talking, doing something with it.

And when you have multi, uh, multiple agent systems or multi-agent systems, uh, like we see by now in the banking space, we see customers having 10, 15 different agents interacting and data are not there. You'll be surprised how fast you get really to wrong results. To your point about AgTech, I feel like maybe we're already moving into the next phase of ai.

The first phase might have been co-pilots and now we're seeing the rise of AI agents. Yep. But these AI agents, or shall we say, uh, have a voracious appetite for data and do we need to figure out, uh, policies for what data we expose to them?

Because otherwise they'll just, you know, grab everything and anything they can and inevitably something embarrassing will happen. Absolutely. And this is, this is a fun part.

This is, we have two very interesting partners in the book. One is rec data, which is about tokenization of PII and data which have a certain privacy layer attack to it. That's one of the key partners which was working with us on several projects, specifically in the banking insurance space.

He can imagine, I mean my biggest nightmare would be that my health record ends up on an LLM. I mean this, this should never happen. So you need to have tokenization of these data.

This is a key part and you need it still in the context of the agent, that the agent is able to make sense out of it. Basically, you cannot just garble everything up. Remember the old days when people talked about we are masking everything and then the masking result was the answer is always zero.

That's, we try to avoid those moments. And rec data is one part. The other thing is getting to the mode where people don't say, oh, everything we are doing is, we've done machine learning yesterday and now we do AI and we do it in a data warehouse.

And they said, this is great, but you're suddenly grumbling up and normalizing data, which should not be normalized back to our context discussion. And I like really the idea that data need to be fit for purpose in the systems right now in real time. And then yes, tokenization is one big part.

Archiving is very important. So I'm a big, big fan of what you can't prove what you've done yesterday, don't try to tell anybody tomorrow. So those kind of functions are all a little bit advertising again in the book.

And we thought about it and we have solutions. Another area, if I may jump a little bit wider is we have a partner called Intellect, ai intellect build something they called purple fabric, which helps you actually to orchestrate multiple AI agent systems in a form that you actually can prove and trace what's happening in the system. That's the exciting part.

Building one is easy data, lineage is hard, and data lineage is one of the things. Whenever an auditor will look at an agenda system and say, where are these data for your risk appetite coming from? Well good that you ask and then people leave the building.

This is where purple fabric of intellect AI comes in and we have great results with these guys. Yeah. Um, as you kind of look forward a little bit, um, the number of AI agents will exponentially increase, but they're gonna be invoking LLMs, um, more frequently.

And the cost of invoking LLMs is based on tokens and you generate a token for each input and each output will not, the cost of this stuff spiral outta control if we don't have some sort of mechanism in place to, um, manage the data flow because the amount of data kind of directly equates to how much processing there is of the LM This is a really good point. This is one of my most beloved discussions I have with people actually because here is one of the things, do you have to go every time to the LLM? So I talk to a lot of people about chat bots and the first thing is the chatbot is beautiful.

10 users pilot, then they switch it live and the money runs like, like, like a coin machine. It's like, and it gets fast and fast and says, why do you do that? Why don't you cash the road answers?

80% of these systems are road and we can do extensive road learning on this infrastructure. Why not using those kind of things. And we have as our real time vector embedding capabilities in MongoDB the capability to store actually the answers.

We don't even need to go to the LLM anymore because we know the answer to my flight is delayed and the flight number is X and the airport is y. That's a road question system can answer naturally and the natural answer is already there because the answer was 2 million times given you don't need to go 2 million times to the system to ask the same question. Right?

And these are the things where memory and state about the various agents is very important. Mm-hmm. And we can solve this as MongoDB.

Really interesting. We store these data and we have this not only the state of you, mark, you are, you are in Cincinnati and I'm in Atlanta, but the question we ask is probably roughly the same. Where the heck is my next flight?

And that answer systems, we can preempt those. And the interesting part is you're not ending up with rule-based answers because that's the old way to do it. Then you get really weird answers and you start yelling the system, human, human, human.

That doesn't work. But it gives you the right answer in the right context at the right time. Again, based out of the, the agent system can figure this one out but doesn't need to go to the LLM to generate the human-like text.

Mm-hmm. Based on the fact that I think people are starting to understand that LLMs and AI are probabilistic and that they will give you their best guess. Are people starting to better understand how to apply that into specific types of business processes?

Because if I have something that is, uh, needs to be done the same way every time, then maybe that's not a good fit for AI because the AI never does the same thing the same way twice. But there are all other kinds of processes where that may not be as important. So are we starting to understand how to kind of match the technology to the right process?

Yeah, this is really the second side of the same question, right? If I ask an LM and I need to ask, there are two pauses to this way. What we are taking with our teams.

One is obviously we have our voyage AI embeds and re rankers the quality of the model. It's a little bit like the classic garbage in, garbage out. The better you embedding model, the better is a hit rate on the LLMs.

And the second part obviously are the re rankers at the end. Today, I tell people, you cannot run a system without a re-ran if you're serious about data quality because you need to reconnect the answer coming back from the LLM. And as you pointed out, it may be great, it may be not so great, but you need to understand is it great?

Can I use this answer for my use case? Is it sensible or is there something going horribly wrong? And maybe, and this is the second part of it, we can reroute to a second LLMI love these old saying, I want a second opinion.

And that is really where hits and the re rankers play a critical role here to ensure you get already higher quality results with the good embedding and your customers here embedding really for your use case more and more. And on the other side, you wanna make sure afterwards who guard the guardians is the output somewhere related and matching by input parameters, which on the question I ask and the output then should be good enough for my, for my function or need to be discarded. So this is, these are all these things, how you move from 90% not driving business value to expand the nuggets from the 5% you become bigger and the these things have for us major impact.

Major impact. And yes, I admit, uh, it's in the book as well. So, and uh, typical example I have, there is a large automotive manufacturer working with, we are building a VIN database.

VIN database exists for a long time, but now we are adding additional data to it. And they wanted to inform the owners in a form afterwards that maybe new offerings or somebody's online says, Hey, I'm driving right now and where's the next good restaurant? So the typical LLM questions and connecting that one to the car, to the location, to maybe the things what somebody did before.

And the results were horrible because they had no re-rank in the system. They, they took whatever came out first and threw it out and the results were maybe less than satisfactory for the people. Partially embarrassing.

You can imagine that some people got maybe location given they definitely don't want to go to and no children appropriate and so on. So we got all of that fixed with the reran and the output is amazing. We can have now not only the system reporting that the cars correctly driving predictive maintenance, digital training, but we can actually deliver services to the driver because we know what they want to do.

And that is another good example where the embed and the re ranker played a major part of it. All right, well folks, you heard it here. We would all love to come up with some great new innovative thing that nobody else ever thought of, but sometimes you're better off just seeing what other folks have done and kind of figuring out a way to apply it to your own business processes and well, that's all in the book that Boris has, so go check it out.

Yep. So and maybe one term, we are structured after industries here, so, and people ask me, yeah, but there are only three cases for my insurance sector. What I tell people is we took each use case, each example for one industry out, but you can use a lot of the cases what we discuss for other things.

So what is good for retail? For example, identifying requirements of a customer, what I call the Lucile for a moment. What is your true desire, right?

We are all in Netflix these days. Um, when we take a look to that one, we saw afterwards that the same technology can be applied. For example, in financial planning you'll be surprised or wealth management is useful, the same processes will be implemented for the insurance industry.

So it's quite funny and it's actually good read. I think we've wrote it in a form which is very, um, digestible. We have as well introduction chapter and important is, this is not all about MongoDB.

We have a lot of partners in there as well who wrote parts of the stories. So this is not about Mongo B tries to sell a database and the vector search. This is really about the architecture of the solutions from our partners And clients.

All right folks. Hey Boris, thanks for being on the show. And I guess to Boris's point, even in the age of ai, what's good for the goose is still good for the gander.

Thank you all for watching the latest episode of the Text Drawing AI Leadership Insight series. You can find this episode and others on our website. We invite you to check all those out.

Until then, we'll see you next time.