Batch Data to Real Time Revolution, Arjun Narayan – Materialize

This is texturing TV. Hey everyone, welcome back to techstroke TV. Our next guest today is our gender.

Ryan Arjen is the CEO founder of a company called materialize Arjun. Welcome to Textron TV. Thank you so much Alan.

Thanks for having me. Hi, my pleasure. So Arjun, let's let's start off with you.

Right you we said we said your CEO founder materialize, but give people a little bit maybe of your life story. Yeah, so I grew up in India. I came to the United States for college.

I did a PhD in distributed systems and databases at the University of Pennsylvania where sort of by accident in that I was really into computer science and and fell in love with databases by way of distributed systems after which I took a job at a small series a company at the time cockroach Labs working on a large distributed database and that was really my education in database. Go to market in sort of building a database from scratch all the way up to production Readiness at massive scales and after working at cockroach for a little shy of three years. I founded materialize to build a real-time database and follow a similar Journey any somewhat different Market segment absolutely.

Look we cockroaches. A brand and product not unfamiliar to our audience. com container Journal Etc.

You don't need a lot of people who fell fall in love with databases. What what was it that made you fall in love? What what it really was was the fact that there were multiple fields that were solving problems that were new to them.

But which were old to the database community in which the database Community had already thought a lot about and come up with principal solutions to and I got the feeling that I was reapplying multiple times things that had been learned a decade ago in databases and I felt that that was a missed opportunity for the world for people who would get the benefits of using databases to solve the problems that they have because most people they don't want to build a query planner from scratch or aquarium you optimize from scratch and sometimes people unfortunately have to dig in and do that because the database is of the day aren't fulfilling their needs and I felt that that was an opportunity to take databases make them a little bit more broadly appealing for modern. Problems so that people could really benefit and from reusing the best and breed Solutions. You know, it's interesting.

In some respects. There's no pun intended the database gets no respect right. We're so busy talking about development and testing and security and networking and cloud and this and that, you know, Over I'd say over the last three years.

We've had a Renaissance of hey, it's about the date. It's stupid right because really all of these things that we've built. and do are about somehow Gathering analyzing manipulating the data.

It's the data. That's the the crown jewels. It's the data.

That's the payload. If you will right of of all that we do around, you know, transformation and and so much of our daily lives and you know, Look, yeah, I grew up. in this industry where you know Innovation and database wasn't necessarily words that went together, right you had relational relation database query languages, you know, maybe what 15 years ago, we started getting no Sequel and my goodness could sequel was, you know sequel became king of the castle right sequel databases.

Everyone had one and then you got no SQL databases, but you know in the last let's call five seven years. We have seen an explosion of different style time series databases for instance a graph databases and another fantastic kind of you know Innovation these widely distributed that we started with no sequel right the Cassandra's and this, you know, but we we've seen those proliferate. Cockroaches great example right another great example, what what's unique about materialize what?

You know, what? What would as I think we spoke about off camera every founder you had to be passionate to found the company because it's not easy, right? We all know that what You know beyond your love for databases.

What about what you're doing at materialize you think is so unique and so it really gets you juices going materialize is and I believe the first truly relational database that is built on top of stream processing Primitives to give a really real-time experience for large-scale queries. And we're talking milliseconds. No sub millisecond tens of millisecond latencies for large query staying up to date that you can use directly in your production system.

So imagine a query that's running in a redshift or snowflake data warehouse. If you could have that always up to date without changing any of that underlying sequel you say just take that exact same sequel could be merging in joining 10 different data sets and you want to use that result of that query directly in production. Materialize is the database that lets you do that.

actually, so real time. It sends me like the real-time aspect here is it's kind of the game changer. Yeah, and traditionally many applications have used batch processing.

It's being traditionally so because traditionally stream processing has been two time intensive costly operating Kafka clusters is is difficult. It's usually what people you know, shy away from and and it's it's the tool of Last Resort, but materialize makes working with that real-time data extremely simple because it's just the same sequel that people know in love. In fact, it's it's our compatible with postgrass which is a which is a sequel that most of our viewers and listeners probably very familiar with and that makes it very simple and cost-effective as easy as using a batch cloud data warehouse and and put streaming in batch on the same level playing field in terms of complexity and difficulty and at that point, why would you choose slow if you're getting the speed for for essentially?

No additional complexity or cost? Yep, I agree, you know. Look this move to real time.

It's interesting like from where I sit. you know, I I kind of sit above the flow of the river if you will, so and so I get to see Trends in the Stream and you know the idea of Wiki capture everything To me it Splunk is the great example of this right Splunk, man. What a great invention.

What a great product. That was you could capture everything right? You want to everything that has a log will grab right?

We can have all this data. what a great thing people were like, oh my God, and then we began to realize. Oh my God how expensive is this the cap not only to capture everything but to store everything and to analyze and perform analysis on everything and how it relates to everything else who has the money for this right?

So now we got to pick and choose what we want to store and analyze and act on and blood Well, why can't we do it real time? Oh forget it. It's way too expensive way too compute intensive and there's fundamental reasons why traditional databases?

a struggle so much with real time because fundamentally they are built on the model where you start from scratch every time a user asks a query and if you think about most people who have large data sets, they tend to have large data sets as an accumulation of the historical data set that they've gathered over years and if you ask that same question say 10 seconds from now, right? We ask a question. We crunch the numbers we get an answer and we do everything fancy scale out distributed, you know, you name it and and we get you an answer in five seconds, right and 10 seconds from now.

We want to ask that question again in order to have an up-to-date answer like say something like Dynamic pricing or real-time targeting or of an advertisement these? Most of the data isn't changing right but the fact that the database has to start from scratch because of nonlinearities that it doesn't understand in the relationship between the input data and the answer that's where the expense comes from. And that's what makes it.

So prohibitively prohibitively expensive to use like a micro batch solution, which is just running a batch query over and over again and the fundamental technological innovation that that's in materialize which I came across during my PhD with the fact that you could do everything fully incrementally what that means is in the beginning the first time you ask Aquaria material, it's gonna have to crunch every thing just as any other database would like that you can't break those laws of physics. The magic is the second time you ask a query it's only has to do work proportional to the amount of data that's changed between the last time you ask the question and what's changed and that's really where you get not just speed increase but the cost savings as well. So it's really about the diffs.

Just analyzing the depths. That's exactly right. In fact, the core technology that my co-founder Frank mcsherry invented.

It's called differential data flow. It's essentially running data flows over the depths. actually All right.

So I think we had a little break through there. Now. Let's let's apply this to business use cases.

Give us some give us some use cases for materialize, you know without naming names. If you don't want to that companies are using here to gain an edge. Yeah, this is the this is one of my favorite Parts because our customers and our users do such amazing things and it's always wonderful to see the first thing I'll say is it's a general purpose SQL database in the cloud, right?

So it is broadly applicable across all verticals and all use cases the usual the the clusters of use that that we see most often one is people building applications or data intensive applications using materialize as their core database. It's ideal for building new features or or taking something that you've discovered is very valuable in your batch Pipeline and deploying it into production. The second one is in machine learning serving.

So materialize can be used as a real-time feature store so that you can use the stream processing and real-time data transformation to score your machine learning models online and also serve those models on in production environments in real time. The third one is taking your analytics and making it more real time. Right?

So having a live dashboard makes you much more. Productive than having a dashboard that's updated once a day that tends to be a common starting point for our users because that's such a pain point that they've already familiar with they have the queries already already spec out, but then they quickly move on to Rolling materialize into their core applications and deploying machine learning and production. Got it.

Got it. Got it. You know what 15 minutes goes so quick here, aren't you?

We got to give people if they want to find out more. How where do they go? How what should they do?

com where you can sign up you can read our blog I've written many of those blog posts myself. I think you'll like them and you can sign up for Early Access as materialize cloud is going to be GA later this year. And and I'd love for you to be one of the first ones to check it out.

I'm gonna talk about that. So what's Early Access? You know, I mean, what are we talking here?

Is this would you say a late-stage bed beta or is it? Yes kind of help me develop it. Where where are we so mature lies has paying customers in production today, right?

But we have a new and a new cloud service where it's a snowflake-like experience, which is fully managed hosted SAS product, which is going to be generally available later this year and you can sign up for early access to that experience on our website. So it's available today. It's more of you could you could run this on your own that's right.

Actually for the folks who may be are looking around. How is it packaged if you will materialize a source available and you can download it and there's links on our website to get Hub where you can check out the source download and run it as well. That's cool, man.

I love it. Thank you. com slash materialize ink slash materialize.

Cool. Check it out Arjun. Thanks so much for coming on our show today.

We've appreciate it, man. Keep thank you so much at work come back. If anything when it goes GA the one that's version goes GA.

Let's let's get an update. I'd love to come back. Thank you.

com or com? Excellent go check it out. We're on techstruck TV.

We're gonna take a break. We'll be right back.