Snowflake – Anand Babu Periasamy, MinIO

This is texturing TV. Hey everyone, welcome back to techstruck TV. I'm happy to welcome back to techstrung TV a non-babu parasami.

He's the CEO co-founder of men IO. His friends calling me being we're gonna call him maybe here, too. io or mid period mid period IO, but it's mid iOS the company's name.

Hey be welcome. Good to be back Colin. great to have you so we we spent a little time on the company name and website, but you know, not every and you've been on here before but that doesn't mean everyone out here watching now.

You know knows what the company is about. So why don't we why don't we start here? If you don't mind.

Give us a little background on Min IL. When I always is an object store. Yeah Object Store is basically the primary storage in the modern Cloud native world.

Yeah, like if you in the past you saw file and block to store your data, but nothing new very big. If you see in all of the cloud the data is stored in objects store. They call it cloud storage, but the technology behind it is Object Store and mean I always an object store and it's designed to store petabytes of data and it's open source, and it's the most popular objects to reach out there like one point three million Docker pools per day.

Wow, it's crazy and you know. Maybe you've been on the show before we we went over a little bit of object store and man is as you mentioned open source as well. And so it's become extremely extremely popular especially, you know, it's part of this new stack if you will the new compute stack built around Cloud native, it's crazy.

I actually had just started interview with Tim Hawk and one of the original group. Came from Google. You know, we were talking about Cloud native being.

As much a state of mind as it is a collection of software. Yeah, and you know, if it's into that and part of that, of course the scalability right when we talk about Cloud native we talk about hyperscale and and that you know, that's part of an inherent part of that cloud native equation and and then IO is You know one of those an example of that, you know ability to really hyperscale up. You are on I think the last time was in January, so it's been about six months.

If you don't mind, why not you share with our audience and update. What do you guys been doing for six months? Yeah, and I I touch upon one of the recent announcement.

It's also a reflection of global industry Trend in if you look at the storage industry overall, right the last big innovation in the traditional storage world was hyper converge interface, but it's still same file and block that is San andast type interface, right and that did not address this scale and the workloads like if you see fundamentally the primary storage was databases and virtual machines. They're operating system image data and stuff in the new world. The object storage is basically addressing the same workload at much larger scale and the class of applications also changed the virtual machines became containers.

And where does the continuity Image store object storage? And where does the database store objects tour and snowflake is actually the one of the most modern databases out there. At scale and if you look at snowflakes snowflake is fundamentally built on objects tour and as the multi-cloud initiatives became the priority for most of these modern organizations the snowflake a snowflake went through a change in their design like our I would say they announced a new feature called external tables instead of you putting all your data in Snowflake.

Now, you can be in control you can keep data at scale and points no click to your data infrastructure and snowflake announced Snowflake and mean I know partnership the last meeting six months that was one of the biggest announcements and it's actually a reflection of a global Trend that is like, it's not just no play Given other databases are beginning to work with us, but snowflake being one of the early Pioneers in this space, it is a significant Trend that is database, which was the primary workload of sand and as are now pointing to Object Store and also they're In the right direction in terms of the databases are getting reduced to the query engines. They focus on processing the data and storing the data they live they're living into Object Store and this combination allows you to build multi-peta by it like your tables themselves who would have thought about a table can be 10 petabytes, but nowadays. It's actually not not that scary at all and customers using it.

Let's just make sure we got this straight here for our audience. So what first of all what we're seeing is the decoupling of storage to the database, right the the database no longer necessarily lives on the same infrastructure as the storage right and and that didn't end up itself. You're right.

That's right radical, right? Because it was always that was attached right? It was like this.

Yeah, that's number one. So now we could have our dad data. Stored anywhere where it makes the most sense whether it's because it's the cheapest most scalable fast this most reliable what have you and then point our database to that data.

And it acts on it, which is pretty cool when you think about that, right? It's what you said was actually an important point, right you may be in control means that this time around right? The data can be in a universal format like Park a compassiously like like they call it high meters store or Iceberg.

There are quite popular open formats. Now, you can actually use data break spark to process your data and dump it on that object store and then find snowflake. I did you couldn't have this kind of collaboration this kind of control before by keeping the data in open format and point Snowflake and pick it up and good run queries on those tables that that's what customers like in terms of control and it can be multi-cloud.

I love it. It's a great dad said it's great. Snowflake is one of the companies doing it and it's about six months.

You guys are doing this. Now. What kind of pickup are you seeing in in the market with this?

This is a new announcement. This has the reason why snowflake work done. This was because customers started seeing more and more data and even like given the mini will scale.

There is already quite a bit of data on existing mean I would deployments for customers to be able to point to it. And also we are seeing that customers are not only having menai own on multiple clouds even on the data is gay wherever data is getting produced whether it's age or private cloud or the public they are now. I mean, I was customers are unifying that as multicloud and then this time snowflake can can directly run queries on this that was the reason that snowflake saw this kind of requirement from their customer base and they It's relatively new but it was very much awaited I can I can imagine that option will pick up very quickly.

Yep, you know I in my mind I'd be it. This is really hitting on two sort of megatrends that you and snowflake have kind of hit latched onto here the first mega Trend I think is that it's the It's the move to data anywhere, right we have work from anywhere during covid. Everyone's working from anywhere and everywhere.

Well day the day goes with them, right and so it's data everywhere and anywhere we do it. We have data on the edge we have data. On devices and things just forget people just things that are connected.

We have dated there with data on the edge. What goes back to the cloud what doesn't go back to the cloud. Which Cloud does it go to how does all that interact and we still have private data centers.

Let's not forget about them. Right so it's data anywhere and if you're gonna live in a data anywhere world You can expect if you're a snowflake or something like that. You've got to go where the data is.

You can't you know, if my mommy can't come to the mountain the mountain comes to Mommy, right? Yeah. You can't expect people to bring the day to you like that.

Number two. So I've been in technology a really long time and you know, the Mantra used to be is that everything can be measured we measure everything right? And so we kept as much data as we could.

And the good news was there was always sort of a Moore's law in effect where they do data storage was always getting cheaper right when you buy 30 40 Meg hard drive. They were this big, you know card hard cards. Yeah, and now I I would a cheap little drone I bought a I think was 128 gigabit micro as deep.

Yeah, you know, yeah for under 100 dollars, I think was like 60 dollars of so, you know storage has gone cheaper, but when you start talking about storage at hyper scale It's not that cheap right? It's not it's not free, you know, and I we see it with our own references this very text drug TV. Yeah.

Yeah, we're adding and we're not, you know, HBO or Netflix, but we're terabytes a month. Yeah of data and I don't I don't get it. Well, where's all this?

You know, but we got a storage somewhere. So we're always on the hunt. Yeah, right and in a perfect world, I'd like to be able to store my date first will not old daters created equal.

Yeah, right. Yeah some of it. I need Closer by some of it and I need access to faster right?

I need more reliable. What have you and and then? I want to be able to offload on load, right but still have access.

Yeah, that's the world we live in today, which is a perfect world for men IO. Yeah, right then I use the perfect solution for that world for that. Yeah.

They you the original was actually centered around the point. You mentioned that data is going to be everywhere this this was the reason right if you look at me now, it's even AWS is three compatible if there is already AWS history. There was no point for me now, right if all of the world's data was inside AWS then I myself would recommend AWS S3, right?

We saw that bulk of the world's day is actually outside AWS data is getting produced all over the globe and I cannot imagine all the data will be produced inside AWS. It's just not possible. Right and if that's the case then what what would the industry do if each one of the cloud even till today?

They're incompatible as your blog is incompatible with that Google Cloud Storage is in To test 3 outside the public Cloud all the sand Nash HTA bus was so fragmented and it's impossible to build applications taking care of all these API on the data will continue to be produced everywhere. And then there are bigger problems for the data the cost part that a cloud is pretty pretty good when you are going from Enterprise it like people classes very expensive you go to the cloud and overall that speed up Innovation refractivity. It's better.

But then as you scale, once you build the devops culture, then you will see that cloud is like staying in a portal for a very long time when he can lease an apartment or buy a home that's where even like at scale. I have seen customers of mean I know moving from the cloud to equinix like colos because they know can take me now on kubernetes and level the ground and they can buy machines on these places on Equinox data centers, and I've seen that like at scale customers do that too other customers they want productivity. And they go to they just go multi-cloud go to whichever infrastructure gives you better deal.

But overall the most important point you mentioned is that the cost part when it only makes sense if cost of storing is a tiny fraction of the business, right if I'm putting a computer in my retail store and now hiring computer experts and computer is more expensive than the profit I make it makes no business sense. Right when computers used to be expensive today. You see in the retail business people are reporting to machines machines make all the decisions of your retail business and that's that has to happen in the data infrastructure.

And this is what other things not only giving them the control it kind of democratized it that you can like most of the time they find that at the cost of a like just few months that they've been on that cloud. They can actually pay for the whole structure and these Genes by these machines on a lease basis and load minivan kubernetes. They're completely in control.

This is that this is another reason for midnight who's success too by driving down the cost of data infrastructure now data starts to work for your business. Love it. Maybe Beyond snowflake have any you know, look there's a reason why snowflake some market leader, right?

They recognize this and they're moving. Any other company's partnering with midnight on a similar sort of initiative? We have seen like a starting from like this the spark type workloads there databricks is Sparks, right like quite we have seen that to be quite popular.

Like there are two forms of databases. We are seeing one is that they basically the query engines that they aren't spark and other platforms like even from press 2 to all these like trino DB all of them. They are basically distributed query engines processing on the data.

They have all they have always embraced objects to her the ones that like sequel server to candidate our vertical to the ones that that data warehouse solutions. They started moving in fast in into this Market. I was actually quit surprised at the speed at which they move the one that I'm excited like after snowflake.

The one that I'm excited big is that Microsoft SQL Server surprisingly. They actually play a big role in the big data analytics. The customers actually don't want to be writing their own map reduce functions.

They Have the skills to operate her Loop scale, right? They actually all they want is give me a SQL prompt on your data and the data can be 10 GB or 10 petabytes. I don't care.

I just want to type SQL query and I don't care how underneath underlying systems work. The data warehouse is coming back in a different form this time. It's better scale at a fraction of cost.

Absolutely but that's a great point. I want to make if you want that, you know, we we all as an industry we have shiny Trinkets and Joe whatever is the latest and greatest has to be the best that we can't wait to have it and we pay money for it and and it's like stars in our eyes. But we lose sight this a lot of people that use Microsoft SQL Server.

There's a lot of people that use db2. There's a you know, all of the Legacy databases. Yes, it's not all tiger graph database and it's not all nine series or or even then IO frankly in terms of storage.

There's a lot of different Legacy systems out in the world. We I just want to you know, where we if you listen to us on text from TV, you think everyone has everything in the cloud it's not the case. It's not the case.

Yeah, so Just forget keep going. Yeah. Yeah.

No the database Market will continue to be fragmented like this because not all data is the is equal like you mentioned. Yeah one problem is that scale? I can no longer organize them.

I don't know which data is important which data needs to be fast. And which data needs to be slow. They just want everything to be online all the time.

That's where they put all the data on objects store. But then by converging on to one universal format, or at least few you popular formats as the master data repository. Now, the databases are transforming themselves into query engines historically databases where query language query computation and the storage system right the back in the how the data got indexed and organized they all of this was part of the database but as database scale to address petabytes of data, they realize that focus on query language and query computation and the date, but we'll keep the data in univers.

Format that way different databases can coexist right? And because each of them solved a specific set of problems a graph data, you mentioned tiger graph neo4j at a bunch of them right then if there is key values stored there is Cassandra to like now given Cassandra went to relay to be object native that's called the Astro DB then postgres modern version. There is neon DB everybody is getting really returned that to go Cloud native and build on Object Store.

But the fact here is that they are all getting their focus is turning towards query engine. And then the storage part is coming to Object Store and when it comes to Object Store if they all converge on universal format, then they each of them can solve this upset of the problem. It is impossible to solve all like if this for me nice approach that one database can solve it all they did try it like multi model databases, but exactly really putting different kinds of database into one.

That's that's a hot. That's a hard. Them you are better off picking one data infrastructure like Object Store and S3 apas.

Obviously now the industry standard and Universal formats and they are even there maybe few right like like iceberg is a evolving compressed CSV and Json are still there as long as they are universally recognized keep the data in one format and then bring different query engines to process them. And these databases are going Cloud native, they're containerized and they can be scaled elastically as an advantage of this you can now have part of your data process by data breaks take the data feed it to bigquery to take other output of that feed it to snowflake. This was impossible just two years before today.

It is a reality and it's it's must have if not your you'll be better locked into one particular solution and it will not be the solution for all kinds of problems and it may not be high for scale either. Hey, hey beware. This was the longest 15 minute interview.

I've done in a while. I think we're at 30 minutes. But anyway.

I want to thank you for coming on and you know, obviously there's a there's a great topic for me. I love to talk about this just a reminder talking about this topic. We August 10th.

So I'm not sure when you're watching this. But if it's not August 10th and your world, we are doing a virtual event on cloud native and we will be discussing a lot of the topics. But so and the great way that they beat us down here, but we'll be discussing a lot of cloud native.

August 10th, check it out. You can probably get to a detect strong events calm and check out our Cloud native a B. I want to thank you for coming on don't wait six months to come back.

Keep us posted and best of luck with men IO. I love to be here. I'm going to be back soon.

Great, man. All right men, I know that's mean that I and I Owe you them out? right back