Daniel Valdivia, MinIO | KubeCon + CloudNativeCon North America 2023

This is Textron tv. Hey, welcome back where we are here at, at, uh, Kup Con in, uh, Chicago 2023. And it's been a busy day.

We have a great day. A lot of great people that we've talked to. We're just around our last conversation, and we'll be here tomorrow.

So don't go away. We'll be back. Uh, but I've had the pleasure being joined by Daniel Valdivia.

Yeah. Daniel is a engineer with Min io. That is Correct.

They're gonna be talking with you. Yeah. Thanks for having me.

And you're, um, not to give you more titles, but you're kind of one of the founder originators of the technology, right? Inventor? Well, I'm responsible for Kubernetes products, and that could be understood as I'm the one, one of the guys who were the operator for Minio.

Right? So I'm responsible for Kubernetes stuff, right? So you, you could argue anything that is has to do with Kubernetes.

I wrote it. Okay. I know you're gonna try and claim all the credit, but you know, your credit, we credit is too, Right?

Yeah. So if there are folks that don't know about Minio, talk about what Minio does. So Minio, it's a cloud native object storage.

So for those, those who of of those who don't know what an object storage is, it's just the simplest form of storage that's meant to be consumed over the internet, right? So, uh, Minio is built for scale and speed, right? So, uh, when you see enterprises and companies that need to start large amounts of data, they look to a big storage, and particularly to Minio, because it's so simple to run, so simple to operate.

And that's what we are going after, right? We want companies, uh, and teams to be comfortable running storage. So they want to, they, they want to look for a solution like Minio that says, you know, it's, it's so easy, mind blowing, simple.

I just put my files there, there, and then I just consume them. And that, that's it. That's what object storage is, right?

Uh, solution for storing, uh, a large amount of data. It's almost like no SQL for programming, object storage for storage. I mean, it's just a, it's, you know, we used to think of it as blob.

Now it's objects, right? That That Is correct. It can contain anything.

Yes. Right? All kinds of, and very large and very large.

Yeah. It's not limited by, you know, so many character size or data size. Yeah.

And then, I mean, one way to reason about it, why, why are we not calling it file? Because, you know, file was meant to be, uh, when files started, it was locally on your drive, right. And object is meant to be stored, you know, in a distributed environment over the internet, right?

So that, that's quite my, like, main difference. Yeah. You can't talk about applications, the cloud virtually anything without the data.

I mean, that's what runs, you know, runs everything, right? Runs Everything. The fuel, fuel over applications.

The, the way I like to see it is, you know, everyone's building all these great products, and I see them as cars. We build roads because everyone needs storage, right? If even if it's a NoSQL database, they need to back up that, that data somewhere and object storage is the place to put those backups.

Very good. So what, um, in, in the Kubernetes world, Mm-Hmm. You know, we're, we're obviously here, there's a lot of interest in, in data and databases, even data databases sometimes in Kubernetes.

Talk about how you fit into the cloud native ecosystem of applications, cloud native database, technology distributed. Mm-Hmm. Processing and data.

So from, from day one, we build minio, so to intend to be running a distributed environment, but then, so, uh, not, not so much as to only make it running in containers, but the mini binary, it's, it's so small, a hundred megabyte binary. You can run it on bare metal on brittle machines or containers. And the idea was that, okay, we don't want to force you to say, oh, if you want to run this storage, you have to buy this appliance from us.

Right? When it comes to the cloud, you don't know what you're gonna be running on. Right?

If it's on a cloud provider, it's under infrastructure. If it's your own Kubernetes on-premise, it's on the hardware that you decided to buy, and you buy the service that you want. So we want a storage solution that's truly cloud native.

That means it will run on anything that you have available. Right? So let's say you, you went with the, uh, vendor of hardware that, that gave you the base deal, right?

You wanna be able to set up your own storage that way, right? So you go and set up Kubernetes, set up mini operator, and, and start your IC storage company if you want. Right?

We wanna make it that simple. And that's what it means to be actually be cloud native, right? Be agnostic of the infrastructure, not tying you down to an appliance or any particular hardware.

Yeah. Very different than the I buy this nas, and so I manage the data a very certain way or whatever product. Mm-Hmm.

Um, it, it seems also that kind of what, that what we containerize can be any variety of things, right? Data applications, you know, infrastructure, full apps, you know, microservices. And, and some of that information could be pulled out of, out of database, out of object storage as well.

So it's, you mentioned like the analogy of the file server. Mm-Hmm. It seems like object storage sort of abstracts, the underlying environment that this is running.

You, you know, whether it's a file server or a NAS or a whatever in the world you want it is behind it or cloud service. Yes. It's still an object storage to You.

It's still an object storage. Yeah, exactly. And the applications are being built with that in mind, right?

At some point, like if you're running, let's say, some big data pipeline on top of Spark, right? The Spark really doesn't care if it's, they're pulling data from HTFS or some object storage. In this case it could be eo, it could be a cloud provider object store, and Right.

So these pipelines are only, all they care is they are agnostic. They just wanna load the data and then do go do its own job, right? So that, that's where Minio comes into place in providing, letting you set up the infrastructure yourself.

Right? Now you work with other database technologies not as a replacement No. Yes.

To necessarily, Right? What what we're seeing is data scaling at, at an unprecedented space space, right? So database vendors they know is relational database, you know, they start having problems when they go beyond certain terabyte capacity, right?

But, uh, a lot of database vendors, what we're seeing is they're actually embracing object store. What they're doing is they're offloading their tables into object storage. And when you need to run certain type of queries or analytic workloads that's meant to run on, you know, petabytes and petabytes of data, the database will actually go and retrieve its own, uh, right ahead loads straight from object storage, right?

And then do its own query and then give you the result. And then we see the trend coming. Databases are actually evolving to work with these large, massive amounts of data, uh, at scale of the data.

And that wouldn't be possible if you didn't have a place to put the data at that scale. Interesting. It's almost like the tiered approach to hardware data storage.

Yep. You know, it's from a software standpoint, database, right? In memory on disc, whatever disc is, and then object.

Exactly. Pull In large amounts of data. That's definitely the right analogy, right?

So just as people were, uh, doing on, on top of hardware, now they're doing it both in a, in data side, right? So you see also, for example, now that all the ai, uh, uh, world is taking off, right? You see these companies that are, okay, we have all these data sets, but what the data sets are massive.

They keep growing month and month, but we only need three months to train the latest and greatest machine learning model. And when the data is staging, we need to tear it off to some other type of storage. So, uh, object storage can be both the hot tier and the warm tier in the sense that you can set up, uh, um, minio tenant with the latest and greatest MBME drives that satisfy your training needs, right?

When you're training a Mac machinery algorithm, it needs to go through every single part of the data set. Mm-Hmm. Right?

But, and as the data is aging, you wanna be able to tier that into another object storage. It's can also be minio, and that's, that's built with, uh, for chip and deep storage, right? Using on top of how these drives.

And all of this needs to be transparent to the applications because the application doesn't wanna know that I need to go to two separate places. Yeah. I just wanna read, right?

I just wanna read my files and they come, and this is what's actually coming at the modern and data lakes. Why is, makes a lot of sense. Why is object storage?

So, I mean, we're all very used to kind of structured data, right? Mm-Hmm. In many forms.

And AI systems can use with structured unstructured data. Why is less structured or unstructured data important? So the, the interesting thing is about roughly like 80% of the data, uh, that we see being place on object storage is unstructured data.

It's machine generated companies like, you know, they have all this telemetry, all these logs, all this data about their business, and they don't know what to do with it yet. So they start storing it. That's why most of these data is un structured, right?

All the structured data still relies on, you know, relational databases and that, that transactional operations that, that still, uh, belongs to the databases. But all this unstructured data that be, sometimes companies are like, ah, I wish I had the data so I could come and predict this behavior so I could improve my e-commerce business. Right?

But if they kept the data in object storage, then they can be like, oh, now I can write a pipeline and go and deduce this, uh, new metric that I want to deduce, right? So that's why all these data companies are just dumping it into epic storage. And they either they have a valid use case for it now, or they don't know if they want to develop a new business case for it later.

But I think it'd be easy and inaccurate to conclude. Just because it, it's object or large amount of data in object storage doesn't mean it's slow. It's actually, you can get high performance out of exactly that data as well.

How do you, how do you do that? How do you, how do you deliver, you know, the kind of performance that applications demand today? So The, I was mentioning that by nature we build minio to be highly distributed, right?

So that already tells you that we are actually aggregating the throughput of all these machines, of all these drives, right? And on top of that, we went to grade lengths to optimize the parts that matter. For example, how, uh, Minio protects the data it uses to ratio coding.

This means you'll gimme some file and I'll take the data, uh, chunk into data and then generate some parity and store it separately to do that. That that's computationally expensive. What we actually went to great lengths to write everything in assembly for this heavy lifting of, uh, operations encryption, uh, TLS and ratio coding, so that it's actually, uh, the hardware is not in the way when we are actually trying to store data or retrieve data at very high speeds, right?

So we actually, the advantage of doing that is we can grab all these nice instructions. For example, on X 86 architecture, we have ABX five 12 instructions. Mm-Hmm.

On arm, we have neon instructions and all these instructions, uh, they, they, they compute very fast. So you want, uh, we wanted to be able to leverage those so that, you know, when you are actually want to retrieve a file, we can pull the file as fast as the drive is letting us pull it. Okay.

How much, just pulling data off a disc, 'cause essentially, right? Yes. What data do you want to pull off?

Um, what, what do you see happening in your space? I mean, kind of what, over the next, I don't wanna be a five year proj prediction, but what do you think over the next six months or more, what's, what's happening in your part of the market and what kind of things should be on the lookout for? So We're seeing two, two main trends.

Firstly, our companies are billing now, uh, they're trying to consolidate their data lakes into like, oh, uh, why do I need to have different stores, uh, or different storage technologies for different problems when I can just have one object store that works great, single name space, everything I need is there, all the security, all the tiering, all the encryption. And then just use that. So we first, we see companies consolidating their data lakes for analytics, for ai, uh, for databases as well, for all sort of, uh, uh, systems, right?

We even see streaming services building on top of object storage now because they, they discover that, you know, it's actually cheaper to run myself. And the second one, we see the repatriation of data from the cloud back on premise, right? And Kubernetes is driving that because people are noticing like, now I can just throw a bunch of servers, put Kubernetes on top of it, and then orchestrate everything very simply.

And nobody, I started being native to Kubernetes as well, makes it trivial for people to say, okay, now I can run the show. Now I can bring my data. Because on cloud providers, when you go beyond certain scale in the petabyte scale, the the cost becomes prohibitively high, right?

You could be buying a lot of servers every month with the bill. Yeah. So when companies find this, okay, they want to build their own data lake and they, this is where part of the first point, and then they start rerating some of the data and keeping it in house because they discovered how easy it's actually to run your own storage infrastructure.

You mentioned distribution, distributed data is one of the way used to get high performance. That's also part of, there's a lot of emphasis on resiliency. Mm-Hmm.

You know, taking more than just uptime and five nines and meantime to recovery, but really trying to build into architecture or the infrastructure layer, cloud service or the application, or kind of all the layers of finding ways to make it so that our applications or infrastructure don't fall down. Yeah. When some fault occurs, or cyber attack or a bug in code.

How, how do, how do object story? How, how is MIN IO help with providing a more resilient infrastructure or application or both? So We, we, we help you protect the data in many ways.

For example, starting with data resiliency, uh, the fact that we are cloud native and hardware agnostic, what if one of your servers goes up in flames? Do you wanna replace it as soon as possible? And the fact that we're not telling you buy our box, it's like you can just buy whatever box fits the need that you need, place it back and plug it back into the cluster and we'll start healing the data on top of that machine.

So that's one, right? Being, uh, hardware agnostic really, really helps in case you have these, these kind of like scenarios where you need to replace hardware or just expand, right? So you don't have to wait for us to procure us.

You go with whoever can give you the service that you need at the moment that you need them, and then just expand your infrastructure like that, right? So that, that's one way we help you, uh, protect for against, uh, individual hardware failure. When it comes to, for example, ransomware attacks object storage.

Because, because we have complete, uh, control over the data, you can actually set it up and we have, for example, over object locking to prepare data deletion in case there's a ransomware attack, right? The, the attacker can try to go and encrypt everything, but if an object has locking or versioning, right, you can always go and restore the other version or the file was not able to be deleted or changed at all. Right?

All data and object storage is immutable. So if, if a ransomware attacker wanted to really damage you, they will have to download the file, encrypt it, and upload it. That will time consuming.

But they, they could, they will just try to delete it. But if you have versioning enabled, you can just restore the other version, right? And what we see also is, uh, when, when you see all these companies building their own data lakes backing up a hundred petabyte lake is very, is very, it's very not impossible, but it's very hard, right?

So we see companies building multiple sites now. So sometimes they have, they like building two data lakes, have them replicate active, active East coast, west coast kind of deployment. Mm-hmm.

And building a third site just for, uh, data recovery, right? It's just a recovery in case there's a, like such types of attack. And then they, they set up the replication and minia makes it trivial to save the active, active close a DR site, uh, kind of setup so that you don't have to worry about, okay, now I need to back up my hundred petabytes.

There, there's no backup. There's another site with an exact replica of your data. How, how about the developer or the platform engineer experience?

If you aren't greenfield, you've already running an application, an environment, a database, maybe object storage. How do you, how do you add to that environment without being disrupted? So that, that's a very great, that's a great question because most of the mini adoption has been driven by the developers and the, uh, infrastructure people themselves, right?

In in traditional it, there's always a team owning storage and they're very seller of it. And that's because of course they bought these expensive appliances, so they wanna restrict how it goes. But, uh, when it comes to a developer, he usually starts on his laptop and runs Minio and says, this works.

I developed my application, test it against the API locally and now I wanna deploy it. So deploying Minio could just be a simple pot and a single PVC, so they can go to infrastructure and be like, deploy me this application. Um, for the infrastructure people, this is just one more application, right?

Because it doesn't need specialized hardware or anything. And that's how it starts through their shadow it, right? And then suddenly slowly companies are realizing, oh, we are realize we are relying too much on that service.

What is that? Oh, that's mini or that's subject storage, right? And then they start formalizing into their companies.

So, uh, for infrastructure people and developers, the fact that they can run a copy of the software themselves locally, uh, or in the inside their CICD pipeline to actually harden the software and be like, okay, it's supposed to be testing it, that it's working as expected at all the time. It's how we actually have gotten one foot into the door, uh, most of the time. Hmm.

Yeah. Very cool. Lots of good stuff happening.

It's great to see. I remember when object databases were kind of a new thing and now object storage. So, um, so somebody wants to kick the tires and try it out.

mm-hmm. MIN IO is correct. Min io.

That is correct, yes. So, uh, like a free account or tier that you can test things on or sandbox? Yeah, actually we are open source.

So you can go to our website and download the, the binary itself and running on your laptop. We run on all kinds of architecture and that's a binary is the one that you will use on production. And if you were to acquire an enterprise license, that would be the, that you run.

Why would we do that? So we are an upstream first company and the open source community actually contributes a lot of, uh, fixes. If we were to break something within 30 minutes, someone on GitHub will be complaining, you guys broke this obscure application that needs this obscure coordin.

Now it affects everything. I mean, could yes, have huge Impact. And, and we immediately patch it, make a new release.

We make releases multiple times a week if needed. And then that benefits everyone, both enterprise customers and the community. So that, that's the best way to, to start, right, to go to our website, get the binary and start using it.

Developers love. Oof. Free source, you know, no sales pressure, right?

No, let me try it out. Let me figure it out If I like it, Try it. Try it.

Exactly. Good. Well Daniel, it's been a lot of fun talking with you.

Thank you. Catching up. I think it was, uh, Valencia when I, last time I talked with someone from your company.

Yes. A lot of good things have happened. io.

Yeah. Super easy to get to. Sounds like super easy to download.

Pick binary po you're running it. So thanks again. Yeah, thanks for having me.

It's been great having everybody here, all of our guests, people like Daniel and from other great companies as well as Min. I hope you'll be here tomorrow. I think we're starting at 10:00 AM tomorrow.

Looking over at our production crew, giving me the big thumbs up. Uh, that's on central time. com, container journal now, security Boulevard, and Textron tv.

So we've in our pleasure bringing this, uh, content to you today, these experts and thought leaders and information. We hope it's been helpful, reach out to us, let us know. And we have, uh, two more days of great stuff lined up.

So we'll see you back here. Same bat channel, same bat station. We'll see you tomorrow.