S3 Object Storage for Real-Time Compliance Analytics at a Large US Global Bank with Scality

Thank you. And hi everybody. So I'm Ben Marge.

I'm the VP of Customer Success at Skelity. And I'm going to, going to present you a very interesting use case, which is a cloud storage for Splunk Data Lake. The customer is the very large bank in the US and uh, the need was to store about 40 petabytes of Splunk Smart store data on two sides.

Each side fully replicated to the other one with a tolerance to up to two minutes for the replication to happen. And the retention of the data is one year. So that represents about 80 petabytes of storage, uh, after era coding and overhead.

So this, uh, successful deployment was a strong collaboration between Splunk, HP, and Skelity. So Splunk buckets, uh, we have multiple temperatures on the Splunk buckets, and here in that case we have two temperatures. So hot.

This is on, uh, flash and, and VMEs SSDs. So past storage, and once the hot buckets are full, they are tiered to another one. Uh, another temperature, which is warm, and in that case, warm is 80 servers of each one petabytes.

Uh, composing a ring, smart storage object storage works this way. The data is stored to the indexers. The indexers upload the data and push the data to the hot buckets that are SSDs.

So as I said, the one petabyte dish, uh, uh, flash cluster available for the indexers, and then the data is tiered to Kel rank. That is 80 servers of one petabytes, uh, through S3 standard S3 scales, three standards, uh, compared to uh, AWS. And when the indexers need the data, that is not available in the hard buckets, they go directly through, through S3, which weve, the data from the rank.

And as I explained, You mentioned that, uh, you have one year retention requirements Correct for the data on the Splunk Out here. So when the, when the, um, the hot tier, I guess the Splunk hot tier, um, migrates to the ring, then the data is retained in tact immutable on the ring for another year. Correct.

So the data is stored on the ring for one year. So every object that is written by Splunk will be, uh, uh, will stay on the ring for one year, and they have an automatic inside Splunk, uh, way to delete the data and, uh, after one year, so the deletion of the data is not made directly by Skelity, but by the application Splunk in that case. Okay.

Uh, yeah. So are you supporting like incremental, um, change tracking or how I, I'm, so, you know, if, if an object is changed at the, the Splunk tier, is it get replicated automatically to the ring? So all objects are immutable because we are, they are versions.

So every time there is a new right, uh, put of an object, we write a new version of this object that is available for guest, and this new version is replicated to the other site. So all objects are immutable on the ring. Okay, good.

Um, and as I explained, it's a multi-site active active architecture with two sites on in the us. Uh, so we have exactly the same, uh, architecture. We have the Splunk indexers and the cluster of hot storage, excuse me, cluster of hot storage.

And the Splunk is responsible for the replication of the indexes, uh, and the references to the objects that are stored on the ring on the lower level on Apple server and is in charge. Software is in charge of making sure that the replication of the object is done for the data that is referenced in the indexers. And this is through cross regional replication, which is a S3 standard.

So of course, this deployment we have had a few, uh, problems. So the first one was during the, uh, the migration of the d uh, the ingest was exceeding the firewalls throughput capabilities, uh, resulting in replication failures. So what we put in place is an infinite replace.

So an object is replayed, uh, for the replication indefinitely until the the replication is successful. That helps us, uh, ensuring reliable object replication despite the network failures. The second system that we have put in place, uh, is, uh, one of the capabilities of C is, as Paul was explaining a few minutes ago, the way that we can, uh, decouple the storage from, uh, the compute from the S3 services.

And if we need to grow the storage, we just have to add storage server. But if we want to add more compute for the metadata layers, we can just add additional, um, uh, metadata servers. And same for the, uh, S3, uh, uh, stakeless, uh, services.

So in that case, we observed get traffic exceeding the capabilities of the servers in terms of CPU with, uh, spikes of up to 80 gigabytes on both sites, simul, and simultaneously that means 160 gigabytes of freight gigabytes and bits bytes, uh, and the smart store flash, uh, cash was rapidly overwhelmed, which led to all the gets ending on the ring and exceeding the CPU capabilities. So we added additional starrs with decoupled entirely the architecture, and we put one new, uh, Starrs with CPU, known for N VMEs, just a bit of CPU and uh, memory, and we can end up in, uh, what we see now. So, uh, 75 gigabytes of threads on one side, and if we added the, the graph of the other side, we could see the same, uh, uh, speed and throughput.

So, and 100 of the percent of the objects are replicated in less than two minutes. We have maintained 75 gigabytes throughput on both sides simultaneously, and we have a successful decoupled architecture, storage, compute, and S3 stateless, uh, services, a customer that has been in production for over five years. Now, Is it, is the tearing done at, at the bucket level or at the object level within a bucket?

So the, the replication is directly made at the, uh, metadata level. So that means all the buckets are automatically replicated and all the objects within those buckets are replicated automatically. So this, we have multiple ways to, uh, implement CRR at, uh, at, so we just say it's at the bucket level.

So each bucket will have a specific configuration. In that case, it's the entire cluster is fully replicated. So in all new buckets created will automatically be replicated to the other site.