Processing and Analyzing Data in Real Time – Subbu Iyer, Aerospike

This is texturing TV. Hey guys. Thanks for the throw.

We're here with Cebu aheir. And he's CEO for aerospike. And we're going to be talking about the ship to real time and the implications thereof.

So boom. Welcome the show. Thank you.

Mike. Great to be here. We've been doing batch processing for as long as I can remember in real time was kind of always some sort of side little niche kind of application in most cases.

Now, it feels like the opposite is happening. We seem to be trying to process and analyze data in real time. How is that changing the way we think about it and databases and computer architecture in general Yeah, you're right.

I think Mike I think, you know, both of us go back to the days when you know, we used to have You know batch processing as you talked about and you know, by the time you had data available to drive some insight and get to the inside. It was already stale. So as but as you said, you know business functions very differently these days, you know, we look at it as you know business functions and moments that matter.

So, you know organizations, you know want to make decisions based on data that is coming in in real time. Not just in terms of ingestion of the data and storing that data, but also acting on it to drive business insights and business decisions. when aerospike started right we saw this primarily in the ad tech industry because you know, that was the first industry that had a large pool of data in terms of demand and supply of ads so they had to match the inventory of ads and on the other end, you know, they had to basically put the right ad in front of you and I as a customer so that we would engage with that ad.

And they couldn't really do that in leave alone tens of seconds. They had to do it sometimes and you know milliseconds. So what started off as really a single or industry has now permeated to pretty much every industry and we see this need to really tap data from a variety of different sources ingested in real time and then drive Insight from it to make business decisions in real time.

It used to be pretty hard to build those kinds of applications. Is it getting easier? I mean have databases abstracted away some of the complexity.

Well, this was the founding principle behind aerospike when the two Founders actually came together, you know one is a database Guru and the other was a networking expert and they could basically connect the dots and see that with the growth of data. And basically you and I as consumers wanting, you know decisions in in seconds. If not milliseconds, they could see this tension.

And so they said, you know any current database that they saw there at that point in time didn't really you know solve that so they went about really starting from the ground of Designing a brand new database. So that's how aerospike was born. It was really, you know, trying to solve this tension that exists between growing data on the one side the desire by organizations to feed more data because more data gets them better decisions.

And on the other hand the tension is that your SLA from a performance perspective does not go down no matter what a data you're working on. So they ended up building a system from the ground up as I said earlier serving the attic industry and then you know building out more Enterprise scale database with all the capabilities that you expect in terms of backup recovery business continuity multi-site replication Global clustering, you know that one of the lowest latencies of any database product in the market and then they bet heavily on ssds at that time. The company is 10 years old.

So you wind back as you know, 10 years back ssds had their challenges, but they made a call that ssds will get better in terms of, you know, mean time to failure resilience and so on and so forth and that's paid off handsomely. So some of the architectural decisions they made then are helping us actually deliver all these capabilities in a very simplistic manner to organizations does that they don't have to replatform if they run into like scale Dimension whether it's growth of data data from gigabytes to petabytes. Hundreds of users to potentially tens of thousands of concurrent users as the application becomes more and more popular or even throughput demands.

We have customers who are running Millions multiple millions of transactions per second. So we believe that what aerospike offers is very differentiated is very unique in the market specifically to solve these real-time challenges that organizations face. It seems like a lot of folks initially at least through cash at the problem and they put that in front of a legacy database architecture and a lot of them still do that.

Is that sustainable? Well caches are you know going back to people point about computer architecture? You know, I you know, my background is an engineering so we were always thought that if you have a issue so to speak as an application Builder from a performance perspective you stick this cash in front of it and in a lot of companies and a lot of developers and a lot of organizations do that, but they're too challenges, you know, one is really Persistence of data.

Number one number two is as your data size, it kind of starts growing. It doesn't fit into a cash per se because you know, you you are basically limited by the amount of memory that's available on that particular system. And the third point is even you were able to kind of build a distributed scalable memory system.

Basically, it gets prohibitively expensive because memory is the most expensive, you know piece of storage that's available out there. So these are the challenges that people run into so, you know, a lot of times we run into customers who started the journey with the cash, but then they realize that either it's not sustainable because of the data growth or it's getting prohibitively expensive and we see that also in this current economic environment Mike with a lot of the cloud database vendors out there, right which is they start their journey and Cloud databases as great that they are from a frictionless adoption perspective. They tend to get prohibitively expand expensive as the data volumes and the transaction rates start going up.

If I think back in time, the prevailing wisdom of the day was always to bring the computer to the data because moving the data is expensive, but then the cloud came along and we started moving data everywhere into the cloud. Are we coming full circle on that thought now are we starting to bring the compute more towards where the data is located at the edge or wherever it may be and we're kind of not as tied up when trying to access things over one area networks. We're trying to process data locally.

Great question. So we see this a lot which is you know, we call this The Edge core kind of architecture. We see this in a lot of our customers where they'll have their Spike at the edge or kind of their immediate processing essentially where the data is created.

So to speak and then they'll have kind of warm storage or cold storage as a core and the beautiful thing about aerospike is we provide the, you know replication between those things between the edge and the core system which is transparent. So the customer or the application developer does not have to write additional code to synchronize data between the edge system and the core system. We also see, you know, what you're talking about extending that a little bit, you know, we have a lot of customers who build customer 360 or you know, customer profile applications.

And what usually happens is whether you're financial services provider. You're a Telco or even a retailer. Your customer data is actually fragmented across multiple.

Within the organization some maybe within the organization some could be you know with in the in the public Cloud but they will be kind of different repositories where some fragment of the customer data is interspersed and to create a customer 360. It's incredibly challenging for them. They either have to rewrite all these applications which really is not tenable or they come and deploy Us in more of what I kind of call the data grid form where they actually pulled data from all these different systems into aerospike and they're getting a much faster way to actually provide a customer 360 view of uni as a customer.

So he see this in financial services in Telco as I mentioned in e-commerce, and then recently we've seen this in healthcare where almost the equivalent of customer 360 is you can think about the patient as a customer and Healthcare Providers trying to get up more unified view of uni as a customer. Do you think in some ways the limiting factor today is not so much technical. It's just our imaginations and we think that a lot of applications may be aren't doable when they are doable.

Yeah, I mean, I think I would say four years back. If you'd asked me this question. I would have said our biggest job as aerospike was to really convince customers of the art of the possible because with every new technology as you know, Mike and new inflection that comes out, you know, unless you show customers that there's a different way a better way to do things.

You know, they don't really you and I necessarily don't you know explore that instinctively but it's changing, you know, we see quite a bit of customers now actually recognizing that there's a different way to approach these challenges. They're no business models that are being stood up. So if you look at a customer base, we have traditional Lodge Enterprises, which are customers across the industries and then we also have you know fast moving digital, you know innovators in every industry.

So, you know, the spectrum of customers really we see customers wanting to reinvent themselves if they're large customers wanting to stand up new businesses and new blinds, and that's only Possible by leveraging newer kind of Technologies in your platforms like aerospike. Are we approaching a point where the amount of data we want to process is just too large and we're struggling with the volume of that data plus how fast it's also coming in and you know, our existing architectures aren't going to be able to sustain that forever. Yeah, I mean, I think we see that a lot.

So, you know one of the initial slides when I talk to customers is really current architecture fail us because we build these architectures over the last decade or so and they don't really they will never designed for today's day and age and you know, they are not going to suit us going forward. So how do you build something that's gonna last us for another decade maybe more so you have to really reimagine the solution and that's what a lot of our customers are looking at. So when you're talking about really real-time ecosystem, we see a lot of customers leveraging real-time streaming engines to ingest data from a variety of sources store it into a real-time database like aerospike in in a very fast Manner and then really run spark jobs or Presto tree no queries for their analytical kind of insight into into the data that is coming and that's happening in a rapid way.

Right. We see customers deploying us within AI ml pipelines, you know where you're building feature stores or other deployments scenarios. So it's really a way that customers have to recognize that, you know, there are limitations within the existing architectures that bill and they're really modernizing if you will there infrastructure Stacks or including their applications.

Are we going to see more usage of machine learning algorithms within the databases themselves? Will they start handling more functions? How automated can automate it get Um, it's already happening Mike.

So we provide the ability to actually filter the data as it comes in Drive Expressions on the data within the database and you know, we'll continue to build on that where you'll be able to actually do a lot of automation within the database itself. The database that we built is, you know, this is this is a word that has been around for a long time. So I'm going to put it in quotes itself healing in the sense that you know, we try to really take a lot of the operational headache away.

So in a distributed database like aerospike, if a node goes down, you know, it doesn't really impact the application, you know, the application continues to perform we rebalance the data because you know, we have algorithms which we try to balance the data says that there are no hotspot on a specific note if you will or a specific part of the ssds that the data is stored in so we take real pride in that if a nor leaves the cluster we rebalance the data transparently. Application and if the node comes back, you know, it kind of rejoins the application doesn't really again have to care about it. We rebalance everything under the covers.

So this is just one capability. So there are several capabilities that we have built to make really the operator's jobs and the sres jobs much much simpler by actually infusing expressions and more and more intelligence within the database itself. And this will continue for sure.

Who's deciding on the database these days for a while? There was the DBA then we saw developers exercise more influence. What today is the who's the individual that comes up with the notion that we need a different approach?

Um, I think you know we see several different stakeholders play a role here developers application developers are really important because they're strong influencers. They are constantly experimenting with new Platforms in new technologies. They may not necessarily be the economic buyer, but they're definitely have a strong voice in terms of you know, which platform which piece of technology to choose the architectural decisions are being made either within kind of a Chief Architect kind of a role or even sometimes the CIO.

So, you know gone are the days where we see cios who have necessarily come from a finance background most of the cios. These days are having a very strong seat at the table with the board and the c-suite in trying to further digital kind of transformation of their own organizations through technology. So CIO CTO architecture Chief Architect and developers tend to be kind of the stakeholders that we talked to.

Put this all together and we hear a lot about the database function itself. We hear a lot about data Ops and we hear a lot about devops is all this kind of converge in your mind at some point or we're gonna have all these kind of specialists in their own individual swim lanes for a while. You know, I I think you know, we've always had the separation between the application Builders and if you will The Operators those lines are getting blurred as you know application developers with the Advent of cicd and with devops, obviously, you know people who are building the applications are also involved to a large extent in terms of operating or at least in the feedback loop, which is, you know, quite quite kind of quick.

We do see kind of data Ops as you talked about evolving in terms of really the data discipline itself. So I do see the read the need to actually have different personas, but I see a lot more kind of tighter integration and feedback loops between these different stakeholders going forward. And so once your best advice to folks about honey move forward from where they are.

Do I just take all these folks and throw them in a room and hope for the best there. Is there some other way to think about Well, I think the it all starts with really, you know, thinking through what i what are you trying to be in the next few years? Because the most expensive decision you could make is kind of replatforming.

Once you start your journey as an organization, and so you don't want to be you know, going down a path where you know, you're going to hit a wall. And once you hit a wall whether it's on a scale Dimension or availability Dimension or whatever that may be it's very expensive as you you and I know to replatform at that point in time. So, you know, what I encourage most people to do is think through hey, what are what are you going to needs gonna look like say two years from now five years from now and plan that way as you kind of think through your architecture fundamentally and then the solutions and the pieces of technology that will fuel that there's no doubt that business is being done in real time.

So you cannot, you know, whether you Financial Services whether your Telco you cannot wait around if you're using PayPal for example, which is a customer. Whereas by when I pay you through PayPal you and I are not going to wait for 20 seconds for PayPal to say whether this is a good transaction or a fraudulent transaction. That creates tremendous pressure on PayPal to really drive an SLE while making sure that this is a good transaction.

Right. So this is repeating itself recommendation engines in e-commerce broad that I just described risk analysis customer 360 use cases across the board even nascent Industries for us as gaming and entertainment. We have a lot of customers now in gaming and attainment who won multiplayer games leaderboards that need to be real time.

So this is something which is right, you know there within every industry. So you as a business need to provide that if not your competitor will all right, folks. I heard it here.

I think you need to assume that data is gonna be processed in real time and start working backwards from there and everything else will kind of fall into place. But if you're thinking about Legacy bad joining applications is the future you may be in a lot of trouble see thanks being on the ship. Sure.

Thank you Mike. It's pleasure. Thank you guys in the studio.