Using the FDAP Stack to Unlock High-Performance Time Series Data Storage and Analytics | Predict 2024

Hello everyone. My name is Andrew Lamb. I'm from Influx Data, and I'm going to talk about Flight Data Fusion Arrow and Parquet, and sort of how we use this sequence of technologies to build our new version of influx data.

And I'm gonna try to connect that back to how, um, and how and why it's related to the ai, uh, theme of this conference. So, a little bit about myself. I'm currently a staff engineer at Influx Data, and I've worked on databases for quite a few years.

I'm also in that context, a part of the Apache Arrow, PMC, which is the governance committee of, of the Apache Arrow Project, which we'll talk more about at the later part of the presentation. In my past, I've actually worked for several database companies, um, but I also spent several years as a chief architect and a VP of engineering at a few machine learning startups. So actually, I feel like I might, my, my background actually might be relevant to, to this particular talk to try to explain how time series and machine learning, uh, fit together and then what we built in influx to, to support that.

So, what I would hope to do today in sort of about 45 minutes or so, is give you a very brief overview of what Time series data is and how that time series data you use it for classic machine learning, uh, and AI systems and how, where, where it's all connected. And then I'm gonna hopefully show where the time series database connects in there. Then we'll talk some about the technical challenges, that story, the time series data, why you need a specialized database, a little bit, what InfluxDB three oh is and why it's a good, uh, good product for that.

And then I wanna spend, you know, as, as much time as I can, we probably won't be able to go too deep, but explain some of the technical choices and, and ways we are able to build InfluxDB three o not just with special, you know, magic marketing sauce, but with real sort of interesting technical stuff. Um, so that, there we go. So let's get going.

First thing I wanna talk about was time series data. What exactly is it? So it won't surprise anyone to know.

Time series data is data that has timestamps associated with it, right? So practically speaking, what that means is it's typically, uh, classic data where you've got rows of, of information that, that represents something often in the real world. And the key differentiator is each one of those rows has time, right?

And actually a lot of real world data has, has, uh, that property. 'cause it's, you're measuring something and, and that thing is changing over time. The a diff an example of some of this not time series data is like your bank account balance, right?

Where that doesn't directly have a timestamp associated with it, right? So your balance is whatever it currently is. Now, the history of things you've done to your bank account, like the, the transaction log, that's, that's definitely time series data.

But the current value of your, of your, um, bank account is not. And the distinction is the, a lot of the existing relational systems were designed for like managing the current balance of your bank account system or ver or type data like that. Not necessarily streams of time series.

Data time series, as I mentioned, is actually all over the place. Anytime you interact with the real world, uh, you almost always have data that originally started out as time series because the real world has time in it and you measure things at some particular point in time. Here's some, some use cases that influx data gets used a lot for specifically managing, uh, IOT devices in the data streams that come off that.

Um, in addition, there's been a classic use case of using this to monitor computer systems themselves, like websites and, and the back end of the distributed systems in general, if you measure anything in the real world, that's called a metric and it has a timestamp associated when you took the measurement. And of course, with the rise of robotics and green energy, that's sort of a sub, a special sub case of, of IOT devices. Um, classically, when people talk about time series data, especially time series database vendors like influx data and others, you'll hear them talk about three different types of time series data metrics, events, and traces.

I think metrics and events are the two most important ones. Traces are somewhat for, for predictive, predictive modeling and machine learning, which we'll talk about traces are sort of a more specialized generalization of one of, uh, of events that are used in like distributed application tracing. So I think that's, we're gonna mostly focus on metrics and events today.

So metrics are things, the distinction is that they both have time, obviously. 'cause they're time series data. The distinction is metrics is typically something you measure at a repeated interval.

That's, that's pretty regular, right? So you're taking measurements of your factories, robots, or whatever, every 10 seconds or every 30 seconds, every minute. Um, and you just do that over time and you, you build up a history of what happened and then events or things that don't happen on a regular interval, like you had an alert in your factory or you got a delivery or one of the robots failed, or one of the processing things got too hot or something, right?

So those, those are still times theories 'cause they, they happened at a particular time, but they didn't happen regularly, uh, at some regular interval. So that's, that's the distinction. So that's time series data at a really high level, you know, I wanna talk, I know the theme of this conference is the dawn of ai.

So now I want to explain how you use time series data in machine learning and AI contexts. And to do that, I'm going to use a, um, example here, like a a little mini setup example. And this, this is like the kind of stuff I did at these machine learning companies.

So, so the way it works, the scenario here that I'm gonna try to describe is like, you've got a factory. It's got robots that are packing stuff into boxes. Everyone's pretty happy.

And those robots, you know, as, as all modern industrial robots are, are highly instrumented. And they can send you information about various sensor meetings and stuff they have on them. So in this particular case, you know, the robots, if you don't give them maintenance sufficient, uh, in insufficient amount of time, they end up with some sort of maintenance problem that takes them off offline, uh, without planning.

So what you really wanna do is you wanna make sure that you are accurately able to predict when they're gonna need to be made, uh, have, have their maintenance done, the predictive, uh, preventative maintenance done so that you don't have unplanned downtime during your your system, right? Nor do you wanna be doing maintenance all the time. 'cause that's overly costly.

So you want to be using the data to, uh, do the maintenance when the machines need it, but not, not more. So the way you would use AI for this typically is you, would you use a process called inference. So that means you have an AI model, which I'm gonna talk about in a moment here, but, so you, you build a model that just, that's basically trying to predict when the machines will need maintenance or when they're gonna fail, or some, some signal that's important to you.

And you feed the input from the robotic sensors to that model, right? So you're whatever sensors these robots head, maybe it's temperature, maybe it's pressure, maybe it's, you know, it's probably a lot of stuff. I I'm not a robotics expert, but this big stream of data coming through, you feed it to your machine learning model.

And then that model, uh, will eventually be able to tell you when it thinks the machine, you know, machine three needs, needs additional pre, uh, predictive ma uh, maintenance. So you should go take care of it now before you have some unplanned outage in the future, right? So that's like the high level where this fits.

So you, you need to build one of these models, right? So I haven't yet shown a time series, I haven't really shown time series data or a time series database yet. So where does time series, where does that come from?

So the reason that you need a time series database in this case is you need to make to basically create this model, right? So some, somehow, or you've created this model based off past historical trends, right? So what the machine learning models do, AI models do, whatever the most, uh, correct term is, is they use past history and then learn the patterns in the past history, historical data that are predicting the events you you care about.

So in this case, in order to train that model, what you've, what you've typically done is you've captured the stream of data from those robots over a period of time and you've stored them in something like if flex tv, right? Or you stored 'em in a time series database so that you have a big long history of, of what's happening. And then you use the history, you extract that outta the database in whatever format is required for the, your machine learning platform.

This is actually what, what some of the other startups I worked at did, they provided this machine learning platform, but then you, you use the historical data to train the model or refresh the model over time. Uh, but you, you, I mean, this is really important to have this training data. Without this, you can't make the machine learning models work.

You can't make AI work without, uh, training data to, to train it on. And you get the training data from the historic information stored in a, in, in a database, typically a time series database. So that's, you know, that's the, that's the high level pitch of, of where you need a time series database.

So let's talk about some of the challenges of storing time series data and what, what it really looks like sort of one level deeper. Uh, I'm a pretty technical guy, so I'm sorry, some of these slides get, get too down to the weeds, but it's, uh, it's something I love. So the time series format that I'm showing here is called line protocol.

It's something that's, well, it's common. I, uh, influx data invented it. It's, um, an open standard.

You can read about it. It's not, not super complicated. And it's basically like a CSV like format.

Uh, other time series databases have similar formats. They, they differ in what they call these, these, uh, concepts, but they're, they all basically have the same idea. So in this case time series, uh, line protocol looks like you have some name of the measurements, right?

So in this case, maybe it's data from your first factory. 'cause perhaps you have multiple factories. You might, you know, this is some way to divide the data up.

Um, when you're going to look for it again, you then have some information that identifies the thing being measured. In this case, you know, the robot has some identifier, maybe al 4 2, 1 54, and it's got some other metadata attached to it, like perhaps it's on the second floor. Then you've got the, the measurements that you're actually taking, uh, the fields is, is what they're termed in line protocol.

So in this case, maybe there's temperature and, uh, pressure and there's probably a whole bunch of other information, you know, but this would be very specific to what you're measuring. And then of course, you've got the timestamp. So this, you know, it literally looks like this.

It's not some binary thing. It's a, it's a textual based format that looks like basically CSV with a couple of, uh, spaces in the key places. So that's, when we're talking about time street data, there's a specific example of what it would look like, you know, so it's, it's not esoteric magic.

Um, it's just a slightly specialized version of, of, uh, of other data. Now, the key things I wanted to now point out is some of the distinct technical challenges of storing this type of data, right? 'cause you, you, you, I just said it's CSV, right?

And there's a million tools including databases and python libraries, whatever that can handle CSV files. So what is makes some of the, what are some of the technical challenges that make this interesting? And you need a specialized database for first one is typically you're not just dealing with a little bit of data.

You typically have streams of data, like large streams very fast, depending on a use case, you might get, you know, you know, hundreds of megabytes or gigabytes every, every minute or second. So the system needs to be able to handle that in that workload very quickly. Not only does it get this high ingest rate, right?

You get lots of data coming in. You might imagine like the individual value of what the pressure of your robot was at like, you know, three o'clock yesterday afternoon. That in individual is not super valuable.

Um, and so you need to make sure that the resources you spend to save that data, uh, is commensurate with the value you're getting. So you, for example, don't wanna be, you probably are willing to pay a little bit more to like manage your bank account balance than you are to remember what the pressure reading was in your, your sensor three day, uh, three days ago. So what that means typically is you have to compress the data, uh, quickly and efficiently, and you can still get at it, but you are able to save it with a, with, um, store it with a reasonable cost.

Cardinality is another thing. The more information you store about each one of those points, right? The, the more and more, uh, potential amount of space it requires to, to save.

So you need to be able to handle that well. Also, it's typically very important, especially for these alert, for certain alerting use cases. Maybe not the historical machine learning, uh, training set that I just described, but other cases where you're driving alerts directly from the contents of the database.

It's very important to be able to see the data that you load as quickly as possible. It's typical. And oftentimes in, um, enterprise data warehouses or something, the refresh rate is like every hour or maybe every six hours or something.

So that means you don't, aren't able to query and see, uh, the new data that gets loaded until an hour after it's arrived or six hours after it's arrived or something that, that has been decreasing in the past. But it, it used, you know, there, there used to be a significant delay and there still is a very significant delay compared to, you know, milliseconds that typical time series databases offer. Also, there's another thing that's very important about time series data is that it's very often the case that the most recent data is really, really important.

And the value of data like yesterday is still far more than, uh, value of data like 10 days ago or a hundred days ago, or even, you know, even a week ago, two weeks ago, that value drops off like exponentially. So it's important that you keep the current data really hot and available. Um, low latency access, you still need to store the historical data, especially for important use cases like ml.

But you have to make sure you, um, you can use the fact that the value falls off dramatically and the access patterns are very different for older data in order to, to again, reduce cost, improve performance. So those observations led to the rise of time series databases as as a category, I think, and in fact this is part of a broader trend in the database industry, where originally there were really, you know, relational databases were basically invented in the 1970s and commercialized largely in the eighties and and early nineties. And those are general purpose databases, largely designed for doing, you know, bank account like stuff.

Obviously it's more than bank accounts, but, but that's a, that's like the canonical example of what they were designed for. And I think what people have observed is that if for certain, for particular workloads that have specialized use cases, if you design a system that makes assumptions about those workloads and you're able to optimize for that particular, those particular properties, you tend to be able to do 10 or a hundred times better than a general purpose relational database, right? So in time series is no exception.

So in this case, by taking, making deep assumptions about time and that, uh, value drop off and the fact that you really need fast ingest really quickly, um, and everything's gonna have a timestamp, you, you can make a database that does much better than just if you dumped everything in Postgres. And that's not because Postgres isn't good or Oracle or SQL Server or whatever, it's because they have been optimized for a broader set of, of applications. And the time series databases can, can focus on the time series use case.

And like I said, part of a broader trend time series is one document databases and searcher are others like graph databases. Recently, um, the hot thing in startups is vector databases. 'cause that's more, uh, aligned with some of the patterns that you use for, for machine learning.

So point of special ed databases are part of a trend time series is, is a, is a key component of that. So let's talk about what InfluxDB three oh is as part of this, this broader trend. So InfluxDB three oh is a time series database, as I described that makes assumptions about the, the data that goes in.

And it's really very, very good at storing time series data. We offer it as both a, um, several different versions of cloud. So that means we manage it for you.

So you can be on a multi-tenant cloud with shared, uh, that's, that's multi-tenanted and shared with, with other users, uh, as a, on a pay as you go model. Then there's, you can also have a de uh, machines that we still operate, we still manage, but are dedicated to you. That's what's called the cloud dedicated.

And then there's also a version that you can run on your own hardware, uh, for large enterprises if you need a self-managed environment. So same functionality in all three products at a high level. I sort of described the flow of like data coming in and you're doing machine learning and, and do some other stuff classically, you know, you'll have, you'll have data that, you know, like the robots in my example, right?

Generate data. Then in your application or your platform, you need to collect that data and turn it into, uh, take it from whatever format it comes off the machine and then get it sent, uh, typically over the network, either locally in your, in your factory or to the, to the, some sort of cloud-based service. You need software and tools that'll, that'll do that for you and deal with spotty internet connections or whatever.

So that's part of InfluxDB. We have something called Telegraph, which is probably one of the most widely used libraries to, to do that data, local data collection and posting it to, to somewhere else in a standard form, my protocol in particular. Then of course there's the Purpose built time series database I'm gonna talk about, right?

You actually have the, the actual storage and transformation that data, and which of course data's not particularly useful if it's just sitting in a database. You need then the tools to, to analyze it, uh, visualize it, generate machine learning models from it. And so then you would hook the data storage system into the visualizations.

And influx doesn't specialize in those. We specialize in making it easy to connect, uh, the, the storage, but we really focus on the data collection, transfer, and then storage and transformation of, of the, the data once it's in there. Excuse me.

So this, you know, is classic, uh, what you'd expect from a vendor. Our new version is, uh, quite a lot better. This is for some, you know, these are metrics that we made based off, uh, from comparing our old, our older versions to our newer versions on a, on a particular workload, uh, is pretty exciting.

I can explain some of the reasons how we got these, these numbers. They're not just magic made up, although, of course, without giving you the backstory and all the details, you have to basically just trust me. But they are, they are real.

It's quite good. And I want to spend basically the rest of the talk, probably 20, 25 minutes, trying to give you some flavor for how we, how we got results like this, right? And it's not magic, but I want to try to convince you of, of how we did it.

So now you're gonna have to bear with me while I put my technical architect hat on, you know, more, uh, more specialized technical things. And I'm gonna spend a bunch of time, I think, delving down closer to the technology. And if you don't understand all this, I think that's fine.

Uh, the, the, the goal is that you don't have to be a database expert, right? This is why companies like Influx exists. So we, we worry about these details, but you know, if you're interested, I'm gonna, I'm gonna talk through some of some of them.

So, uh, the classic time series database architectures, this is also applies to the current InfluxDB versions, uh, one and two. They, and, and, and a whole lot of other existing systems that are out there. The classic, they store data in, in a structure called an LSM tree, which I won't, uh, I'm not gonna go into any more detail.

Don't worry if you know what that is. It doesn't really matter. Point is a particular data structure that looks, that's kind of like a specialized version of, of a common data structure in other databases.

It works very good for certain kinds of time series queries. Not, not others, which I'll explain in a bit. They often have custom query languages.

Influx is, is no, um, exception. We have a language called influx ql. Again, there's there's reasons for this, but it's also, there's also challenges.

And we're not the only ones. Basically every other time series database has their own, um, query language, right? There's prom ql, there's Log ql, there's a whole bunch of 'em that are, that are basically designed to make writing queries against time series data easier.

And then they also typically will have custom file formats not, and the reason they're have a custom file format is for the reasons I was describing earlier, where you've got this specialized information, uh, as time series data comes in, it's very important to compress it and efficiently handle it because there's so much of it. And because the value drops off really quick, you need to be able to have, uh, good compression in order to, to actually be able to afford it, to store it. And so the way to traditional databases took care of that was they had a special file format that that basically specialized, uh, special case, the time series.

So, as I said, there's actually some great things about this architecture. First thing is it's actually really, it, it hits the ingest time is really good, but as soon as you store data in there, it's, it's very quickly readable. If you're looking for information about, I said single series, this means like a single machine, right?

You want the information for a single robot. It's all stored in this exact same sort of next to each other in these structures. And it's very fast to get, um, also, if you've ever tried to write time series queries in sql, you'll know it's, uh, it's a nightmare.

Typically, if I'm totally honest, to express certain kinds of very common time series patterns. Like I wanna generate the, for example, I just wanna find the difference between, I wanna bucket all my, my time and find it for every 10 minute interval. What the average value, uh, like what the change in the average value was or something for my, for my pressure or something like that.

If you try to express that in sql, especially with the gap flowing part, it's, it's a nightmare in almost every, uh, general purpose relational database adds like s SQL extensions in order to do that gap filling. 'cause there's no good way to express it in sql. And then gen doing the difference that I described is also, uh, unpleasant in sql.

So all, so the point is, a lot of these custom DSLs actually makes it really easy to do. And so they're, they're very for, for specialized time series queries. They're very nice.

However, there are of course some limitations, right? The first is, if you're doing queries that are more complicated than just give me the current, you know, or past values for a particular robot. Like, tell me something about all the robots on the floor.

Two, that can get quite bad, very, you know, because they, of how they've laid out the data and because of how these engines are designed, that type of query often does not run particularly great in a classic time series of database architecture. The other challenge is because of how, 'cause of the, some properties that TSM tree or LSM tree structure, which again, I'm not gonna belabor you with details due to the properties there, uh, it's often very expensive to hold and, and basically too expensive to hold data, uh, beyond too far a window. So like two weeks or maybe a month is common for the, the maximum retention interval.

And if you want any farther than that, you have to have a second system to hold the longer data. And the other challenge too is in addition to the custom query languages, you also typically have custom APIs, right? For in the sense that they're proprietary, they're specific to the current system.

And if you do that, that just means, now if you want to connect, for example, your database to machine learning like your ML tools or your your Python machine learning, uh, workflow, you've now gotta have a connector for each one for that that's special for that time series database and knows how to pull the data out, get whatever you need, interact with the database in its own special way, rather than, you know, more standard tools than like standard other database tools. So that leads us to bring, like, well, what I think this was basically what Paul Dick's sort of, uh, pitch for me to join Influx for is like, hey, you know, we, we've done great with the classic database architecture has great properties in certain areas, but other areas we really wanna improve. Specifically, you might be unsurprised to know that it's the areas we highlighted as challenges, right?

So what you really want is we wanna be able to make it really fast running more complicated queries that typically involves something called, you know, type, uh, class of technology called ola, which stands for Online Analytic Processing. It's basically pretty well understood by now how to, how to write query engines that work on large amounts of data quickly. But, but that wasn't applied to time series databases.

Um, we don't want to have these problems. If you are trying to write queries or store data that have large amounts of, uh, distinct information about each thing you're measuring, that's Cardinale limits. You also don't wanna be thinking about like, well, how long can I store data?

And how, when do I have to reten detain it? 'cause I, you know, can only afford two weeks worth of time? And you wanna be able to apply also more resources to that system, scale out, you know, as, uh, as your workload grows, and also scale it in if you don't have as much work.

So also, um, in the intervening years, the cloud, in particular cloud services like Amazon and Google and Microsoft Azure have, have grown. And designing for those systems is, is actually quite different than you would've designed for a dedicated data center 10 or 15 years ago. So the modern times, which was basically what the existing Time series database is designed for.

So you really want, um, to be able to put historic data on Cheap Object store AKA on S3, and then have, use your local disc in memory as, as caches and move data intelligently between them. You wanna be able to add and remove, uh, capacity as you need. And as your workload changes, you don't wanna just be stuck with a single size.

And the other thing which I was hinting at earlier is you, you know, being able to be, uh, integrated with the ecosystem is much more important. So yes, time series and SQL is a giant pain, typically, however, however, SQL itself is has such broad, uh, broad adoption that it's, uh, if you talk sql, you immediately get access to a huge ecosystem of tools, uh, that, that you wouldn't, if you have a specialized language, you also probably wanna use a special file, uh, widely supported file format if you could. These days people are trying not to end up with a data in a single silo, right?

Like you put your data in a single database that manages it in some special file format. Now to do anything with it, you always have to pull it out and do something else. People are much more interested now in like keeping the data on S3 in standard file formats on in object stores, on file formats and the tools then go directly talk to that.

As I said, sql, despite everyone loves to hate sql, it's, it's basically the defacto standard for data processing. It's not going anywhere anytime soon. And it has enormous ecosystem adoption, and not just SQL to language, but the actual, uh, interfaces to run those queries.

You know, if you talk, if you have a JDBC driver and, and you, there's you, you make a driver for the, for the database for in a standard database stack, you then get access to a wide, huge, your customers can access your, your system using a huge variety of tools. So that's basically like, you know, some of the major key points that we're trying to get at within InfluxDB three. So how do we do it?

And really, you know, so of course Paul Dix being the li Divisionary founder that he is, you know, his answer to everything is let's build a new database that's not, you know, I love Paul, Paul's a great guy, but I think the idea was you can't really just retrofit the existing designs with this tss, uh, this LSM based tree stuff and, and have it have all those properties, right? You really needed something different. It was, what's his, what's his thesis?

Now it turns out it's actually quite hard. And by hard, I mean expensive to build a new database. So this is, um, I had spent some time trolling through Crunchbase just figuring out what popular database companies, like, how much, much money they raised, and it's easily in hundreds of millions of dollars, even in the days when this was, you know, the, a lot of these companies were founded in the midst of the, of, uh, the last funding boom.

So maybe these are higher valuations and strictly necessary to, to build news, but, but there's significant amounts of money and even like recent raises like Duck db where Mother Duck rather, you know, it's, it's still a lot of money. So the point is, there's a lot of effort required to build one of these things. You often don't get a chance to do that, uh, at a, at a, at another, you know, build a second database at a company, right?

These are, these are building one database companies, so how are we gonna do it? So Paul's, Paul's vision and pitch. Actually, part of why I joined Influx was he's like, we're gonna do, we're not just gonna, we're gonna build a new database from scratch, but we're not, well, not really from scratch.

We're gonna build on top of existing components. And the ecosystem has gotten big enough and be, and good enough that we're able to do this. So we combine forces with people.

So that's, that's, this is the thesis that he started off. This was right, uh, in April of 2020. I don't know if you can read that, but this is the, his basic pitch.

This is when I joined Influx. And, and the basic idea was we're gonna go build a new database engine, yes, a whole new database engine, but we're gonna use the preexisting technologies that are already out there that have matured sufficiently to, to really be able to, to build cutting edge analytic systems. So that's, that's what we did.

I tried to make this, this slide. I feel like sometimes you can go from like, you know, Hey, we're gonna do this thing. And then three years later we unveiled.

And, you know, actually just about three years later, right? We unveiled the new database, announced it, people started using it. You know this, I love this, um, uh, comic here because it, it kind of implies that, well, there's something hard we don't, and then we just kinda like, well, we don't know what happened.

And then poof, out came the result, right? Miracle occurred. Well, no, that's not what I, like, A miracle didn't occur, or we didn't have the best, you know, I, I like to believe we have good, good programmers, great engineers, whatever.

But like, we didn't have another $300 million to invest, right? Instead, what we did was we used this f these F ddap technologies. So FD DAB stands for Apache Arrow flight, uh, D stands for data fusion, A stands for arrow, and P stands for parer.

That's what F dev stack stands for. I think these are some of the key technologies, uh, Paul basically bet on. And I've, I've both helped develop and then also become a real believer in, in, in them.

And with those, using that technology stack, we've got a whole bunch of features that, um, you know, I, I don't think would've been possible in traditional databases, uh, traditional time series databases. So I, all right, so I'm gonna talk about how, how, what is this, how, how we do it. I've been talking about doing it together.

So I think this is something I definitely didn't appreciate when I started influx data. So unlike a lot of open source projects where like a single company, you know, owns it, and yes, it's open source and people can contribute, but really it's one company that, that controls the destiny. You've seen this happen recently with like Hashi, uh, HashiCorp and Terraform and stuff.

Like as long as the company's doing what you're happy with, everything's great. If the companies decides that it needs to change strategy to like make money, for example, or to stop competitors from ripping it off and changes like the license in terms of you're basically out of luck. Well, not out of luck, but, but that, that causes real, real potential problems.

So the, one of the things about the f Ddap technologies is that they're governed by the Apache Foundation. 0 set of things where basically you could, you could provide web service for free is very high quality is implemented in the open, but it wasn't run by any, any single company. Instead, it was governed, it, it, it, they innovated a governance structure where there's shared ownership, shared, uh, responsibility of where it was headed, but, um, you know, but not run by one particular company.

So with that governance structure, what, what the Apache Arrow Software Foundation did was basically use that same governance structure and lets other projects sort of follow the same governance structure, the best practices that they, that they developed Fort building, these big infrastructure, lower level pieces of software that get shared, um, across, uh, different people and are not controlled by one company. So the primary thesis of this, right, is that, like, what, why do you, why, why do you care about open source, right? Like, um, I'm not really a ideal, uh, an ideal idealist when it comes to open source.

Like I, I'm a capitalist at heart. I enjoy getting paid for my work. I'm not trying to give, give it all away for free.

You know, as an engineer, I love seeing the stuff I build, get used by a lot of people. But the, the real rationale for like, well, why is the company interested in this? Right?

This isn't just a hobby project. The reason is because the, the thesis is, is working together, we can develop far more advanced technology than any of us could create on our own as either as individuals or as as companies. And I was trying to motivate that before where like, we're not backed with hundreds of millions of dollars of VC funding to do this project.

You know, influx data itself has raised a significant amount of money, but it's, it's done a lot of other stuff with it too. So the idea is how do you, how do you get people working together with a common cause to, to develop this technology that they can all benefit from? Sound sounds idealistic, right?

But I think it's very capitalistic, actually. It's, it's part of the commoditization of this type of technology. Some specific ways that the Apache software recommendation enables this is that it's, as I mentioned before, very vendor neutral, which basically means it, the projects are not run by companies and companies can't control who gets, who gets, um, on or off the thing.

PE companies can definitely pay, well, I'll talk about that. You can pay to pay people to work on it on individual projects, but the, the actual foundation and the governance of the projects is not done by the companies. It's done by individuals.

It's also something I think is super important. Something I only probably appreciated later in my career, is that there's a huge emphasis placed on community rather than, you know, whatever notion of good code. So the rationale behind this is because any strong community can basically rectify whatever problem you might or might not have with your code.

If you don't have a strong community, like you don't have a bunch of people who are interested in working together to make things better, it's very likely you're gonna struggle to maintain that code base over a long period of time, right? You can have a single genius programmer keeping, keeping a project running for a while, but they will inevitably get burned out or life circumstances will change or whatever. And if you haven't built, uh, a self-sustaining community, your project's gonna not survive over the long term.

You know? So the point is, by being part of Apache's knows about this and is wor, it works very hard to avoid, uh, make sure that projects don't end up in that space. Another major careful is, uh, pieces that communication happens.

Everything, uh, everything is basically done in the open, which is takes a little getting used to. And as I mentioned, you can't, like, a company can't just put people on the governance board, even if they, um, are major contributors. Instead, what they could do is they can, the, the governance structure of the projects themselves, the, the p the current members have to accept, uh, new members.

And typically that's by contributions to the project. I think it works great. It's actually done a great job withdrawing industry con contributions too, right?

There's a lot of people on these slides who are either direct or indirect competitors with each other, but under the aegis of the Apache Software Foundation, they can still find ways to work together, use the classic up into the right slide, showing that, uh, over a long period of time, these projects have been able to sustain, uh, community growth, which is good. And now I'm gonna switch, you know, in a couple more minutes, we're gonna blast through some of the technical details of what Flight and Arrow and, uh, parquet and Data Fusion r. So Arrow flights, um, you know, what it, what it is at the high level is it's a way to pass data back and forth across the network in, in an efficient column form.

I'll try to explain a little bit more about what that does over the next couple slides. But here's the basic use case, right? If you, if your ecosystem looks something like, like this horrible spaghetti mess where each service, I'm sorry, these names are probably, you know, five years outta date, they should, there should probably be some, uh, more, uh, more recent systems.

But, but the, you know, these, these are good, good names. If they all have their own APIs to talk to each other, the challenge you'll get is, as a new vendor, you want to go fit into the spaghetti, you know, which one do you con connect to? You only have resources to connect to a few, which ones do you pick?

And then what happens if you didn't, didn't get the right one, right? If your customer earlier wants to put your data and get it out of Cassandra for whatever reason, but you only connect it to, to Pandas, like, you now have to go invest in maintaining a bridge to Cassandra or something like that. So this, the idea behind flight is that by a standardizing the communication patterns, uh, and best practices of how to efficiently ship data back and forth over the network, we didn't have to invent this in influx, right?

We could just follow the same, um, standards that that other people had already already mentioned. And it's, it's built on top of sort of the industrial end research best practice that have come out of the last 10, 15, 20 years in column or database, uh, and, and column execution designs. So that's what Arrow flight is, and it allows us, you know, it's, it's quite cool actually.

We can, writing a, like writing a Python client for Influx DB three oh is, you know, it's like 20 lines long, not because there's only 20 lines worth of logic in there. It's because most of the heavy lifting of actually transmitting the data back and forth is handled by the underlying ar uh, arrow flight implementation, which is shared across a bunch of different companies. And then we have just a small amount of additional logic on top for that's influx specific.

So that's, that's one of the key technologies, right? This is a big e part of the ecosystem integration piece we were talking about. Apache Arrow itself is another arrow that the project has grown to encompass a lot of different things.

So even though it's Apache Arrow, this, this variety of different technologies that fall under the same umbrella. So the original Apache Arrow project is actually a memory format, which I'll explain in just a second. Um, here's the, you know, a technical picture of it that you can read if you want.

What it, it, it's, it's a similar to the counterpoint of Arrow Flight where, um, there's a standard way of sending data over the network. Arrow is a standard way of representing data in memory and, and not just to any way of memory. It's a way of representing data in a column or form in memory, which is, I don't dunno if you're familiar with, uh, row measure or row minor forms, like for, for a raise, and it's like scientific computing.

It's basically the same idea. It's like if you put all the rows, the values of each row, if you put them together in memory, we put all the values of the same column together. That doesn't sound like maybe a as big a difference as, as it as it might actually be.

The reason putting stuff together in memory, uh, for the same column is, is very important, is 'cause modern CPUs are really efficient at processing the same thing over and over again on a large contiguous block. So if you put the same column, type in the same, you know, like a big contiguous region of memory, which is what a does, there's then lots of specialized, uh, hardware support for, for really like that. That's how you get the maximum, uh, benefit from your, from modern CPU hardware.

So that's, that's what Arrow is. It's this standardized memory representation. Representation itself is, is good.

You don't have to reinvent that. But actually turns out, once you've standardized the me and, and in fact you get a similar, um, integration story in memory as you did across the network with Arrow Flight, now in memory, you can easily pass data that was like created in influx or created from one of these other pandas, right? And easily send it to, to influx or the other way around, um, and not have to copy the data back and forth or write code that will do the translation, uh, which actually is quite, quite good.

That's valuable. But actually turns out, really another thing that turns out to be very important is when you're implementing a system to process large amounts of data quickly, like you need in a database calling or technology is well understood. However, writing the actual calculations to operate on those arrays is, uh, it's not as simple as you might imagine.

It's, it's not complicated in the sense that it's well understood and you can go re like there's plenty of research papers about it, but it's, it's also just takes a lot of time. So I'm just trying to give, rather than just say that I thought I'd give you, try to give you an example of what I'm talking about. And because Arrow is a standard, the libraries have these standardized models, um, you can then invest time and effort in making calculations, uh, very, very efficient.

And then other systems can reuse those calculations. Whereas if we weren't able to use Arrow, you know, influx, we would've had to re implement something like this ourselves. And in fact, if you look at all the proprietary vectorized engines, they almost certainly have something like this in them, right?

Like Snowflake or Duct DB has this or, um, photon and from from Databricks, you know, they'll all have something like this, but they had to implement it themselves. 'cause they didn't use the Arrow implementations. They, they did it themselves, which is not impossible, right?

It's definitely examples, but it's, it's expensive and it takes, takes time. So for example, if you wanted to compare the values in these arrays, uh, because you're trying to find ones that match a certain pattern, for example, the most efficient way to do this in hardware is you actually, there's specialized instructions that will compare multiple entries of each array together, uh, in a single instruction is you need to arrange your code in such a way that actually use those specialized instructions to get the maximum performance outta your hardware. And the exact instructions, exactly how you structure your code depends on what instructions are available on your particular CPU.

It, it depends on exactly how these numbers are represented, right? If they're four by integers or two by integers or eight byte integers, there's, there's different, uh, differences. So you, so you need to take care to make sure that the, that they're done most efficiently.

And of course, if it doesn't evenly divide, you have to worry about the remainder or whatever. So I'm not trying to, I'm just trying to give you a flavor of like the kinds of considerations you need to consider when you're operating at, in data at this, at the lowest level. Um, and what the Arrow libraries give you is they basically give you someone else worried about that, you know, or the community as a whole worries about making sure that's really fast.

And then you just have some function that you can call that does it all for you. That, so that's, that's called the colonel. Oh, I guess, I guess in this case it's, uh, I'm explaining what that does.

There's like 50 or 60 of these things last time I checked in the arrow implementation we use. And none of them are earth shattering on themselves. But if you just imagine having to implement all 50, 60 of those yourself and maintain them over time compared to reusing the existing libraries that do all the, the best practice, oh, by the way, you gotta handle MO and a bunch of other stuff that, that I left out here.

Uh, it's very, you know, it's very, very compelling, uh, from a, not reinventing everything from scratch perspective. All right, got a few more minutes. We'll see.

I'm, I'm sure it's, it's already kind of a long talk. Next thing I wanna talk about is Parquet another one of these F ddap technologies like this, the p and fd dap in this case, parquet is a file format. It came from a, well, it's actually, it was originally implemented in the Hadoop ecosystem.

In this case it was implemented, I believe it was part of Apache, uh, hive perhaps. I can't remember the exact sub project that it came out of, but it was, you know, it's a, it's a file form, but it's actually based off, um, a paper from Google called Dremel, which describes sort of a, a class, uh, a storage system for, for storing data efficiently and still being able to create and then act actually itself based off, um, industrial res, uh, like a, like a research paper called pax. So the point is, Apache Parquet did not reinvent this stuff, right?

It's actually sort of a, the end of a succession of research, got it initially, um, implement and described in various proprietary systems that eventually made its way to a standard. It's so it, but from my perspective, basically a state-of-the-art, uh, open source column oriented data file format, it's, and it's fast becoming the defacto format for efficient data storage. Um, to give you a sense, I guess here's one, uh, there's a quote from one blog article that shows, you know, for their particular dataset, they had a terabyte of data on S3 when they stored it in Apache Parquet, it was, you know, almo almost 10% the size a little bit, a little bit more than 10%, 13% the overall size.

So it went from a terabyte to 130 gigabytes. And then like not just is it smaller on disk, right? It cost you less, but it also takes much less time to query.

Now, partly that's, 'cause you don't have to fetch the terabyte off data off S3, but partly it's Apache has a, uh, Apache Parquet has a bunch of special ways and indexes built into it that you're able to quickly find pieces of the query that you care about, um, that you can't do with CSV files. Um, but that, you know, so that might be interesting. This is some random guys, I should say some random, some, some blog posts on the internet picking some particular CSV file set.

Like, but what we were really interested in was how, uh, time series data was stored, especially given what I've said that the, the specialized, you know, the original time series, uh, wave of time series databases had specialized formats for precisely this reason to get good compression for time series data. So let's talk about, so we actually did some experiments early on of TSM, which was the format that we used in influx DB two and, and one which is basically an LSM tree plus some extra indexes to be able to find, find the metadata. It's actually kinda interesting for our dataset, which is some internal dataset we had, um, monitoring dataset, you know, the original, if we, it was five, it actually took 5 6, 591 gigabytes in this TSM format and the same data in Parquet us took 118 gigabytes, right?

So, so it's, you know, 20% the size. So it's actually quite a bit better compression than, than even our specialized TSM file format. It was, there's a bunch of reasons for this that i, I can talk about, but I, you know, if anyone's interested, I can, I can tag it.

And just for comparison, if you just take the tsm UG zip, it, it, it does get better than, uh, than Part A, but now you'd have but the query performance and that's just terrible 'cause you have to g unzip it first and then look at it. So, so it's quite bad. And so that was the results in aggregate.

We also did, you know, if we just pick some random files that were, um, smaller, not the whole thing, uh, and we basically found the same thing, you know, there's some variations. Sometimes, uh, it took 18% of the time, sometimes it was only, uh, sometimes actually up to 11% of the fi of the files so we can store in separate files even. So this is, uh, was good validation that you can actually make, use Parquet, right?

Instead of a specialized form and actually get better compression. And that's not because Parquet is magic, but it's because it parquet encodes a lot of additional learning that that happen both in the, uh, academic research, right? And then also in, in industrial paper.

So it has a bunch of features in there that you have to effectively independently reinvent if you want to, um, beat its performance. And it's not that you can't do that, it's just a very expensive proposition to do. And if you're trying to avoid, um, if you're trying to build some new system and you can avoid having to reinvent all that stuff, it's, it's a, it's a significant it innovation accelerator.

Final thing I'm gonna talk about, and I'm a little running behind time, I wanna make sure we have time for questions, is data fusion. Data fusion is a subpart of the Apache Arrow project as well. Uh, at a high level, I'm not gonna talk in detail what it does or, or how it, how it's implemented that have a whole sequence of other talks if you are interested find on YouTube.

But at the high level, you know, any database, you know, you basically need something, which is by the way, called a query engine. Typically, that takes the query, right? Which basically a text string that describes what the user wants and takes information about like what tables exist and where the data is and, and some stuff like that.

Maybe it takes sql, maybe it takes influx, ql, right? This is what we needed to build in flex three. It takes all that information and then very quickly generates the output of the, of the, the query that the user asked this very, there's a lot that goes into this, right?

This whole conferences every year that talk about various aspects of building query engines. It's a lot of, a lot of them implemented in industry, but a lot of them are proprietary. So one way to implement this is you just do do what some other companies have, have decided had to do, which is you just throw a bunch of money at it.

When I say money, I really mean highly paid engineers who, who are familiar with the database internals. The other alternative is you use data fusion, right? That's basically the pitch is that data fusion has cutting edge features.

We can get it to do what we need to do, what we of course have to invest in it to help, to help it grow. But by doing so, we, we had a relatively modest investment of like two of us basically. And then the rest of the engineering team that's working on InfluxDB three could focus on InfluxDB three, three specific features rather than low level query planning.

You know, optimizations, you know, it's, those are very important. You have to have those in order to have fast performance, but they're not particularly special to to, to what we're doing, right? They're, they're basically, well-known general purpose things that you just need to do, much like the arrow compute kernels, because that's what we use data fusion for.

Here's the how data fusion's, uh, organized internally. I'm not gonna talk a whole lot about what all this means. If any of you are familiar with the database of database internals in general, which I have no idea if you are, but if you are, this picture should look basically like it's right out of a textbook because it 'cause this, this is like the textbook organization of a, of a, a query engine where you have information about the you know, the, the metadata and where the data's coming from.

You have the queries that come in. They get created, uh, into some description of how the plan's gonna go to executed, which is called a logical plan, which is typically go undergoes several rounds of optimizations. That's then transformed into something that's slightly lower level, that's closer to how it's gonna actually get evaluated, which undergoes even more transformations.

And then eventually there's optimized implementations for all the different operations that appear in these, these plans. Uh, data fusions, I think special because it's designed, you can basically plug in whatever you want, uh, along the way here, which is important 'cause there's certain operations in a time series database that don't appear in classical relational databases, like the gap filling that I referred to earlier, for example. But that's fine because data fusion's not gonna have that built in.

It's fine. You can use the extension APIs to, to to, to add the missing functionality, which is exactly what we did. Um, and it's worked out pretty well.

So that's, uh, yeah. So that's, you know, basically the end of my talk. I think we have about 10 minutes left for questions.

I'm gonna leave these slides up here. You can go sign up if you want for the multi-tenant cloud today. Um, you can buy it either directly from us or through the cloud marketplaces on the different cloud vendors.

We have all sorts of online content for you to learn about how, how all this stuff works and how to build your applications. So hopefully that was interesting.