InfluxDB Native Collectors – Brian Gilmore, InfluxData

This is texturing TV. Hey everyone, welcome to another text junk TV segment. I'm really happy to have back on our show.

My friend Brian Gilmore from influx and the folks behind influx DB. Hey, Brian, welcome. Thanks Alan.

It's great to be back. Are you I'm good, Brian. I want to make sure I get it right because I've messed it up in the past influx DB is the database and that's an Open Source Time series database you all right in flagstv, the companies though is in flux influx data influx data that I all things data exactly got it.

So it's influx data and the URL for the company is in flux data to come it is absolutely and you can also find us on GitHub, too. Yep. Okay.

Let's Well, as long as we're on the subject what the heck Brian talk to us about influx DB what it is what it does as well as influx data and and what you guys do there. Sure. Yeah, I mean we are we're hyper focused on a type of data called time series data.

It's really easy to think of this is sort of any numeric progression that you would you would want to track over time. I think the simplest examples of this for most people are things like the prices of stock over time or the you know, the temperature outside of your home over time, but most of our customers are using it for use cases that are in either the it space monitoring like very precise metrics coming off of infrastructure and and applications. We're seeing a lot of use in iot as well.

So all of the sensors and the actuator movements whether it's on the industrial side of the house or on the connected products like all of the smart home here, and then finally we're seeing a lot of emerging use of our product in And scientific academic and Technologies like a lot of places where technology is interfacing with things like outer space or you know, theoretical particle research at CERN those types of use cases. So absolutely, you know a big theme that we're going to be exploring over the next months. Is is What's called the devops of things?

I don't want to take credit for that. I actually learned that word from my friend. Show me Ben.

I am who's the CEO and Jay frog but devops of things right? We have all of these. You know non-traditional Computing devices.

that are running software. Yes are gathering data. And of course anytime we gather data, we have to store and analyze that data and all that good stuff, right so and you know, A lot of it is there's a lot that's scientific and Manufacturing and Industrial and and something like time series.

Databases is a great. Use a great application. Yeah kind of that that kind of data right for the insights it provides for sure.

I mean, even if it's just to understand how that stuff is working and Performing, you know, you're you're talking about, you know, there's a sort of term for a lot of that technology called would you know, it's being called The Edge again, which is sort of related to the old on premises but Edge technology, especially at the sort of scale. It's being deployed becomes very difficult to manage and make sure that you know latencies are not a problem CPUs memory used on those devices from a pure devops perspective. So yeah, we see a we see a lot of opportunity there too.

It's kind of like, you know, we we went in all cases whether it's somebody sort of monitoring the underlying applications and infrastructure of a highly deployed hybrid Edge in Cloud environment, or if they're actually analyzing the data that you know, those Edge devices are interfacing with across that sort of cyber physical divide like the you know, the sensor readings the ACT. Greater control that type of stuff. So it's it's an exciting sort of new paradigm for for technology and and one that requires databases and other technology that really sort of understand and accept the distributed nature of those applications.

And that's that's one of the reasons that we build EDR which we sort of we we talked about the last time we were on the show and it's one of the reasons that we're we're really excited to launch our Cloud native offerings. Absolutely and Brian, you know, we didn't mention your role it influx data. Sure.

Yeah, so I I help with all of the emerging technology sort of application of our program. So, you know, whether it's iot augmented reality virtual reality blockchain artificial intelligence sort of making sure that you know, not only what we're building but how we're presenting that to the market and representing it, you know, it's home for those users who are who are sort of out on the bleeding edge of Technology. Cool great stuff.

All right, we got that out of the way. Let us let's turn to the news and earlier this week you guys announced. I'm some native collectors for like for influx DV and you know, let's assume our audiences in this technical it is as it is and they don't even understand what it collector is let alone a native collector right?

Make us smart on this tell us. All right. I will I will do my best.

So if you think about especially when it comes to iot, but in a lot of other sort of application architectures these days machined machine is like the functional component of the application. So you have machines talking to other machines you have humans talking to machines as part of, you know, automation orchestration coordinates coordination, like autonomy all of that. Now these are these sort of message Brokers that exist at the central Hub of most of today's and the future's machine communication.

So right now these are Brokers for things like mqtt mqtt amqp Co-op you see some use of Brokers like Kafka Amazon Kinesis. And so these are these are already the sort of portals that these many machines are using to communicate with each other. Now getting data from the machines connecting to brokers in the past was difficult because you either had to address the individual machine directly and ask it for information to store in the database or you had to write script or codes that could kind of like, you know, look like another machine or look like a client to the Brokers and could accept data that way.

So what we've done with these native collectors, we've actually created a new service that runs in our influx DB Cloud that allows you to take your influx DB and connect it up to any message Brokers mqtt to start that you are authorized to communicate with and then you can very easily sort of plug into those streams of machine to machine information and essentially eavesdrop in and grab information off of that and automatically ingest it in your database so that you have sort of an automatic and Easy Button way to create histories of the machine to Messages without really impacting or having to change or update or modify your existing your existing applications. Thank you for that. You're here very we've been thinking about it a long time.

So I hear you why because people You know, it is in the tech world right people Bandy about words and phrases and things and and we assume everyone knows what we're talking about. And and then you know, you find out people some people will it be man enough for you know, not proud enough to admit that they don't understand and then others will not be right, but they'll just assume they understand but it's a mistake and assumption and they just compound it. So thank you for that.

Yep. All right, let's talk about influx. And what what what this all and influx be being influx day and and what we're doing here sure.

So if you think in the past, I mean we have ways to get data from these sort of machine to machine Brokers, you know, but typically what happens is you have to deploy a second agent either our Telegraph agent or something else. You have to configure it. He's in configuration files to actually like You know address specific topics on a broker so like a topic like if you think about like a CD right or any type of radio communication a topic is like a channel right where you know, a machine will say I am going to broadcast my temperature on this particular topic and then anything that knows that can plug into that channel and listen in and hear every regular update of the temperature for example, and you could do that before but it was a little bit clunky and especially where the you know, you already have a cloud-based broker say like a high them Cube broker for mqtt or AWS iot or the mqtt sort of interface to Azure iot you ended up having to deploy like another container or another virtual machine just to manage this plugin.

And it wasn't it didn't make a lot of sense especially seeing that most of you know, the communications were already available on a cloud to Cloud influx DB to whatever basis so we created this new service. Now, the user goes in configures a cloud native subscription. They give it the IP address of the broker that they want to authenticate against the get the they put in the authentication information and then they configure subscriptions and those subscriptions can be to one very specific topic like this machine ID this temperature or it can be all machine IDs with that temperature or it could be, you know using wild card.

You can really get smart about which topics you are subscribing to and then once you know, you can figure that you said it the authentications verified influx TV will just monitor those channels for you right in the background and anytime that New data is posted to those Topics by individual machines or many machines or whatever that entire message will get ingested into influx DB. Now one of the things that a lot of developers know about things like mqtt is that they are a Communications Channel and a protocol but not sort of a data format or semantic. So the message itself in mqtt can really be of any format and that's been one of the most challenging things about integrating data from Brokers into databases of any type whether it's a Time series database or a relational database.

So what we've done is we've added a parsing mechanism in between the sort of reception of the of the message from the particular topic that it's subscribed to and you can then apply like Jason parsing rules or regular expression person rules to actually take the raw text of The message and pull out and label the individual pieces of information in the body of that message that you want to store an influx TV. So you can tell it exactly where the timestamp is where the measurement name is where the fields are where the tags are and then basically you are on the Fly subscribing to collecting messages reformatting parsing and storing them now and that data will just appear in real time and all your other influx DB based analytics, whether it's you know stuff that you're doing with flux or even just with our regular dashboards. I love it.

you know, so not only is it, you know native collectors as you mentioned, but cos it's native and because it's from influx data, you know DB itself you I mean, it really is a very quick simple, but powerful way of getting that data in and analyze and used, you know, like harnessing the power of that time series database, you know on data in Is in a really efficient way of doing it. Yeah. Yeah, it's you know, I mean, I think reducing the number of steps and the number of obstacles just the overall technical complexity is always key right because you know good building Enterprise Solutions or customer facing Solutions, the fewer services that they have to manage the happier.

They are so by us taking that sort of piece over for them and just making it available and easily configured. We're hoping that it will actually allow a lot of developers to connect to data sources that before they just didn't have sort of the resources or the town to get to so It's also the United coach kids in football. And my kids are younger every time you hand off the ball.

There's a chance of a fumble the less hand-offs. You have the better off you are excellent. Hey, Brian for people want to get more information on this way.

Do they go sure. com forward slash products forward slash data collection forward slash cloud native data collection Cloud native hyphenated, but I'm sure they'll be a post in in the in the link here. Yeah.

The other thing you can do too is just, you know, take a look at our home page. I'm sure there'll be a flag or a banner, you know, and there's a press release out and there's a Blog company blog and stuff like that. But really I think the best thing to do is just go sign up for influx Cloud account.

You will have to choose the sort of pay as you go commercial model because you know, it's very very inexpensive. You may get offered some Cloud credits, but this is a is a pay as you go only feature right now. So if you sign up for one of Accounts you'll have access to it immediately.

And if you have data already passing through a broker, it should just be minutes before you have that in influx DB and you can analyze it. That's cool, man. I love it.

Hey, Brian. Thanks for coming on and thanks for the education for me and the audience. I appreciate it.

Keep up the great work. Say hi to all our friends and influx data. I will I will and thanks to you all again for your time and for helping us get the message.

Yeah, appreciate it. All right. a native collectors Yep on influx DB from influx data.

Check it out. We're going to take a break here on techstrong TV. We'll be right back.