Democratizing Data Engineering – John Lafleur, Airbyte

This is Textron TV. Hey guys. Thanks for the throw.

We're here with John the floor who's the chief operating officer for airbite and we're talking about data connectors and the fact that well, hey, they're free. But B. It seems like they're being democratized and just about anybody can make one if they want John welcome the show.

Thanks, Mike. Thanks for having me. So this quiet revolution has been going on.

We used to have. It folks would kind of work with ETL tools and take forever to kind of move data in and out and then we started making graphical tools available that simplify the process and then everybody started moving data in and out now, we're creating connectors and just handing them out like candy. So my question to you is what is going on John and what's driving all this.

So I think the first driver is actually is adoption of the data warehouse. To please Snowflake and that's how when you started to see elt tools when we switched the model from ETL to elt because at the point the transformation could happen and that's no flake. So that's one thing that helped a lot and second one is you just have more and more tools just in marketing you have tens of thousands of tools.

So this is a deep need on that and with airbite we've added up to the open source approach to adapt to address that long term connectors and the only way to make your connectors strong and reliable is through usage. Because every use case every user will come with some Edge case and you won't know about it until the connector is used this way. And so without usage you cannot make all those connectors.

We really reliable And that's why at air but we've chosen to make our all our Alpha and beta connectors for free like on air by Cloud. So we have Alpha connectors better and GA connectors GA connectors come with an SLA of 99% And we provide support on that and on elephant that are the way to get them to GA is to have enough users. So we identify all those edges address them.

And then from the success rate we can say okay zrga. And so that's why we've put we've made them completely free online by Cloud. It's been two weeks now and adoption has been really great.

So we're not experimenting on people anymore by creating connectors and telling them boy. I hope that works for you. We're kind of giving them the connectors that when they do get a support contract.

They have confidence that this thing actually works as advertised. 7. So a percent so we try we put in the alpha beta connectors and in Cloud on his ones that we feel confident about and we just want to make sure that we are addressing all the all the edge cases sure.

There are a few features that might be missing for some but that doesn't prevent you from being able to like to to move the data. as you need What is the future of data engineering at least for a while there was one of the hottest jobs in all of it, but now it seems like we're making that simpler for anybody to do. So, what's the role of the data engineer?

And what's the role of the so-called citizen data engineer? So the part of their role is about building and maintaining connectors like data pipelines, but you can ask any of them that will tell you that's the worst part of it like the most boring part. So with airbag with open source approach where we also enable you to build a connector within 30 minutes and we're building a UI on top of that and like we have a demo video where you can say, you can see that you can do that in five minutes now, so like building will be very like very simple now and so if we when we address long tail at that point just moving data will become a community ties.

That's that's our mission that I buy so at the end that just means that you'll be shipping connectors faster so that you can spend more time more valuable project in the end moving data is now what is valuable. She's a company in what is more valuable is the activation of that data what to do with it like so does that means transforming the data so you can do six orders activated another tools for instance So we need to rationalize the number of connectors we have because I feel like everybody who has a platform bills and connector and then if I have platform a and somebody has platform B and each built the connector to connect back and forth. So now I got two connectors and two different vendors to do the same thing and hopefully they might come to an agreement to maybe have one connector one day, but that doesn't always happen.

So do we just have too many connectors out there? And we need to figure out a way to simplify all this. Yeah.

That's what we'll see. How is the market goes like the industry which direction it takes but we hope at one point that's just gonna be some standard the way we have built our connective Builder like lcdk connector on kit on that point is that it's very if it's a rest API at that point. It's very easy to build.

So we've taken the rest API as a standard format if you want to make it super easy now, we could have suggestions on how To to build your API so that building connector even for yourself so that as a as a product as a stat product, you can offer your your clients to your customers to export to data in your any data warehouse. And I think that's a future that is like possible in the next four five years where everybody can even enable them to do that very easily. Right.

So what is your business model? Exactly, you sell them the support for the connectors and everything else is open source, or how do you guys actually monetize any of this? So we have a bite open source, we've got more than five different deployments for the months on that.

So huge adoption of everybody open source, then we have got there by cloud and a back Cloud comes on top of Open Source. So it brings hosting and operation like operating so you don't need to do that. But also it comes with premium features such as user access management multiple workspaces SSL.

So everything that address the need of the team in the company will be more premium all the open source features are more focused on the data Movement by itself. And we have also what we call 800 price where we offer you more privacy more security compliance features, but also new deployment option where you can have the data playing within your own private cloud, and so that they did I never leaves your your private cloud in terms of The protection that's like Best in Class. How active or end users becoming in all of this or they much more involved it seems like they're the ones that identify the initial need to move or shift some data around and do they really want to engage with it and Engineers or they want to do it themselves.

Usually they ask the data team to do that. So mostly we've been interacting with data teams and the only part where we are so we have another audience will be Engineers for database replication. Usually it's Engineers handling database of this product databases that they need to replicate either in the warehouse or another database.

So there will be the injuring team or the data team the data Engineers within the data setting we've certainly talked to like marketing or or sales who will eventually like use the analytics that is built on top of the data. But usually they just those teams interact with their own data team and the data team is the one shooting the tool. What's your sense of?

As we go along here. Do people have a greater appreciation for the value of data in these days because it seems like for a long time we kind of treated it as a burden and now you know everybody we talked to says you know data is the new oil but it doesn't seem to me like they have any real way of refining that data. So it kind of just sits in the ground as it were um, are we getting better it kind of using the data in and intelligent way.

I think we've identified data as an opportunity for more efficiency for more impact in marketing or sales company like and so to remain competitive you need to leverage your data. That's like the trend we've seen and in the economical context as today that makes it even more important. So for instance, if you can say to I can provide the data of support data marketing data on on Salesforce directly to the sales team that is very valuable to the sales team.

And so with the warehouse that is made this possible. Okay. So what's your thought process about?

What is the role of data in terms of the organization and we saw the rise of Chief data officers for a while but is that changing in any way? I mean, I'm asking the question because if data goes mainstream do I need a data leader or you know does everybody have a greater appreciation for data now and the mission has been accomplished. Yes.

Exactly. That's why we've seen we've seen like data becoming a first class season one more. So for instance, our typically is a buyer for everybody will be either the head of injury or the head of data at that point often times.

We will see data report to injuring but sometimes it's his we have a chief data officer. This is something we see more. Okay, cool.

Alright guys, so if we're currently in a world where we're thinking about Data having this kind of level of value. Do we have too much data is there too much duplicate data? Do we need to clean up the data?

Because it feels like there's not a really good sense of how much data do we have and every organization seems to have data that's inconsistent and For that matter not just duplicate triplicate and quadruplicate. That is a very good question. I think it comes down to the quality of your data team in the end.

Like so the whole goal of the what we call the modern data stack I think and that's another topic but I think it will evolve a lot in the next five years is really automating so that you can import the data that is important. Centralize it transform it in the automated way so that then it is ready to be to be analyzed or to be activated in other tool but there's strong value in that integration of all these stores. So that's all automated and the end you don't have that much manual work to be done.

And so you can really leverage the data as well as you could okay, and it's all Maximum potential. Now the big issue I guess is that most of the tools priced on volume or compute. So the more data you have the price here it is and so the data budget is increasing or not.

Definitely. So that's another topic. Right.

Well, you heard it here folks. Dana is both hopefully an asset, but we got to figure out a way to manage it in a smart way than we currently do John. Thanks for being on the show.

Thank you very much. All right back to you guys in the studio.