Data Management and Observability: Hydrolix’s Catherine Johnson

Hey everyone. Welcome back here to Tech Drunk tv. I'm really happy to have our next guest on because, well, I love to talk football anyway, but we're gonna talk about more than football.

Let me introduce you to Katherine Johnson. Katherine is the field CTO of a company called Hydraulics. Katherine, welcome to Textron tv.

It's great to have you on here. Thank you. It's great to be here.

Absolutely. Um, I just gotta ask, that's a really nice room you're in. It doesn't look like one of these fake backgrounds either, so It is not, this is my actual office.

I like it. I like it. It looks like a nice place to work.

Um, speaking of which, Catherine, tell us a little bit about your work, how you got to be field, CTO, a little bit of your journey. Yeah, sure. So I have been, uh, doing stuff in data, working for data vendors for about 25 years now.

Um, I started working with, uh, high speed in memory data grids in the early two thousands, so think like trading platforms. Um, spent some time at Oracle doing more integration work. Uh, ended up back in high speed computing at VMware and Pivotal.

Um, so again, working with trading companies, uh, things like the Brazilian stock market, for example. So being able to recalculate risk really, really quickly on lots of fast data coming in. Um, from there, uh, I had a good friend who was one of the first employees at Elastic.

Um, he convinced me to come over to Elastic, and that's where I really got into observability and mm-hmm. The thing that was interesting there, elastic started as a search platform. Yes, he did.

Um, Yeah, but the, the bigger, larger, more interesting deals were around observability data. So if you think about something like Wikipedia, that was like 10 nodes when we used to talk nodes of Elasticsearch versus somebody who's doing a really big observability platform would be hundreds if not thousands. And so what we were seeing at Elastic was that just kept getting bigger and bigger, and it was getting to the point where it was kind of untenable.

Like we would talk to customers who would say things like, I have 135 terabytes of data that I need to analyze every day. And we were, you know, it's like, we're not really sure how we're gonna do that and not even how we're going to do it. But if you just added up the cost of running something like that, you needed to be really clear where the value was coming in, that expense you were gonna pay, it was going to be millions of dollars.

And if you didn't know what you were looking for in that data, right? That's, that's not great. That's a lot of money.

That's A lot of money. Um, yeah. So from the, go ahead.

Go ahead. No, no. I was gonna say from, you know, to me that was always a lesson of just because you can, doesn't mean you should, right?

We, we tried so hard to be able to get our ha hands around that big data issue. Mm-hmm. And then when we started to all of a sudden, like for instance, I remember this with Splunk in security, like we want in security, we always wanted to know everything, right?

Yeah. Because we can never tell how far back we had to go and when the breach happened, but who had the money to pay for that? And so you had to start making trade-offs, and some of them weren't always easy, but I I I digress.

Go ahead with your story. Yeah. Well, it, it's, it's part of how hydraulics came into being.

Um, so I, I spent time at different observability companies at Grafana, at New Relic, and while I was at New Relic, uh, Marty, our CEO had looked up, he's in Portland, Oregon. I'm in Portland, Oregon. He was looking for, you know, people that had this background.

He's like, you popped up and you're in Portland. And so he came with this story of this database he wanted to build. That sounded way too good to be true.

It was like, we can store all the data in S3, we get this great compression. You get, uh, you know, real time response, like human real time response in terms of query times. And I, my brain was just like that.

That's not possible. That, that sounds great. That's not possible.

And it's another database. There's so many databases out there and different ways that people are already looking at data. So it's really hard to sell into that, right?

Like, why do I need to buy another database if I already have 10? So, fast forward a couple years, um, I already started the company. Uh, I came on really early after he had hired a couple of my friends who are engineers.

And so one of them was someone I had gone to grad school with a long time ago. Uh, and they were both like, no, it works like it, it's real. This, this isn't like some overstated marketing shtick.

Like it is very real. Um, so I came on board, uh, I left briefly did some of my own stuff, but, um, when I came back, I came back in as field CTO because of my depth of knowledge in both the hydraulics product and, uh, observability. But the thing that was unique about how Hydraulics was Marty's background and Hassan, his co-founder, were in massive observability data.

So application logs and things like you mentioned Splunk, like that data keeps getting bigger and bigger and bigger. The world of CDN and media, the observability data, and that was even larger by orders of magnitude, right? So if you're a media, you already know that.

And so that, while that was our first use case, what we really went after is how do you all the things that you just said, how do I keep my data without it costing me a ton, without having the trade-offs of am I sampling? Am I throwing stuff away? Like what am I giving up to keep all of this data?

Like how do we give the ability of somebody to keep all that raw data at a price that's reasonable and be able to query it quickly in, in real time, like human real time? So that's what Hydraulics is really about, is taking massive amounts of data, storing it compressed, which is one of the things that we're really, really good at, compressed in a way that it's not costing you a ton of money and you're able to query it without it being hard, without having to rehydrate it, without having to take a long period of time. Um, so that's what we really do is we're a massively scalable database built on stateless microservices, um, with using object storage, which is the cheapest storage available as the backend.

So you clearly consider yourselves a database company, not an observability company. We're both, um, so okay, That's why I asked. So our first product market fit that we found was in CDN observability.

Um, and observability data is really the largest data set that's out there right now. You can use this for any data problem you have that has large data associated with it. It just so happens that that observability data does tend to be a large bulk, right?

io, and hydraulics is spelled H-Y-D-R-O-L-I-X. Excellent. Now, in your role is Field CTO, Catherine, what do you, you're out on the road talking to customers?

I talk to customers and a lot of them are the more complex use cases. So something where we haven't done that before. The customer has not been able to solve this problem before.

Um, so massive observability use cases for, you know, a lot of live streaming events, which we're going to talk about, um, and are really important customers. So where is it that we, we know that we really need to succeed in order to gain more traction as a company. So those are the places where I tend to get involved.

Excellent. Alright. Let's, you mentioned massive sporting, live sporting events.

None, you know, the granddaddy of live sporting events. Well at least in the US maybe the World Cup and the rest of the world. Mm-hmm.

But here in the US the Super Bowl, is it? Yeah. Right.

Doesn't get any bigger. Hydraulics had the, I don't know if I'd call it the privilege, but the opportunity to work on the, uh, super Bowl this past year. Why don't you give us, set it, you know, set the table.

Tell us what, what the story is here, Catherine. So we had done the Super Bowl the prior year with Paramount. We had done the observability for all their CD, CDN data coming in.

And what that means is every single request that viewers are making, so you don't down, you don't get the whole video at once, right? It's not finished yet. So it's coming down in segments.

So it, it was about gathering all of the raw data about all of the users and all of the segments that were being downloaded to view in order to get to, in order to maximize the customer experience, right? So we, or a lot of us saw the Mike Tyson fight a couple of months ago where, you know, there was a lot of buffering or, you know, things were coming late or you couldn't see the video at all. And so for companies that were streaming Super Bowl is really important to them that customers have a fantastic viewing experience, that there weren't complaints afterwards.

You didn't see any negative social media on Twitter or any of the other platforms about, you know, how terrible the quality of the stream was. So for Super Bowl last year, uh, that was with Fox and we started from a place with them of, yeah, we've tried to do this before and we can't, like we've, we've broken everything that we've tried in order to do this level of observability. So we started with a proof of concept with them, uh, and this was nine months before the game, right?

So it, it had, we had to sort of build up to it. So we started by looking at the data, um, and understanding what data sources they were going to have. Uh, we started building that part out, um, because we needed to normalize the data we needed, you know, if it's coming from a bunch of different sources, you wanna be able to query across them.

And so that means like naming things the same, you know, normalizing units of things. Are you talking bytes? Are you talking seconds, milliseconds, being able to normalize all of that.

I think it was around August last year, they said, Hey, things are changing a little bit and we want to live stream this through Tubi. And Tubi doesn't have a paywall. Um, it does have a login wall, but it doesn't have a paywall.

So suddenly, like you don't know how big that traffic is, right? Like, you, you have no idea. And depending on how the game goes, that can change really dramatically, right?

So we were told things I'm, I don't watch a ton of football, but it was, depends on where things are at halftime as to what the viewership is going to be in the second half. Makes sense. Yep.

Um, so we started building from this perspective of, we don't know how big this is, right? We, we have some guesses. Uh, so we had built out and, and assumed that the traffic was going to be three times as large as what it, it eventually ended up being.

Um, but it's really hard to scale test that, right? It, it's really hard to generate traffic that would be representative of global traffic, really spiky, sporadic requests coming in. Um, so we had, there were several games that we monitored for them leading up to Super Bowl.

And the two that were really the big tests were the playoff games and so on. Both during both of the playoff games were like the first real live scale tests that we had, uh, of the platform. So during the first playoff game, everything went great.

No major problems. During the second playoff game, however, things changed quite a bit for us. And one of the, uh, one of the things that changed was, um, how their paths were constructed.

It sounds kind of like a detail, uh, but they were embedding, uh, basically user level information into the paths that, uh, customers were requesting. So to get like a, you know, one of the game segments as you're watching it without all this information embedded in it. And that really changed things for us because they wanted to query on that data that was embedded in the string at really high volumes that is hard to do.

So if you're talking about like searching in a string over a couple terabytes, no big deal, you're talking a couple hundred terabytes or petabytes string matching suddenly big deal. Big deal, really big deal. Um, so after that game, we made changes in order to accommodate those queries.

And so a lot of the work from the second playoff game until the Super Bowl was about what are all the things that we can anticipate that might change during that game that we hadn't already thought about? We got a hint from this change between the two playoff games. Um, so the next several weeks we're really focused on how do we optimize as much as possible for all the things we think might happen.

Um, and then we also implemented some things on our side. Uh, so we have the ability to have different, um, sets of compute dedicated for queries. So we set up a query pool that was just for the operational folks.

We set up a query pool that was for the executive dashboard so that the operational queries weren't, uh, contending with the, what the execs were looking at. So we all know like that that's the most important thing, that's what they're looking at. Sure.

And then the final one was for ad hoc queries. So anything that would go wrong during the game, we needed a separate set of query compute so we could run, you know, gnarly queries basically without impacting either of those other two groups that still had, you know, things that they needed to observe. Got it.

Try now the, the playoff games are, I'm just trying to, if I read, you know, I remember, 'cause I always read the articles about audience size and everything. The playoff games are like a quarter of the size of the audience that the Super Bowl is, if I'm not mistaken. Right.

Something like that. And, um, I mean, I guess the question is, look, it's a tremendous engineering feat, no doubt about it. I mean, the scalability here is truly scale and they must find tremendous value in it if they're doing it year in and year out.

Yes. As we look to, I don't know if it's too early, but looking the next year now, right? What changes had, does AI help for instance, or I mean, what, you know, how, how do we improve on this?

Yeah, so the, the way that our platform is built, we depend so much we built for the cloud. So we depend so much on the scalability of the object storage, of the compute that's available in whichever cloud we're in. And so what we had seen to date before the Super Bowl was we hadn't really, and truly we haven't, still haven't, we haven't found like a top end limitation other than the underlying storage limitations of the amount of data we can't handle or the amount we, we just haven't hit that in terms of cardinality, in terms of volume.

What we did this past year for Super Bowl, which is sort of like a prep for the next year, is I mentioned we had to do, we anticipated three x the traffic, right? So we went ahead this past year and were able to deploy, uh, our technology in a multi-region way in order to make sure that we weren't saturating any given region. This is running in AWS we weren't saturating the compute in any given region, so we did a lot of work up front before the game to prepare for a much larger load than what we had.

So that's already in place. The things that are going to be additional are, like you mentioned, um, we started offering, uh, an MCP server with our technology. So there's ways to integrate and do more natural language types of queries.

Um, we're looking at, uh, supporting other query languages as well. We've already been doing that. So for example, you can, uh, query our data from Splunk, from Spark, from, uh, elastic search query languages.

So if there's more real time analytics are going into it, we can look at stuff like using Spark, like for larger amounts of the data to do that analytics in real time and look for anomalies. Um, if somebody is already using Splunk and they're used to that tooling, we can plug directly into that as well. So we're really looking at how do we make, for us the magic in what we do is the statelessness, uh, the cloud, the leverage of the cloud technologies.

And then we are really good at compressing that data down, uh, 20 to 50 XA lot of times. Um, and that's part of our secret sauce. Uh, so all of that combined together is, you know, we'll, we'll help us move forward here.

So we're looking at, we focus on our secret sauce basically, and then we try to figure out how do we plug into things that people are already using, either the pipelines are using to get data in or the things that they're already using to query. So for us, the expansion is really in supporting more tools around the edge. Um, we're feeling really solid about our scalability and our ability to handle those bigger and bigger loads.

Got it. I love it. I love it.

Um, you don't get tickets to the Super Bowl for this, do you? No, I, I was actually a little disappointed because they're like, we're gonna need you at Super Bowl, and I'm from New Orleans, and I was like, yes, yes. And, uh, no, it, uh, they box had a, a war room big amphitheater set up at their headquarters in, uh, a Tempe, Arizona, and they had all of the different vendors that were involved in delivering super in the room together.

Um, so everybody's sitting in one room. We had, uh, two days of like, you know, basically game days, right? So we did dry runs, um, Fox went through, tested all the, you know, possible things that they could think of that could go wrong, made sure that we could see all the way through.

They tested things like, you know, being able to switch between CDNs. Um, so we had two full days of practice before the game itself. Very cool.

Very cool. Now, next year you mentioned Tubey was this year, I think I actually watched it on two B this year. Next year is AWS is the partner for it.

Is that the story or? Uh, it's actually whoever the, uh, the whoever's delivering the Super Bowl that year. So, uh, 2024 was paramount.

This past year was Fox, I believe this NBC Oh, peacock. Yeah, sure. And I, if I'm not mistaken, P Peacock might also work with Pluto.

You know, all of the, the relationships on the backend among the streaming providers now are just crazy. Anyway. Well, We, we did do, we've already been working with NBC, so we helped them deliver the Olympics and uh, yeah, That's a big one too.

Absolutely. Catherine, what a great niche and you know, to, to, it must be fun for you, but I'm sure you have other customers you talk to, but ke keep up the great work at Hydraulics and, and come back and keep us posted here. I mean, you know, the scale and then their scale.

Yeah. Yes, that, that's true. Absolutely.

Catherine Johnson Field, CTO Hydraulics. io you said? Right.

Sweet. All right. You're watching Text Drunk tv.

We'll be right back.