Scale, Production and Expertise – Robert Nishihara, Anyscale

This is texturing TV. Hi everyone. Welcome back to techstrom TV.

I've got a really exciting a new person to introduce. Well, he found a new person. He's just new to our audience.

I really a new a new guest here on techstrong TV. I'm excited to have them on and and hear his story and talk with them a bit. Let me introduce you to rob it and shihara Robert is the CEO founder of a company called any scale.

Is that correct? Robert, right? That's right.

Fantastic Robert. I really want to start with your own personal story if it's okay if you wouldn't mind sharing with the audience and we'll go from there. Absolutely.

So I've always been interested in AI I did a PhD in machine learning before we started this company and actually, you know where the starting point for me was really trying to design better algorithms for machine learning training for deep learning for optimization much more on the algorithmic side. But what happened over time? Was that in order to actually try out the algorithms that we were designing machine learning is very compute intensive and we needed to actually scale these algorithms across many machines and to do that.

There was a ton of just software engineering work of expertise of infrastructure work that had to be done to to actually build the infrastructure for scaling the machine learning algorithms. And that's where we ended up spending. All of our time is like, you know, 10% of our time on the actual machine learning part, which is what we thought was our expertise and 90% of the time on building the tooling and infrastructure for actually running machine learning so that kind of and it wasn't just us, right all of our friends who are doing machine learning we're running into the same challenges.

And so we started you know, some of my you know co-founders and I'm of course it wasn't a company at the time it was we were grad students and and people around UC Berkeley but we started building open source tools for trying to make it easy to build scalable applications scalable python application scalable machine learning applications. And that is you know, that's the project that became Ray the open source project that we're building and you know, so that's how we got started. And of course as I can say a lot more about that, but as Ray was taking off as more and more, you know started with other people around Berkeley using it but as it grew to more and more companies adopting Ray, you know, and that's when that led us to actually start any scale to really commercialize rates try to make it easy for developers to build these and scale these AI applications without actually having to be experts in infrastructure.

So that's sort of how it all. Got started got it love it. And and we should mention you mentioned Europe PA.

You were a PhD student at Berkeley. I you that's right graduated. I've been awarded a PhD also.

I think had a master's degree from Harvard. I did I was an undergrad at Harvard. Yeah, underground Harvard.

Okay, so not you had a little education not a bad thing. So but I I would I would pause it that it wasn't just you and your friends in Berkeley who are having these scale issues scale issues. Scale is hard.

Hyperscale is harder, right and next week. I'm in Detroit for a kubecon cloud native con and and you know. two projects coming out of there Prometheus, which you probably are familiar with and And open Telemetry not necessarily AI but you're around this whole observability Space.

Yeah, the scale issue was just it was so hard. And and I guess that's sort of a path in today's world. Look for these hard scalable things.

Let's start with an open source project where we can get co-opetition if you will. Yeah, but absolutely Trump on it and and these are you know, the products you mentioned are very successful and you know, it's they're actually quite complimentary with what we're doing because if you were building You know, if you're building these large-scale applications and deploying your machine learning models a big challenge is not just developing the application but actually debugging is something goes wrong. You know, it's actually knowing how do you even know if something went wrong and that's and so observability and these kinds of this kind of tooling plays a huge role and so it's very complementary with what we're doing to actually do machine learning successfully and you know, there are many companies are at least number of companies that get a lot of value out of machine learning.

There are many pieces to the picture right? Of course, there's the model training and deployment there's the observability tooling there. There's things like model Registries for storing and versioning your models, you know, there's experiment tracking there's feature stores.

There's all these different pieces right just to sort of describe how we fit into that ecosystem. The the part that we are focused on is really the compute piece like managing and scaling the actual compute for things like training or the data ingest and pre-processing the deployment so actually running the machine learning and then all of these surrounding pieces for the metadata and everything we integrate we try to integrate with all the other tools out there. So things like, you know, open Telemetry or Prometheus that you mentioned sure.

So Robert, I want to talk raid any skill. But before we do yet you brought it up and I I want to bring I I'd love your opinion on this. Yeah.

There are a lot of people out there who? Who have kind of dipped their toes into the machine learning stream? And it was a little too cold for them, right?

They didn't get the results. They were expecting or hoping for. I think one of the issues is is the the just the very term machine learning AI.

Are they interchangeable? Are they the same? What's the what's the difference?

Yeah, you're the you're an expert. Well, how do you kind of delineate? Right.

So I guess when I'm just speaking I do use them interchangeably, but I think of machine learning, you know, I think of Machine learning more as the the technique to achieve AI or one of the approaches for building Ai and AI is a bit more of the kind of results or the, you know, the description of the properties of the thing you're creating and machine learning is a little more like one approach to creating intelligent machines. Got it. And if you could imagine having other approaches to designing and creating a programs that have intelligent Behavior other than learning for data other than learning from data.

I I think I think that learning from data is the easy is the lower hanging fruit on the AI tree, right? I think. look, I for some people some people say not so much right?

They're not and actually that's a good point because I want to say a lot of like certainly companies like Google, you know, or meta or companies like that, you know Uber excuse Ai and just all their products and get significant value out of AI but but it is extremely hard and many AI projects fail right many companies try out Ai and you know have the intuition that they should be able to get value out of making better predictions or you know, or things like that, but you know, it's very hard to do in practice and there are a lot of or they're a lot of ways things can go wrong. It could be, you know, I mean, it's possible the date is not in place say maybe the data isn't isn't good enough. It's possible that's you're able to prototype it and this is actually one stumbling block that we see quite a lot is, you know, you are able to prototype things and get started but then the shift from development and prototyping to to production is just A completely different story.

It's a different Tech stack. It's a different team. It's it's like scaling things and and that's you know, that's one one place worth a lot of things fail as well.

There's and there's maintaining it in production and all of the other, you know, tooling around. You know, what we talked about with observability and monitoring and so on. today would you scale is probably one of the hardest thing because it seems to me you can't really have great machine Learning Without a certain scale.

right, and and so if and then you getting to that scale and then managing. that scale to extract You know to extract real value. In in you know in machine learning, I think that's where people stumble right in the lab in a smaller kind of.

com eight or nine years ago. It's great for the hyperscalers. The Facebook's the Google's of the world the twitters of the world.

It doesn't work for you know, those were unicorns back then they called those unicorns but it doesn't work for horses, right? That's what that was the same. Are.

We seeing the same thing with AIML kind of stuff. It's great for you know horns. I think not horses.

Well, it will be good for everybody. It will at least I am hopeful that we'll get to the point where AI is transformative for every business and every industry. Now if you're talking about today, yes, a lot of a lot of companies.

It is quite hard to do Ai and to get value out of AI and scaling certainly that's where you know, I was stumbling when we were we were doing Um, and it's you know, tons of people are learning to program in Python, right? Lots of people are taking introductory computer science classes or learning python. And so getting started on your laptop and and you know copying and pasting an exam pytorch example or running your training your first machine learning model like that's doable but going from programming on your laptop to now running scaling these applications across clusters.

It's a You know, there's so many new challenges. You have to think about it's how do you divide the work across the different machines? How do you handle if one of the machines fail so if you need it to be elastic, how do you scale up and down or how do you there's a scheduling challenge is how do you move data efficiently between machines all of these kinds of things, you know are super hard distributed systems problems.

And this is why companies that do machine learning often have very large infrastructure teams. The machine learning infrastructure teams to actually build the systems to do this. And of course, this is we're trying to get to the point with with Ray and any scale where all you need to know is python right where if you know how to program in Python on your laptop, you took that introductory computer science course, that's enough you can you can build the same kind of scalable applications that you know a Google can build But you don't have to become an expert in distributed systems.

So now I'm not saying we've 100% gotten all the way there yet. But that's what we're it's that's what that that's no that the great goal too Robert. Let me let me turn really any skilled race specific here.

You know what? I I I've been involved in open. Communities for a really long time 20 plus years 25 plus years.

time You know people doubted open source business models, right? How can you build a commercial entity on top of an open source product Red Hat was the shining example, but it was few and far between that's not true anymore today. We know.

There are open source business models that are very successful. There's tons of companies. They're good data risks, right?

Yeah data breaks is one of many you go to cubecon and Detroit next week. You'll see tons of them, right? So I'm already public some, you know, very well financed Etc unicorns.

Talk to us about the relationship of Ray in any scale. Obviously. Any skill, you know?

Kind of built on Ray if you will, but it adds functionality that the open source. only project doesn't have what what the differences so first of all Ray is a open source project. It's a python library is basically a framework for an open source framework for scaling compute right for scaling python applications and machine learning applications.

Okay. So that's all over Source any scale is a managed Ray service. So basically we run Ray for you in the cloud basically, you know, so a lot of people just don't want to be in the business of managing infrastructure, you know, and and if we can do that for them, you just click a button or have some simple API call and you have all the ray clusters you need you can run your applications.

The Clusters will scale up and scale down and and you know All of your outputs will be the results will be persisted and shareable and you know, you have all of the tooling and dashboards around that then that's you know, that's what we're giving you. So one is just the managed nature of the product. So it's a stats.

Yeah, really if you will. Yes trying to get to the point where developers just focus on their python just their application logic a lot of what we're building in any scale is around really trying to remove friction from a lot of from the developer experience. So these are things like, you know, better tooling around debugging and observability, right if something went wrong do I have the relevant logs and dashboards and and metrics to understand what went wrong right A lot of it is also around simplifying in any scale simplifying Um movement moving from developments and production.

There are many companies that say once they develop a model it takes them six weeks or 12 weeks or some, you know, huge amount of time to actually get it in production and there are different reasons for this but one is that you often have to rewrite a lot of code and and it's a different use different tools and hand it off to a different team and along with any scale. You can really go quite quickly from development to production using the same kind of environment same kind of tooling. So those are some of the values now a lot of Of course a lot of the the value that we are working on as well is around the actual infrastructure.

So, you know making Rey run fast and making more performance right making it cheaper with like great support for preemptible instances spot instances things like that. So those are some of the directions we've been going in. Love it.

Now Dad it's time for people. about Any sale and Ray we're can you give them websites to go to? com, you know, you can try it out.

And if you Google or radon IO is also great place to you know, check out the right documentation learn about Ray. com slash Ray Dash project slash Ray. Okay, you just had our first in-person race Summit a couple months ago.

You're in San Francisco. Yeah, and you know had a bunch of people and there's a lot of recorded talks from from this event. So you can hear people, you know Uber talking about how they run all of their deep learning on Ray or open AI talking about how they train their largest models using Ray or Shopify, you know, they built their entire machine learning platform using Ray.

It's the car trains tens of thousands of machine learning models using Ray and was able to you know, reduce the training time from Days down to hours. So incredible number of very cool use cases absolutely. Sounds great.

Hey Robert first of all congratul Patience on everything you've done here from the education to to you know, putting this together with Ray and now any scale. Come back. Keep us posted on this.

This is you know something I think it's really cool in our audience finds very very interesting. So we'd love to have you back on. Thank you so much.

Really? Enjoy the conversation. All right, Robert nishahara CEO founder of any scale and also the folks behind Ray hopefully making ML and AI much more.

Successful for you out there. We're going to take a break. We'll be right back.