Groq Cloud is Changing the Rules of the Game in Generative AI | The Six Five Summit

Welcome back to the six five Summit. I'm Dave Nicholson and I've got a special guest, sunny Mara, general manager of Grok Cloud. Sunny, welcome.

How are you? Awesome. Great to be here.

Thanks for having me. Tell us a little about Grok, for the uninitiated of the uninitiated, for folks who, um, I can't imagine who doesn't know about Grok, but just give us a little bit about Grock and what, where Grock came from, what Grock does. Yeah.

Uh, really actually, great background. Grock is a, uh, company that was started by Jonathan Ross, who was the founder and adventure of the TPU at Google. He left Google to create a similar AI chip for the rest of the world.

And what Grock is today is a fully vertically integrated stack that, uh, you know, offers tokens in the world of generative AI to developers and enterprises, uh, across the world. And, you know, we've really been able to ena people by, you know, the following things are the speed at which we generate tokens, the low latency, which we do it at, and then the cost at we can do it at. So that's what's been so exciting with Grok this year.

So that full stack includes, uh, hardware acceleration, right? Yeah. And as well as full stack that developers can take advantage of.

What's the, if, if, if it's not a TPU, what do you refer to the grok hardware accelerator apps? Yep. We, we call our chip the LPU, the language processing unit.

And what's interesting is obviously that, um, you know, makes sense to a lot of people for LLMs, but what people don't always appreciate is that the, you know, the LPU, uh, is actually able to do all different types of models. So it's not limited to just large language models. And that's really exciting.

I mean, uh, if I understand correctly, it's all about the way the math is being done on the device and uh, and All that. It, it really is all AI is just math underneath, right? And so it's, it's a chip that's very, very good at doing math quickly.

So you've come to gr to, uh, to build the Grok Cloud where people can access LPs as a service. Is that fair to say? Tell, tell us about grok Cloud, what you're doing now.

Yeah, and maybe I'll give you a little bit of my background 'cause it'll, it'll help folks that are listening to this here, which is, so I've been a serial entrepreneur and I've built and sold a couple companies. And I was working on my third company when, you know, I connected with Jonathan, who I've known, you know, since he had left Google. And really what I saw was the company had this incredible technology, which was built for, you know, uh, all different types of use cases, but specifically for the cloud, which they had talked about and thought about before it really had an opportunity to break out.

And so our skills in basically building cloud companies before combined with the great technology that GRA had built, brought us together to bring, you know, gra GR cloud. And so our group runs GRA Cloud. com, and it's a serverless, API for AI inference.

So what you're able to do is you're able to go there, log in, get an API key, and you can basically start doing inference right away on a, on an API that API, we have our own SDK for, it's also OpenAI compatible. So it's really easy for developers that were maybe using OpenAI and want better performance or lower cost to use our inference engine to generate tokens with. So talk about the growth of Grok Cloud, but more importantly, the growth of the developer community, because that's been pretty staggering in terms of the ramp up that you guys have experienced.

It. It has been, and you know, look, uh, the growth of GRA cloud and the growth of developer community is one and the same in that, you know, we did a soft launch in the middle of February. In that soft launch we went viral.

And the reason we went viral is people were amazed by this great technology that was finally available for them to use on their own in a self-service capacity. You know, since then we've had 225,000 developers sign up. So just in a short amount of time, 12 weeks, why they're really coming to the platform is developers are hungry for low cost tokens, and that's because they wanna make this new class of applications, but they want the tokens to be low cost and they want them to be low, uh, latency.

'cause they want applications that have that same, you know, look and feel of the best applications either on your phone or on the internet. And if you think about inference today outside of rock, you go to, you know, any, any chat site, it sort of streams at you like it's a dial up connection versus if you go to, you know, Google and do a web search, it's instantaneous, right? And a lot of work happens there to generate those results quickly, or an application.

So what we really brought to developers was a low cost experience, which is always important for them, but also a low latency and high throughput experience so that they can create those same type of, um, user interfaces and interactions, um, that for their users that they're used to experiencing in, you know, this traditional web applications. And, you know, you sort of alluded to this idea of speed, uh, contributing to low latency. Uh, can we double click on that a little bit because I, I think it's important for people to understand that there are significant architectural differences between what you do with your hardware acceleration, and then out of that, the sort of stack that developers are interacting with.

There's a lot of misconception in the marketplace, I would say that Nvidia in particular has locked up the developer community with PDA stuff. Yeah. Uh, can you, can you speak to that if you know Yeah.

An educated developer who's saying, I want the most bang for my buck, the lowest latency I can get, and I want to participate in this development community. What, why are they, why are they so drawn to rock? Yeah, it's, that's a really good question.

So firstly, you know, we should give credit to Nvidia, right? Uh, creating an incredible business and Kuda, which is incredible technology. The real difference that's emerging now is Kuda is not for the everyday developer.

Kuda is for the developer that's making large language models, right? And they want to basically create performance gains with them by leveraging, you know, cuda kernel specifics. What, what's really happening now as generative AI continues to mature, developers are not making models.

De developers are consuming models. Those models get hidden behind an API and so they're not interacting at all with the specifics of whether it's Nvidia, you know, GR or a MD that's running those models because it's been abstracted. So think about it, in, in the cloud, when you go to AWS and you ask for a machine, you're not really caring about whether that machine is run on an Intel CPU or an A-M-D-C-P-U, you're looking for a certain compute unit to complete your task more.

So if you go to Amazon and you're using S3, you don't really know what type of hard disks are sitting underneath that storing your information. So we're seeing the same thing happen in generative AI where developers know the model they want, but they're less concerned with like, what the underlying infrastructure is. And so by taking all the infrastructure that Grok had been building for seven years and abstracting it behind a simple API, it's really allowed developers to experience the power of grok in the same way that they were experiencing it with NVIDIA based or a A MD based solutions on other clouds.

And so that's, that's why the community's been so excited. Now, why is speed, uh, you know, as in throughput or latency as in timing? So important goes back to our previous, every a hundred milliseconds that you shave off an internet search leads to hundreds of millions of dollars of extra revenue.

And in the world that we live in today, when we're used to such short attention spans, we're used to being quick, we're used to being able to quickly go through content. You know, we are not on diop internet anymore, right? We have high speed at home.

We use, we get the latest 5G phones. We do all of that in the pursuit of speed. And so we basically bring that to generative AI with our technologies.

Would you say that you exclusively offer a value proposition for inference, uh, or is it a mix of inference and training? How, how do you come down on that? And then, and then under the heading of training, of course there is the foreseen million GPU model that's going to be trained, um, as opposed to No, no, no, no, no.

I have my own data in my company that I wanna train my model on. So how does, how does cloud, how does a brought cloud fit into that model? What's the best use case?

Yeah, Yeah. Look, what, what I, what I'll say is when it comes to training, uh, NVIDIA's best in class and still is, and they've done a lot of work and, and honestly, their developer ecosystem, specifically the ones around the ones that are making models, they've done a great job to support those folks. And so, you know, we focus on inference and we focus on inference because we understand that that part of the market will be much larger than training.

So if you think about it, you know, a model will get trained once, maybe over a course of several months, but get used several, you know, hundreds of millions if not billions of times, uh, by people. And so we focused on what we believe will ultimately become the bigger part of the market. And so that's, um, that's sort of the, the focus aspect.

Um, in terms of, you know, your, your second question, which was around, um, you know, the developer speed, the around the, sorry, data and training, right? Um, when it's doing inference, right, we don't really focus on taking any data and that makes it easier for, for developers to con, you know, work with us. 'cause we're not really, um, taking data and using it for training and potentially, um, you know, consuming it.

Uh, in fact, we don't store any data on our rash machine. Anything that comes through and it from an inference perspective is completely passed through. And we've done that on purpose because we really wanna give folks the comfort and understanding that we, uh, are not participating in any type of data capture exercises or data retention exercises.

Got it. So yeah, it sort of puts an exclamation point on the difference between inference and training, because when you say performance, when you say speed training, it could be how long will it take to train this model inference? It's in how many milliseconds can I get a response?

That's Exactly It. You, you gave, you gave a a a Google example. It's a, it's a classic example.

We've all been using that predictive text. If it can't predict the next word before you can figure it out, it's useless. Yep.

So we may be okay with the sort of experimental phase we're in right now as consumers playing around with generative AI and waiting for four or five seconds, or 10 seconds or 30 seconds. It almost seems kind of quaint. It's kind of fun.

Yeah. But in the real world, you've gotta have the kind of speed that you deliver. Well, that's exactly it.

And, and you nailed it. And I think that experience is what's starting to emerge with folks that are making applications. And what, what also happening is, when you think of agentic use cases, so like a, you know, agent based AI that are doing multiple tasks for you, not a single shot question and answer, those use cases typically can take several minutes because there's a lot of back and forth that the AI is doing with whatever service it's interacting with.

So think about like asking an AI agent to book a ticket for you. It's gonna have several interactions back and forth. So if those things can get sped up, it can be the difference from, uh, uh, like a use case taking four or five minutes going down to 10 seconds.

So, um, not to be cynical, but basically what you've done is you, you said, Hey, you know what a very specific task with specific requirements, let's build something that does that thing really, really well. That's it, it what a novel, what a novel concept. But, but on the, on the, on the, you know, back to the kind of hardware acceleration part of it, um, I understand that there are unique aspects to the decisions you've made about how you build these things and what your supply chain looks like.

Uh, is that relevant from your perspective, uh, in the, in the cloud conversation? Or is that something people should, uh, should, should look up online? No, it, it is, and, you know, we, we can touch on it quickly.

I think there's the following aspects. I think one, you know, we've built a chip that is, um, does not use any external memory. So there's no HBM.

So from a supply chain standpoint, you know, HBM is sold out, you know, well into a, a year in advance, right? And so if you were trying to build something and you didn't already have access to supply, it'd be very hard for you. Second are actually, chips are a 14 nanometer process, which is, you know, four generations old, right?

And I think because of that one, it shows sort of the in incredible work that our engineers did to create such a powerful chip that's competitive with something, you know, chips that are four generations newer, but more importantly, it really unlocks supply chain for us. The, you know, the fab capacity at 14 nanometer is easily available compared to the fab capacity at the latest technologies. And so, and then lastly, which we're something we're really proud about, our chips are fab in North America, right?

In fact in upstate New York. And so being able to do that, we really don't have to really get at risk of any kind of supply chain constraints that can happen over shipping or conflicts or anything like that, that can happen. So that really gives us the ability to scale our cloud, um, really, really fast compared to, you say anybody else that would try to do it with some of the supply chain constraints.

Yeah. Some would say, oh, Dave, those things don't matter. It's a service.

You shouldn't share what's happening behind the scenes. But it's, it's so important. I wanted to ask you about that.

Well, sunny, finally, um, what do you think is coming down the line in terms of the future of generative AI in particular? Anything that excites you specifically personally or professionally? Yeah, No, there, there are, and, and there's like the following things which I, I I'm very excited by, I think first is multimodal ai.

And, you know, we started to see that with GPT-4 oh and seeing that in the open source community, which I'm pretty confident we'll see that in the next 90 days, uh, is gonna be really powerful. So that's where multimodal means. You can interact with the same model using voice images or text, and I think that's gonna be incredibly powerful.

The second thing that I'm super excited by is, um, these agentic use cases. And why I'm, why I'm excited by that is today we're living in the era where we're taking human tasks and replacing them with a single AI driven task. And what I really think about is sort of what happened in the industrial revolution.

We, we went from say bespoke car making one car at a time to factories that made cars. And now what we can do is we can basically say, instead of replacing a single task with an agent, why don't I have a hundred agents look at my trip options and then have another agent look at the results they come up with, because that's way more powerful. So we're kind of going to the industrial revolution of technology.

We haven't seen that before. So that's super exciting. And then lastly, I think just sort of the advancements by the open source community, right?

There's been so much rallying by, you know, different companies, definitely led by meta and a lot of different companies to make the open source more available and grow it quickly. Um, and just seeing, and, and look, I've been a big benefactor of open source through my career in startups. So seeing the companies rally around open source, um, and so they're not being like a winner take all in a single company.

That's really exciting. Those are the things that, you know, I really think about a lot. Very interesting.

Thank, thanks for those thoughts. Sunny Madre, general manager of GR Lab, thanks so much for being with us here at six five Summit. For the rest of you, stay tuned.

Much more to come.