Code Metrics, AI and Observability – Peter Pezaris, New Relic

This is texturung TV. Hello, I'm Mike vizard Chief content officer for Tech strong, and I'm here with Alan shimls CEO for Textron, and we're talking with Pete pizzerias who is senior vice president and general manager for strategy and experience. There's a title that's a mouthful at New Relic and we're talking today about code metrics and Ai and how the whole world of observability is changing Pete welcome.

Thanks. Thanks for having me. One of the things let's get started with the code metrics because I think historically people think of New Relic as an observability platform for the it operations team, but now it seems like you guys are reaching left to the developers given them some insights and of course that brings the whole devops thing together.

So what exactly is New Relic trying to accomplish here and what is the role New Relic sees for itself in the devops world? A good question. Thanks for asking.

So when we talked to companies the typical software company today. And by the way, that's every company like we all have software engineering practices. Of the developers that they have in their engineering work.

It's only a fraction of those developers that are actually paying attention. What's happening in production? So if you let's say Acme Corporation, they have a thousand engineers in their Department.

Probably around a hundred to 200 of them are really looking at production on a day-to-day basis. So it's our thesis that if given better information application developers will write better code. And so it's our mission to bring that important production Telemetry into the context where application developers live so they can see how their code is performing in production.

And we think the only way to do that is to go to where they are rather than trying to bring them into the production oriented tools developers like to write code they like to build new stuff they like to test and apply and and see what's working. And and what's not not a lot of Engineers want to spend their day staring at charts and dash but what production So yeah, whatever we can do to make it easier for that to see that information is going to be helpful and informing them what's happening. So this is classic shift left and it closely mimics what we've seen insecurity right?

And you said something in the beginning Pete that I think is really important for the audience and that is developers taking a tremendous amount of pride. In their product in their work product in their code, no developer raises his hand and says, I want to develop shoddy code or a buggy code insecure code. They're professionals.

They're they're this Generations. This errors Guild Guild members, right? These are these are tray.

I mean, there's a reason why they get paid what they get paid and the problem is though when you tell them and you hit it right on the head whether it's security or anything else right regarding quality when you tell them I'm sorry. You can't see that information in your environment. You got to come over here.

You taking them out of there out of their space and it's a problem. So I applaud New Relic for doing this. I think it's a great idea.

And there we're solving to problems here. Number one is just bringing the production Telemetry into the environment where application developers live, you know, so that they'll see it and they don't have to go seek it out and the second thing and this is subtle but very important is we're also scoping it to the applications that they work on. That so they don't have to like a lot of observability instances.

Like if you're working a large company when you go to a tool like New Relic or some of the competitors, hopefully not hopefully, it's New Relic as you sign in you see the entire estate the entire production infrastructure and for a lot of individual application developers that can be intimidating because they don't know where their code fits in right? And so one of the one of the things that we're doing with this code level metrics release is that we're taking only that production Telemetry data that's important to that application and we're surfacing it in the tools that that developer uses as they're working on that application and it's really not only localized to that application, but down to the file down to the function down to the line level. So every time that application developer opens up a source file in their IDE They get to see a layer on top of it, which is how is this code performing in production right.

Now? Wow. Sorry, so I can hear developers right now.

And you know what, they're saying they're saying well, that's great. What took so long? Well, honestly, yes.

What mentioned IDs that they're working and which IDs are you supporting? So we support 11 different ideas. So most of the popular ones vs code is become super popular these days the jetbrains family of editors also very popular.

Yeah. So for most engineering teams, we cover most of what they use. There are some sort of old sort of Die Hard stalwarts who live in their ad and they refuse to change which I would encourage everybody, you know, try different tools.

See what you like. I don't know why there is no fun. Yeah son.

Yeah, I agree but one of the major unlocks you say why does it take so long? One of the major on logs has been tools like vs code because they were developed so vs code runs. Is built using web-based Technologies.

So HTML CSS and that allows people who want to augment or improve the user experience to create plugins that go into vs code and make it better. And that's the way that we deliver this functionality and so vs. Code itself has become somewhat of a platform that allows lots of creativity lots of innovation and they've, you know, it's both an open source model and it's free to use for application developers.

So that's why it's taking off like wildfire. What about like, you know, we hear so much about GID and get Ops is this available yet in for folks who are working in get yeah. So so this code level metrics release is part of a product that we call a New Relic code stream and code stream has Integrations with a lot of the productivity tools that Engineers use every day.

And our goal is to make all of them more collaborative and informed with production telemetry. So it integrates with your code host whether that's GitHub get lab or a bit bucket. It integrates with your cicd pipeline, you know the integrates with your planning tools like jira, and it brings it all together inside vs code.

net and and you're constantly switching between tools and that about you but whenever I switch over to Chrome the idd kicks there's another tab there or maybe I've got slack open and then like I get sucked in right and it takes me out of the zone is a developer. I I this may impact monitor sales. Exactly.

You three monitors anymore right anymore? Yeah, I mean, that's great. This is I mean I Echo might man this is a great thing to happen.

But sometimes like you said you can't make wine before it's time you needed that container for this to play it exactly how smart can all of us get because we have the observability data. We have the traces and the metrics and the logs we're hearing about AI every day. Are you going to apply that to the observability platform and the metrics and can the developers get a heads up as to you know, this piece of code may not perform as well as you think or there's dependencies here.

I mean how deep can we go? There's some super exciting developments that we've got in in Prototype form today that we're working on bringing to Market and I'll describe the use case. It's really an end and use case.

So something some code is not performing well in production, right? Capture that whether it's the error rate has spiked so you will be able to have be alerted to that the application developers who are working on that application, right only those applications. So exactly the right application developers will get a notification in their IDE that there's been an anomaly detected.

Like the error rate has spiked. We'll do that automatically for you through the use. Guy and our machine models, once that's detected you get notified as an application.

It'll take you to the line of code. In your repo open it right up in your ID. So you're in your ID you get the notification you click on it it goes to it.

It'll show you some representative errors. You can click on the stack Trace to go through the frames list activities. It'll take you exactly to the lines of code in the version of the software that's running in production.

And suggest changes route to fix it. com. So it really brings Dev an Ops and AI together to solve problems or efficiently.

Let me just be clear for all you out there. It's not available today, right? There's a Proto coming very but this brings up another issue Mike and I kicked it around a bunch.

So you're saying it's going to suggest code? Yeah look again. I come from the security world.

I remember when we went from IDs to IPS, right? So we detect versus prevent intrusions. How long is it instead of look most of the suggestions I'm making are going to be 90% You're going to use them.

Anyway, how long is it until I just make the change and if you don't want to change, let me know but right often verse out. Yeah, I think it'll be a while and I I say that I don't know what the timeline is, but we're not there today like today. You still need a human to look at the generated code because And you know Microsoft themselves recommends that you run a scan on the code on generated code just to make sure that it's not introducing any security vulnerabilities or any other anything anything.

No, I I and I get it and I agree. I absolutely agree. I will tell you that in the ideas IPS phase.

It was maybe eight to ten years until people started trusting. Yeah, and that was only for the most basic right of intruders. I think it's gonna be a lot sooner because it's like, you know, the people who first buy their Tesla, they keep their hands in the wheel and then like three weeks later.

They're like eating lunch reading books and all this other stuff. Yeah. Well, so and I think that is a lesson though that we've seen is that this the pace of this Is accelerating, you know, I I remember I did a an interview with Luke Kelly.

He's the one of the co-founders of puppet. This is seven years ago eight years ago. I said, you know, what do you see in the future coming up?

And he said well I Envision a time with software right software. Yeah. I said, when do you think that'll happen?

He said certainly by 2035, but maybe maybe before as we said here now 2023 2035 it'll be well done yesterday. Yeah, like I said a few examples recently that that kind of blew my mind one is that gpt4 can create real programs like not parts of programs not fixing them out. No, no not online here describe something and it will output HTML in JavaScript that you can literally copy and paste into a browser and it'll it'll you can create a pong game pretty simply describe and it just makes it that's one thing the other thing which is really kind of eye-opening a little bit.

Scary. Is that somebody asked? GPD for to replicate itself and the way that it did it was it wrote a python script for that person to install on their computer.

And then it had apis that allowed. Jet GPT to control it. Wow, that's a little too and I it's kind of right.

Yeah, it wrote the code and it worked really it did what it that I yeah, I don't really so you guys also just extended the observability platform into ml Ops and those processes. How do you think ml Ops and devops are all going to converge and because when I talk to people they struggle with the data scientists creates the models and then somewhere along the line somebody tries to inject that into an application, but they're on a different Cadence and nobody seems to know what's going on. Yeah, so we very recently released the first of its kind ml Ops product that that will help you observe your models including if you know you're running them locally and it really helps you get a you know, I think we can all agree that we need to keep an eye on it.

All right. There's a lot that's happening a lot of innovation and right now not a lot of guardrails and in some ways like I am very appreciative that that open Ai and Microsoft are being very very deliberative about how they're controlling the software like they're putting on the god rails and I have confidence that will continue to do that. But at the same time it's a little bit like the fox guarding The Henhouse right?

Because they're releasing the software and they're putting the guard rails in place. So I think that people who adopt this technology, it's good for them to really know what's going on. And that's this is where our ml Ops product comes into play.

We have a plugin that allows you to you know, two lines of code you just inject it and we start observing what your models are doing. And it's really really easy to set up really easy to use and the reason that we're excited about it is that we've been developing this technology for a while. So it's pretty mature for us.

Yeah, and we think it's a world that's about to explode because in the same way that in the mid 90s every company had to transition to become an internet company, like none of us had no business was online and then all of a sudden every business had to be online. There's a school of thought that generative AI is going to be the same way that that today not every company has an AI Department not everybody has AI as part of their product but very soon that's going to be the standard by which we expect to interact with the services that we use. It's something to that too.

No knock on Microsoft and open they are they are doing I think but you know, you're talking about Microsoft. Yeah, right. And this isn't mid-90s Microsoft.

They're not quite as arrogant. But but you know, they they have a responsibility. They recognize their responsibility Google.

I think the CEO I think at his name of Google of alphabet said 90,000 googlers. Tested their Ai and now it's ready to you know, start hitting some public. Google's a big company too.

I you know do know evil. What about the ones under them though? And those are the ones that I really worry about right that aren't going to have the ethics that aren't going to have those constraints that don't have that.

Compass if you will and you know, it was the same thing with the Internet. It's perhaps the greatest gift to Mankind in our lifetimes. right, but it's also Been used for evil purposes and sometimes intentionally sometimes not I I can't help but think it has to be that way with AI as well.

We're ready for devsec ML Ops come on. It rolls right off the tongue. That's what it's funny.

We joke about it because this you know devsack ml prods, you know, it's just you guys that's it. That's what we do it call it here. Right Pete.

Let me ask you we're gearing up. We're gonna be actually Mike and I will be there in Amsterdam next month for cube. Concloud nativecon.

Great. I know usually New Relic is there you guys gonna be in Amsterdam. I don't know if you know, that's probably from the European folks.

Yeah, so I don't know the answer to that question. That's a different department within New Relic. So I have to get back to you.

Well, if you're there we'll talk to someone. Yeah. No, we always interview New Relic.

Yeah. I remember fantastic Mike anything else. So ultimately we hear about observability Not everybody knows what to do with it.

Right? We've had continuous monitoring and we have these metrics that we look at and that's fine. They're pre-configured but most folks I talk to are like, this is great.

I can query anything and then they go I have no idea what that we're yeah. So how do we make people smart enough to use these platforms or will the algorithms themselves kind of ants create the question and then we'll figure out what the answers are from there. I think we can tell that problem in two ways one today and one that's coming soon.

So the one that we have today is actually what we're here to talk about which is code level metrics and what this does is rather than you having to go out and seek the information and know what to query or know. What dashboard to build or what dashboard even look at. What code level electrics does is it takes the production Telemetry for the file that you have opened in your IDE?

So all you have to do is keep doing it what you're doing like keep developing software. And when you open that file in your ID, we will show you with annotations how that code is performing what the golden signals are what the throughput is what the error rate is how often it gets called and and that will help in including anomaly detection. So did the error rate go up or down with the last release and you can do a comparison.

What is this release performance versus last? What is production performance versus staging or you're load environment? So we service all these insights right to the developer right in context so you don't have to think about it.

You don't have to know what to go seek out. So that's the first answer the second answer which is coming soon is as we talked about this integration with generative AI tools they're great at two things. Number one is Creating new content and the other is understanding content.

You may be familiar that New Relic uses a query language. That's proprietary. We call it nrql miracle and You have to learn how to use Norco to get really good at using observability and New Relic now our competitors have different query languages that they use.

So it's part of the observability practice to be able to write these queries. Well, Gpt4 understands an equal. really just as good as any engineer on our team like It's expert level understanding of local so you can ask in English like show me all my applications that have a latency over two seconds and it'll convert that into Norco to our query and then fire off the query and alternately you can flip that upside down too, which is given an existing dashboard.

Maybe when the we created for you maybe one that when your teammates created it has some pretty complicated Miracle under the hood. You can say explain to me what this does and it explains it in plain English so that anybody can understand. Not only what that query is, but why why was it written that way and it's kind of spooky that like how good it doing?

I think things are getting weird folks because we're gonna use AI to create the code and then have another AI engine check the homework. That's it. You know that there's something fascinating about that.

I was thinking, you know, my example is In the not too distant future. In fact, you can do it pretty much today because Google is integrating these capabilities. Yeah Gmail.

I can use AI to an author an email. And maybe because you know, it writes really well. I'll just give it a prompt and it'll write me a two-page email.

I'm going to send it to my boss. Doesn't have time to read to whole pages. So he's going to use AI to Summer that summarize it down to like two or three centers personally agents, right?

We've talked about it for the gym AI created the content and the AI summarize the content and that's gonna be the way that we communicate. I I saw an article today actually that this who needs websites. This is a better interface because we've never you know, the whole ux design and you're an experience guy, you know, right the experience at websites people struggle people struggle and maybe this is the ultimate interface.

and Pete thanks for coming by. Thank you. Thanks for having me Pete.

This is so for those. You know, we're our studio headquarters down here in Boca Raton, Florida Pizza local and so you're gonna be saying a lot of him like guarantee you but thank you Pete and thanks to the folks at New Relic. Thanks for having me.

Thank you all for spending time with us. Once again. We'll see you all next time.