From AI Copilots to Small Language Models, RAG and Smart Cities – Techstrong AI Podcast – EP14

Hello and welcome to the Techstrong AI podcast. I'm Amanda Ani, and with me today is Mike Baard. How are you doing, Mike?

I'm doing great. How are you? Doing well.

We have a long list to bring you today, many AI topics and we are gonna start with AWS and GitHub. So Mike, you recently posted an article and GitHub has shared an approach to building software using generative AI to complete tasks. So can you share a little bit more about this with our audience?

Yeah, well it turns out that both GitHub and AWS are down the same path. And so this past week they both announced general availability of a set of tools that you can use to orchestrate tasks that you would use for software development, including, you know, running the tests and debugging software and running the documentation. And the interesting thing about this is they're using their reasoning engines in an LLM to, uh, asynchronously manage those tasks.

So you would put into a prompt that says, um, I want to build this application and run these tests. And, uh, so it would on the one hand generate some code to help you write the app, and on the other hand, it would also asynchronously run the test. So it compresses the whole window of, uh, what we today refer to as kind of the DevOps processes.

Um, A AWS has a version of this that they're calling, uh, I think it's Q for developer or something like that. And this is a version of a thing that they showed at their last conference in last fall. And uh, GitHub showed their version of this last December and they both magically decided to announce general availability in this past week.

And so here we are. Um, it's not clear to me exactly how this is all gonna play out, but clearly a lot of the tasks that are bottlenecks today in the DevOps process are going to be automated. So it will dramatically change the way we think about DevOps workflows.

Developers may be able to manage more tasks that they previously leaned on a DevOps team for the DevOps team instead of manually creating all these scripts may be supervising a lot of AI assistance that they will orchestrate together and make sure that they're all doing the right thing and it's gonna fundamentally change the software engineering jobs. So to me, it's moved past theory and we're now into the realm of reality and people can see this, they're gonna play with it. And over the course of the next six months that think AI's gonna change everyone.

We think we know about software engineering, we just don't know exactly how. And it seems like a lot of companies are racing to be the first out with tools in this area and hoping that they'll, they'll be the ones picked by these companies. Yeah, I'm a little dubious about that to be honest.

Uh, if everybody's gonna do the same thing, right? Everybody has access to the same LLMs and the same reasoning engines. It's not clear to me that AI is a reason to move to a new platform.

I mean, not having AI as a reason to leave a platform, but um, you know, as, as I continue to look at this, I mean somebody may be ahead for a couple of weeks, but you know, it's basically, uh, it a small game of leapfrog before somebody else catches up to them. So I'm a little, I'm not quite clear that AI is gonna be, is such a great differentiator as people initially thought. Yeah, especially it's constantly evolving and it's evolving.

The technology itself is and then to the company's needs as well. So those are always changing. Alright.

Alright, well moving forward, we're gonna talk about LLMs. We've heard a lot about large language models, but we're starting to hear a lot more about small language models. And to that note, apple and Microsoft are rolling out small language models and these will allow for them to run locally on devices rather than in the cloud.

So what are you hearing as far as why are businesses looking to this direction instead of moving to the cloud? All right, well the laws of physics have not been suspended just because of ai. I'm gonna go out on a limb and say that, and here's the issue.

Um, I can't keep routing everything up to the cloud to have some AI model processing. I gotta run it locally and I gotta and be able to ultimately even one day, maybe even train locally. But, um, so speed is everything.

I gotta have these LLMs at the point where the data is being created and consumed. So we're starting to see just about every piece of software pushed out to the edge. So in that regard, AI models are no different.

I think there's gonna be a lot more AI models running out at the edge and then we'll be in the cloud. And I think that this whole thing will wind up being federated in some interesting ways. But who knows, maybe I'm only using the cloud to train the AI models and I'm running everything on an inference engine that's at the point where the software is at the edge.

It's gonna be interesting times and I don't even know what that processor at the edge is gonna be like. 'cause well GPUs are expensive and hard to find. So if I only need to run the inference engine at the edge with some local LLM capability, I might just run that on, I don't know, arm X 86, pick your choice, pick your poison.

But uh, if there's gonna be a lot more processor choice, let that will be for certain. So we will see what happens in the future. But my bet is, uh, this is gonna be crucial at the end of the day because once again, it's all about the speed.

Yeah, absolutely. And another thing to look at is these smaller language models, because they do have less compute power, they're putting out less heat, they're smaller carbon footprints if you divide 'em up into smaller models. Um, so that's something to look at too.

And plus, you know, a lot of these companies don't need these giant large language models. They need something specific, specific tasks only so you know, it's more efficient and the return on investment might be better too. Absolutely.

Yeah. All right, so moving on, you wrote an article about Tiger Graph and they've made available a co-pilot for beta use currently that uses graph tech and vector databases to customize these large language models. So can you share a little bit more about that and and how would businesses be utilizing this tool?

This tool? Yeah, this stuff is pretty cool when you think about it. So today if I wanna customize an LLM, I gotta grab all my data and I stick it in something like a vector database and I gotta go find a bunch of data engineers to go do that.

And then, uh, hopefully I get some output that I then incorporate into some sort of response. And that's how I kind of extend an LLM 'cause each LLM is only trained up to a certain date and may not have seen my type of data. So that's how we're trying to customize that.

Tiger graph went a step further and they said, well we have these knowledge graphs that are built on a graph database and so we know the relationship between all the data that's in your organization. So we can now automate that whole process of showing the data to the LLMs. So you don't have to have a bunch of data engineers figuring everything out.

You as a average person will now be able to kick in a prompt and Tiger graph will manage that whole backend rag process in a way that's transparent. So, you know, to me this was pretty cool and I'm not quite clear what it means for all our friends in it, but it means that we can customize LMS at will. Well, and that brings us to a, an interesting article.

You mentioned RAG and Jeff Evernham over on techstrong ai. We have an article from him about why retrieval augmented generation is the key, um, better known as rag. Um, and a big reason is because the data is more reliable using this system and it's more company focused.

So what are you hearing as well from business leaders Leaders? Yeah, I hear the same thing. And and this just goes back to the issue of well, all this LLM stuff is awesome, but uh, how do we operationalize it?

Because other than the fact that I'm using an LLM that maybe craft a better email, I'm not really taking it to the next level 'cause I can't get my data in front of it. So to me, uh, figuring out all these rag issues is the, is the key to the whole thing. If we can get that process down and make it so that there's no friction, well yeah, I mean rags gonna be more important than the LLM ultimately, but maybe just hopefully as transparent as possible.

Yes, and I feel like I'm posting a lot more stories about use cases for RAG and a lot of business leaders are looking this direction. Absolutely. Hi.

Sooner the better. Okay, so moving on, the National Institutes of Science and Technology, um, that program has, um, they've put out a platform to identify AI versus human outputs. So, um, and they're utilizing this to try to, you know, scan for, um, you know, fake in fake information and, um, information, information pulled that shouldn't be being used, um, copyright issues and cybersecurity issues.

Um, there's many reasons why they're putting this platform out, but what are you hearing? It's a good thing. Um, it's just my question is, is, okay, so now NIST is built essentially a framework or at least identified a framework, uh, somebody's still gotta go actually write the code to make that thing a reality.

Uh, so I believe that all these things are gonna be critical. I mean, security once again is chasing after the latest and greatest innovation, but, um, we're getting to the point now where these people understand what the issues are. So if we can get some common, uh, solutions for doing that and the capabilities are there, I'm not sure if they can identify the content as precisely as they might like, but uh, they certainly can do a better job and we can identify all these deep FA fakes and we will need to figure out, you know, how to do all this stuff.

It's a work in progress. It's interesting that the government's involved at this level. Um, I know that, uh, the folks at NSA are worried about cybersecurity AI and then there's NIST and then there's CISA and all that other alphabet soup of agencies in in Washington who were turning their attention to this because I think there was a, an executive order from the Biden administration about it.

I I'm just not quite clear how the public-private partnership is working out here. Exactly. But um, everybody's saying the right things.

Yeah. And we seem to be seeing all the government entities, uh, across the world starting to get really involved in AI regulation, AI laws just because the technology is evolving so rapidly and with all the good comes, all the bad that has to be looked at. And, and how how can they fight against, uh, threat, you know, threats from AI is a real concern.

Yeah. And the bad guys are fast and the governments are not so fast. So I'm hoping that the companies can step in the middle of that and maybe even that playing field a little bit.

Yes. Well that brings us to, this is really interesting. Uh, Paul's kind of in this, um, realm of smart cities and LA their metro bus system is utilizing AI to help with identifying illegally parked vehicles by scanning license plates among various other things.

And we're seeing more of cities utilizing AI and camera systems and things like that. So what are your thoughts? I'm hopeful, I'm not sure.

I mean, we've been at this technology thing for traffic for as long as I can remember figuring out how the optimal routing thing for um, essentially it's a system, but it involves people and people do weird things for no apparent reasons that are hard to predict. So you've been in traffic, you've been driving and uh, you know, somebody ships lane not for any particular purpose just 'cause they got bored in the lane that they're in. Well, I don't know how a machine is gonna predict that and then translate that back to something a bus can consume in, in an interesting way.

But, um, arguably probably can't get any worse than it currently is. So there's something to be said for that. But, um, we, we've been at this use case for as long as I can remember.

I'm hopeful it's something is happening here that is not just throwing tech at something 'cause cities have budgets, but um, maybe they'll come a day. Uh, maybe it's just a matter of making sure that there's less traffic in the first place with things like congestion pricing. I'm sure that's that popular in a lot of places, but something's gotta give.

Yeah. And I think that, um, it'll come down to how they harness this data, how much data are they collecting, the reliability of the data, and then how do they sift through it and then use that to make smart decisions. Right.

And so, and even if you get my license plate right, well, I might be driving on that road a lot, but 10% of the vehicles are probably from outta state or somebody who's just kind of there in a random moment. And how do you predict that? Yeah, absolutely.

Still some things to iron out there, but it seems like, uh, an interesting possible solution. And with that, we are done with our list for today. If you want to go to Techstrong ai, you can read the full articles there and get all the information and let us know what you're interested in.

So until next week, have a wonderful day. Thanks Mike. Thank you.

Take care.