Operationalizing AI: Strategies, Challenges and Innovations for Seamless Integration | Predict 2024

Hi there. Welcome everybody. Boy, with that countdown background, it kind of felt like I was going to watch Oppenheimer or something Looked really cool.

So, but no, this is not the Oppenheimer session. This is the session on operationalizing AI strategies, challenges, and innovations for seamless integration is their title of our talk. My name is Mitch Ashley.

I am principal and general manager of Techron Research and also the CTO for Techstrong, the organization, putting on our event today and, uh, running all the technology platforms and all that great stuff. You know, AI is like, I would wanna say tip of the tongue. It's like in front of everybody.

Like everything. I think it's on our screensavers everywhere. We just can't get away from AI and probably because generative AI has really pushed it to the forefront.

You know, we've been working with AI for, for some that time now and organizations implementing that. So we're gonna talk about kinda where we are at the beginning of the 2024 in the state of AI and where we're headed and some of the challenges, opportunities, how do we integrate it into our companies, things like that and have a great discussion. Speaking of that, I'm joined by a very esteemed panel, uh, some really experts in many different fields, but, uh, in ai security software.

A lot of, a lot of great things. So we're gonna, uh, jump right into it. I'm gonna lead off by having the panel introduce themselves.

Uh, before I do, if you've joined us and you want to participate in chat with us, feel free to ask a question, jump in with a comment or reaction. I really like what Mark said, or, you know, the interesting perspective that Robin brought up, or here's my experience. Feel free.

It'd be part of the conversation with us, or you can sit back and, and, uh, kind of enjoy the ride right along with us. So let's begin. Uh, Josh, I'd love if you would introduce yourself.

Sure. Josh Black welder, deputy CISO with Sentinel One. I run, uh, day-to-Day Security for the company.

And, uh, happy to be here today. Amazing, great to have you. Thank you for doing this.

Thanks for joining us, Joseph. Good to see you. How about to introduce yourself?

Good to see you too, Mitch, and great to see everybody on the panel. Josephine, I'm managing Director of Emerging Technologies, AI and ML from Enterprise Vision Technologies, and looking forward to the panel. Fantastic, Robin.

Pleasure having you join us today. Hey, nice to, nice to see everybody here on the panel. Um, my name's Robin Gman and I currently work at Carnegie Mellon Software Engineering Institute.

Uh, I specialize in the space domain, so I am the space domain lead there and I'm super excited to hear what these guys have say. I work excited to hear what you have to say as well. Amazing.

Great. Great. Mark, mark, uh, introduce yourself.

Yeah, I'm Mark Henkel. I am the CEO and founder of an artificial intelligence agency called Perty Labs, and I publish a weekly newsletter around artificial intelligence called the Artificially Intelligence Intelligent Enterprise. And before the panel, I think we've established I am the comic relief of the bunch.

So Be prepared. Good to see you all. We, I think we knew that before we started, not knowing you, but that's great.

Awesome. Good to have everybody join us, and thanks for taking the time to be on the panel today. You know, it, it's interesting, I, I did work at AI in the eighties.

I mean, this tell you, been around for a while, you know, in fact doing prologue and lisp and lot the expert systems and early on, and of course then we were thought we were gonna be emulating the brain and how people think. And of course, that never really happened, never really kind of popped to the surface. Machine learning came along and really helped kind of advance ai, uh, in, in the minds of how people can leverage it.

And generative AI has really brought it, I think, to the forefront for everybody, frankly, because you can use a kind of a query, if you will, type interface to using a a large language model. Let's talk a little bit about, um, where organizations are in their adoption cycle of AI in general. We can talk about different aspects of it, whether it's generative AI and, uh, you know, maybe what some of the successful strategies, uh, for those are who are being successful that they've adopted about how to, uh, integrate, embed, utilize, um, AI as part of their strategies.

Uh, Joseph, do, uh, do you wanna kick us off on that? Sure. Um, I think, uh, you know, as we were speaking earlier, um, IBM conducted a recent study that showed around 82% of enterprises either have or are planning a gen AI related initiative.

Um, and if you look at some of the industries that, to your point are, have really sort of taken off, obviously financial was listed as one of the, the main industries there, but one of the other ones is, uh, healthcare. And I think that one was kicked off really early on by J and J's use case, um, during the pandemic, right? Identifying locations in South America and South Africa where they could actually do their studies to bring, um, you know, these vaccines and test to market recently with the halon creation with, you know, let's say MRSA sort of things that could address that, um, retinal scans for Parkinson's, uh, re and recent advancements for a LS.

And so I think in that scenario, there's been a lot of, of movement in those two industries. I would say that other, um, industries, um, taking advantage of, uh, gen ai, right? We're about, I would say 50% done with the research.

Um, and I'm being generous there, um, on, on how to, to deploy, deploy generative ai. 'cause there's still some unsolved problems there. Uh, one of them being, uh, hallucinations.

So I would say right now, while we're working on identifying and, and finishing the research on generative ai, I would, um, caution organizations but have them spend time really identifying what's holding the business back and pinpoint where AI can be a game changer and really try to truly capture what really matters. It's not solely about ai, it it, it's really about empowering your people and your teams and sort of driving customer satisfaction and really letting AI be a catalyst Jump right in there panel. So, uh, from my perspective, um, you're seeing a lot of movement within the government space.

Um, so Department of Defense in November, you know, put out a, a political, um, uh, memo, whatever about AI and the responsible use. Um, you're seeing that we, we now have a, a chief intelligence officer and CIO and that, um, they're really looking across the board on a variety of things that AI can help with. But one of the barriers I think that we see within the Department of Defense is there's a huge amount of data, but the quality of the data isn't always to the level that's needed for this, um, to, to get these great insights to really be able to, to use the power of the tools.

Very interesting. How about your perspective, mark? Can you hear me?

Yes. Yeah. Yeah.

Can you hear me? Yes. There's a Big message that said my, my microphone is not stopped, so I'm sorry.

So I Think that it's working good on this. Great, uh, 2023, I feel like was the year of the, um, AI pilot, and this is the year of the AI of scaling ai. And I stole that from a friend of mine who's, uh, a CEO of OPA systems.

But I really think that, that nobody was prepared to see open AI become such, uh, like global force and it really made it accessible. So now I think what we're seeing is lots of companies that are overcoming these final hurdles of, you know, having the expertise internally to adopt AI as well as, um, you know, they didn't budget into their 2023 cycles, uh, AI projects. So 2024, I think I've seen stats where, you know, 27% of new IT initiatives are ai, um, in their budgets.

So I think that's where we're at this year is the big, the big hurdles are really security though, and having the expertise to do it inside. 'cause it's moved so fast. If you have a year's experience in gen ai, you're a, uh, a gray beard if you want to use the tech term for the experienced folks.

And, um, unless you were in a university or somewhere doing research, there's not a lot of people with a lot of AI experience out there. You know, there's almost a, uh, I, my sense of it, mark, is there's almost a FOMO about we're missing out of, I've gotta get it get going on our AI projects, how are we gonna use it? Sometimes that can lead you down the wrong path 'cause you're just doing it to do it versus figuring out where you really can leverage it.

Um, but it seems like everybody, if they aren't doing something, then we gotta do something. We've gotta get some projects started and figuring that out. Um, I'm curious maybe Josh or someone, if you have perspective on that.

Are you seeing that with customers? You know, I think there's a, a couple different perspectives. There's, you know, how, how are we gonna utilize AI internally to help our business and generative ai?

And then what are we gonna offer to our customers and do that in a safe way. And, um, I think with most ai, the more specific problems that you can solve, the better results you're gonna get. If you go too wide on your scope, uh, sometimes it can be a little bit daunting and you don't get as good of a solution.

We've been, um, you know, uh, giving our customers AI solutions for quite some time. There's no reason that all data needs to be centralized or that you need to train the models with your customer's data. That's, you know, a misconception.

And, you know, for on-prem solutions and, and things like this, um, you know, you can, you can distribute AI models to, you know, edge devices to, on-prem solutions that haven't been trained with customer data. So there's lots of solutions for, for many different implementations and how you can distribute ai. Um, at Sentinel One, we've taken the, to err on the side of safety, uh, due to some of the security concerns with, with ai.

And so we definitely believe that in, in guided AI where we provide results back to the users, they can see how we kind of arrived at that and explain it to them, and then allow them to verify that before moving on to the next step. And we feel like that's kind of a good hedge against hallucinations. And, and, um, you know, some of the other problems that you can have with generative ai, You know, we talked about not necessarily having the resources internally there, certainly around generative AI given the newness for, for the vast majority of us, some groups have been doing, you know, a lot of work around machine learning and some of those kinds of algorithms before it, it seems generative AI has kind of started with the bot, right?

It's everybody's got their gen AI bot that answers whatever questions for customer service for looking things up. Um, and that's kind of the entry level or for starting point for that. How do you go, how do you go beyond that?

What's the next thing to tackle with ai, whether you're trying to advance, you know, other kinds of projects or, or you're pursuing generative ai? I, I think, you know, I think actually it's probably, Go ahead, mark. Yeah, go ahead, mark.

Go. Okay. I was gonna say actually, um, just one step before that is before you, you know, tackle the, the nuts and bolts of deploying it, having sort of a framework for, you know, what's your acceptable AI use, what are your policies and sort of like your data governance before you start down the path of like actually enacting a gen AI project.

So that would be my only thing is, I know I did it the opposite way, so do, as I say, not as I did, is like what's my, you know, you know, I, I, because there is that fomo that you mentioned before. But, uh, you know, I bet you J Joseph probably has some more once you've done that tactical advice on how you can really, um, have success with these projects. Great.

Yeah, I think that's, that's sage advice mean, I think we may have all gone down that the, the side path before getting back line. And just to note, I think, I think Mark May have a little bit of a delay his his signal back to us. So, uh, we'll give you a little extra time to respond if, if you speak up, mark, don't worry about it.

Jump right in. So, um, I, I think Joseph, you were gonna comment too. Yeah, I, I think resonating with Mark a bit on this navigation of the AI tool and dis deployment landscape, to your point, and creating sort of this framework and, uh, you know, we were talking earlier in the panelists together about, um, sort of analogies for our organizations that we work with.

Um, and, and the analogy that I was kind of trying to get some feedback on was the analogy of like enterprise or government as sort of like a train that's kind of moving down the tracks, right? And we're all trying to move our organizations in the right direction, but, uh, we have this moment where AI is really like a, like a vehicle that can turn to the left and to the right. We've got individuals in our organizations that we want to be able to allow them to find these new things and, and do so in a safe way because the vehicle can turn left and turn right and can do it in an autonomous fashion.

And it really creates these, let's say, thousand points of light. But as an organization or as the, you know, government entities, what we eventually we want to do is take those thousand points of light and turn it into a mesh so that we can get the benefits of the, the train, if you will, along with the vehicle and move the organization forward. And, and as Mark indicated, there's some, some guardrails that we wanna put in place.

We wanna give people safety barriers to be able to test and validate how their solutions can be augmented, um, with these tools. But again, do so in a safe and secure way so that we don't, you know, let these risks get out to our customers This year. Um, you know, kind of to piggyback off of, of that, uh, Carnegie Mellon really stood up their own artificial intelligence, um, group, right?

So there's multiple groups that the Software Engineering Institute supports, and they set up ai, and it's primarily to really help, um, the DOD and, and the government workforce to be able to securely and safely use ai, right? How to make it responsible, fair, transparent, how to get some repeatable results. And so we have a number of researchers, I am not the expert, I know enough to be dangerous, not enough to be smart.

Um, a number of researchers heavily focused on trying to, you know, help DOD leapfrog ahead and really be able to take advantage of this technology. Yeah. Um, I agree with these points that it, I think with what Mark said, it's super important to have governance and, and sort of some guardrails around what you're trying to solve with your ai.

And, um, then that translates into how can you contain the language models to the context that's appropriate given your definition. And, uh, we, we do this through a combination of like large language models and small language models that help contain the scope. Initially, we're very focused on security use cases, so we use some of those models to contain based on our specs to that smaller scope of security type use cases, right?

And then expand into other language models to sort of help solve those. But without that sort of initial definition and guardrails and, and, um, definition to, to start with it, sometimes you, you allow too much capability into the system, which causes, you know, concerns. Um, what, let's talk a little bit about, you mentioned security, of course, and that that comes up fairly quickly when we talk about ai, especially in large language models.

Um, given the potential for data loss and, and what they're used for, maybe the potential impact of that, the safety and control, what are some of the challenges you think we need to focus on this year in helping with AI adoption? Yeah, I definitely have a strong opinion about that. As far as the security goes.

I think that, you know, if you look at almost every report or survey of why companies aren't adopting ai, it's security. And I think that there's a, you know, it's one of the few times in it where we have more demand than we have supply. I don't think that there's a huge number of, um, security vendors out there that are there.

I I'm sure there's some people on the call that probably can give us some, uh, insight into some that do. But, uh, um, I think that's probably the number one, you know, de deterrent for people putting, you know, real sensitive enterprise data into these large language models is they don't know what happens after you pour the data into the black box. Yeah, I mean, you know, resonating with the group on this, um, and I think Robin touched on it a little bit too, the data that we're interacting with, but also transparency in the data that's gone into training these models.

You know, some of this, uh, emergent behavior is directly reflective on the data that's gone in to train these models. And I think a big challenge with, to your point, uh, hallucinations and security is we don't necessarily have transparency in all of the data that went into these models. Um, and so those emergent behaviors are questionable.

And I think there was a study done recently by a Luther AI where they found out that 92% of data in the large language models is actually not used 99% of the time. And so in that scenario, there is a, a signal versus the noise, but we're still, I trying to identify that signal in the pre-training. And then as the group is discussed, right, how do we, if there is sort of toxicity and vulnerability and hallucination, how do we put guardrails on it to refine and identify and then level up and, and fine tune these models in a way that can do so in a secure and, you know, non hallucination type fashion.

I think at that generative AI hackathon we, we got to attend this summer, um, you know, uh, Joseph, you really helped us, uh, understand how we can take portions, right, those small language models and maybe move them into, um, pretty much like Josh said, domain specific content as well as, you know, being able to, to try it out safely, um, versus putting your data into those. I I perceive that that's potentially one of the ways, um, at least from different geo perspective and leverage generated AI without actually putting that data out into the wild. Yeah, I think it's a good point.

You know, I think there's a lot of confusion on how models are trained and enriched and, and come to their conclusions. And, you know, training a large language model is incredibly expensive. Not, you know, most or organizations don't have the resources to do this, you know, above the fortune, maybe five, um, very expensive.

And typically all large language model needs to, to provide output if it's properly trained in the first place, is additional data in context. And providing that data upfront is very difficult to retrieve or remove from the large language model. I mean, you're talking hundreds of millions of dollars to retrain the model.

So not getting in that situation in the first place is, is key. Like, don't stick your data in to the large language model to get trained in the first place, um, and train it with, you know, behaviors and, and use cases that you're trying to solve. Um, and so, you know, making a a, a robust, large language model, um, to solve these problems is, is important where it can be simply enriched onsite or in the customer's inter instance with their own data and context and output the right, you know, results from, from the model, right?

And, and that's really what we're trying to achieve here. And so I think, um, you know, some, some of the capabilities around redacting or removing data from these LLMs is, uh, I think something we're gonna see this year, you know, come, come to be, I, I think folks are very interested in solving this type of problem. Let's talk some more about the data.

'cause I think that's fast. To me, it's fascinating because, uh, I was talking with someone about testing AI systems. It's not as deterministic as normal software is, right?

We put in these, goes through an algorithm, get x answered, you can validate what you got the right answer. Learning, learning, uh, algorithms as well as generative AI are constantly being fed new data. So the outcome may be different or may, may vary even between each different time that you query it or prompt it.

Uh, it seems to me data is the big frontier for AI of, and management of data. Um, avoid, you know, what are you doing in what you're talking about Josh's, uh, kind of data poisoning, right? What happens when invalid data or incorrect data gets into a system to generative AI and suddenly is influencing outcomes that are incorrect now because of that.

But you gotta go back and how do you, can you back it out? Do you retrain, you know, all of that's just one of, I think many data challenges. Yeah.

Or even worse, your IP ends up in the model Mm-Hmm. Even that you don't want anymore, or even even Worse could be that, you know what, um, because because we've commoditized the model that actually maybe people that would mean to do US harm would directly try to poison the model to give you bad outcomes. Right.

Which is, which is also very scary. Yeah. Yeah.

I think that the, since this is a AI panel predict, my biggest prediction is that there's probably a, a industry that'll arise, uh, around, uh, data supplies chain the same way there was around software supply chain Mm-Hmm. Which is sort of ne nascent, but I really think establishing the provenance of data, public data training sets, internal data sets, and seeing what's going in as well as what's coming on out will, you know, we'll see daisy chaining together of multiple models. So you wanna make sure that all of this data that's coming in is clean, safe, or you least know the source and what's coming out is unbiased and not harmful because, um, it's probably gonna be part of the workflow and not necessarily an end onto itself.

Yeah. Uh, uh, couldn't agree with you guys more. I think, uh, uh, agree with you on, you know, we have the EU regulation along with the executive orders that have come about.

So some of this stuff is going to be, you know, facing us, you know, as an industry that's, that's coming our way. Um, and I think to your point on the data, I think on the front end we will have tools and guardrails and, and I think to your point, maybe purpose-built critics that are in line to assist us with dealing with these things until other, um, of more of those, uh, mature software supply chain mechanisms that you talked about. And, and Robin, I know you and, and your folks that, uh, co-authored Investments Unlimited and you know, bill and, and the team, I, I, I anticipate that that sort of like that policy as code type scenario that you guys sort of pioneered as a, as a group will have its roots sort of moved over into sort of this data provenance and, and quality as code, if you will, for the foundations of how we pre-train and fine tune these models.

So I anticipate, um, that we, we, we'll be seeing more and more of that and agree with the panel for sure. So interestingly enough, um, I've actually, 'cause I I didn't actually help write Indus or, um, investments Unlimited, but we're closely with those guys and, um, we're looking at forum, you know, uh, this year to come together and be able to integrate, um, industrial DevOps, which was the, the, the book I published with that so we can really build in, right, because we're talking about safety and security of these systems build in kind of that compliances code, you know, regulation as code and make it make it that much easier. Yeah.

Forgive me on that, Robin. Yes. Uh, oh, apologies.

I'm, I, I'm feeling like pretty cool 'cause Bill Benzing is an amazing guy, so hey, I'm all about taking credit. Love it. Love it.

He is well. Um, and we need to put a tech on the board. Mark got our first prediction out there, so thanks for leading off.

We're gonna be getting to some more, uh, predictions as we wrap up. Um, I, I'd love to talk a little bit about the role of AI in, in software development. You know, we have co-pilots left and right of helping people with, you know, that have been there for a while already with code completion and things like that.

Now writing more of the code. Um, uh, one of the, one of the, um, sessions today was by, uh, a person from CYSTIC that had a kind of state of AI and code. And that's some interesting stats about how many people bypass the security measures that the organization has to use AI generated code in their applications.

How many times did they see, um, you know, code generated that might have security issues in it? You know, we're still early on in this process, even though we're used to using copilot, uh, code code completion kinds of algorithms. And, you know, not even to talk about whose code is it in that model anyway, and is it okay that we're using it?

What, what, what do we see is gonna happen in 2024? Are we gonna see some an, some an questions answered and kind of helping people move along? Or is it just gonna be a still a wide open?

Who knows? Go ahead and use it and see what happens. Yeah, I, I, I, I think, um, it's interesting that you mentioned that Mitch with copilot, there was a statistic about GitHub that 40% of the code on GitHub has now had its, uh, uh, some adjustments from, um, copilot and the things of that nature, which was blew me away.

Because, you know, to your point, code creation is just in, in its infancy. But I, I would predict that we're gonna get better at that. ai, and some other, um, projects that are coming out in the agent space, um, and our experiments internally at EVT have really shown this relationship where we have a specification that's built and we have a code, um, you know, writing an agent that can write and understand a source code repository to a certain scope.

Um, and then we have a, a another agent that can validate the safety and efficacy of that code. And, and if it's not good and doesn't meet the standards and doesn't pass all of the, um, you know, unit tests or things of that nature that it gets kicked back to the code rider. And that sort of virtuous circle seems to be an, a very interesting pattern.

And I think once we get those models sort of fine tuned in those relationships here in 2024, that we will see some, some more capable models in, in, in code development. Obviously it's not gonna be replacement of developers, but it, it may free up developers to, to not deal with a lot of the drudgery and free them up to do more creative thinking and focus on more critical tasks. Yeah, I heard that.

Um, I heard a bunch of developers were like, oh, it's gonna put us outta business. And then another developer said, know the people who don't learn how to use AI for development are gonna, you know, be the ones that are outta business. Meaning you're still gonna need developers.

We need those humans. And, but, um, we need them to be smart on the latest technologies so that they can do exactly what you were saying, Joseph, which is really like, focus on the creativity part, not the fact that I missed a semicolon. Maybe something else can help me get that.

Yeah. I have a more optimistic, um, look at sort of this as well. And I think the two things that we're gonna see in code are more documentation and more unit tests and better code coverage.

'cause those are two of the tasks today that, um, GitHub co-pilot and, you know, Amazon code whispers seem to tackle. I mean, at the end of the day, um, your context window is, so, it's gonna be so big for complicated software that models are gonna be better at breaking down and writing sub routines. But you still need someone who has an overarching understanding of the whole so software life cycle.

And I really think the thing that I've noticed is when I go to some of these projects now, there's a lot more documentation and it's not written in the most, it's not written like most developers have written their documentation in the past. It's more neutral. There's no snarkiness, there's no little Easter eggs in there.

And I can only attribute that to, you know, the power of AI for good as opposed to replacing it. But I agree with, um, uh, what Robin said earlier is it's not that we're gonna re um, AI is gonna replace developers. They're gonna replace developers who aren't effectively using ai, and you could replace developers with marketers and salespeople and a litany of other jobs.

So that's just my more positive take. Yeah, we've been talking about shift left for security for quite some time, and it's a little disheartening to see kind of that most left piece, the co-pilot generating insecure code on a pretty, on the majority of the basis. The, the stat I last heard was like 70% of the code generated from these code tools is insecure.

And my prediction for this year is that we'll start to see, um, you know, more improvements around, you know, the generative code, uh, code completion tools around security. And, and that will be definitely a welcome addition. Um, you know, outputting more code is great, being more productive is great, but creating additional security holes, uh, faster is, is, is is kind of working against a lot of folks.

Right. I dunno if I'm gonna predict this, but, um, I'm looking forward to possibly having a snarky token that will determine how snarky my documentation is in my code. Sorry, Joseph, go ahead.

No, no, no. I, I love it what Mark's saying, because I, you know, as I look through these repos, I have noticed that a lot of the documentation looks, looks so much more well organized. And I'm wondering, wait a minute, How'd this happen?

So nice. All of a sudden, huh? Chat GT had nothing to do with that.

I'm sure. I Gotta figure that. We're gonna be, oh, I'm sorry.

Go ahead. I was, I gotta figure we're gonna be more secure given that the most of the vulnerabilities, or at least like the top 10 have been the top 10 forever. I mean, what a great place for, um, AI to help us with is, you know, basically clearing those up.

Like maybe it's an intentional approach, you know, to make, but like we, we see the same top 10 vulnerabilities almost every year. Yeah. They haven't changed much.

Well, let, let, let's turn to predictions. We've got about 15, a little under 15 more minutes to go. I'm, I'm gonna throw one out there.

And this is back to the, the software. And I very much believe it's a productivity improvement for developers. It's been for me, and one of the things that, uh, you turned me on Joseph early on is to have generative AI explain code to you.

What does this code do? How does it work? You know, that kind of thing.

And it got me to thinking of how many legacy applications do we have where the people who wrote it aren't here anymore, they work somewhere else or moved on or whatever. And people are afraid to touch a lot of parts of legacy systems 'cause they don't wanna break it and trying to understand what it does. They're large, they're complex, et cetera.

And, um, I have a hope, maybe it's a also prediction that we're gonna see, uh, generative AI and coding tools targeted at helping us understand how software works in legacy applications as we look to modernize, um, update, integrate. Maybe it's not the entire, you know, monolith application, but parts of it that would be a massive productivity boost. 'cause you think about how often we're, we're saddled with technical debt that we don't know how to solve because that code's been around too long and people that, that are around don't understand it.

Comments about that Join in? Yeah, I kind of agree with you more. Go, go ahead, mark.

I was just gonna say, I think this is the great hope for us to finally modernize COBOL systems. But As we say that, I know that every banking system in the world that uses a COBOL system is playing the world's most expensive game of Jenga. Worried about changing any little piece of their monolith.

But I do think that it's, it's inevitable that AI will be the, since, you know, most coval programmers have been long retired. Um, that's the only hope for things like that. There's another thing in the, that's in the, uh, um, uh, national hospital system for veteran, the va, sorry, the va, they have, you know, systems written in something called mumps.

And I think mumps came about in the seventies. So there's things, I think you're right on Mitch, I don't know if it's this year, but someday they're gonna have to pull those Jenga blocks out and hope that everything doesn't fall. Yeah.

I really hope that, that, um, you know, organizations do follow you, the lead that you guys are proposing on at least identifying, um, de-risking initiatives, right? And to your point, maybe this isn't the year where they go and, you know, wholesale start, uh, migrating off of these systems. But I think this is a good time to start identifying where those sort of, um, high risk environments are.

You know, where, you know, things are just, um, you've got no one to maintain it. It's very key to the organization identifying those and building strategies sort of to de-risk. I also think that'd be a good way to bridge relationships with the compliance folks, right?

The three lines of defense folks that we have out there to protect our policies and things of that nature so they can really see and get a, a feel for what this this does. At the same time, while we're de-risking, um, portions of our organization, I gotta tell you from, from the Department of Defense, we have so many legacy systems, legacy, safety, critical systems that are written in things like, like ADA or other, you know, really old languages. And being able to update those or create, you know, new capabilities, leveraging AI to do exactly that.

Refactor some of the monoliths help us, you know, maybe put it into some modern languages. Maybe we can even increase the workforce, you know, because how many kids coming outta college wanna code in aada? Not many.

Um, so I, I see huge benefits. DODI Predict, go ahead a lot. Oh, sorry.

A lot more. No, please go ahead. Um, use of ai, uh, to ensure quality across our organizations and, uh, every aspect of quality, whether it's code testing or customer service or team quality, et cetera.

If you think about like how AI works, it's, it's, it's all a probabilities game. And where those probabilities are really safe is, you know, which work in, in our organization should be reviewed for quality. Well, you need to take samples.

AI's really good at taking lots of samples and giving you the highest probability work that was, you know, not done correctly. So, you know, using AI for like the right set of, um, use cases, this is a great use case. Improving quality where you can get the most high probability things that went wrong in serving our customers and reviewing those and, and putting that back in the loop to get fixed and improve quality for our, our customers, our teams, you know, every aspect of our, of our organization.

So I predict we'll start to see a lot more of that coming out of, um, you know, generative ai. Uh, 'cause I think it can help us like hone in on those problem areas in our organization About, About risk management. Hmm.

How about risk management, Robin? Yeah, I mean, I'm, I'm actually curious once you said that about the, um, quality, I'm thinking, well, risk management is all about the probabilities. Yeah.

Um, that, that seems like a great area to consider. Yeah. Stay tuned.

Mm-Hmm. Who you tapped on something there of interest. How about the, the subject of the energy, um, and the consumption, especially around generative AI and how much processing power.

I know, mark, that's something you've, you've given some thought to as well. Yeah, yeah. I think where we're at today in the early days is we see, you know, the default for training.

A lot of these models are GPUs and they're very power. They suck up a lot of power. You see open AI and you see the stats of what kind of usage there is around those GPUs.

But the, these were for large language models, and now we're starting to see these small language models that are designed to run on form factors as small as a cell phone. So, you know, Microsoft has fee, um, Google release their Gemini models, including a Gemini model for the phone. I'm calling these s lms small or SMLs, uh, or s SLMs small language models or Slims.

And these Slims. Um, we'll, we'll push a lot of that AI to the edge in the data center. We also see, you know, Nvidia just became one of the top 10 largest companies in the world, but there's 10 other manufacturers out there and startups that are coming for a piece of that pie.

And I think competition is going to drive the efficiency of these chips down beyond the fact that, that we will just see, um, you know, the same thing that happened with Moore's Law as far as, you know, efficiency of chips. We'll see that around systems that process ai. I don't know what the equivalent is.

I've been tr lobbying for Henkels Law, but just so I get a Wikipedia page, but I don't know if that's gonna happen. But, uh, but no, I i, I honestly think that there, there will be, miniaturization has always been a theme in, you know, hardware and I think that'll happen here. Um, even though the market's gonna grow, I think it'll be a, uh, um, asymmetric and curve between, you know, power consumption and productivity of those chips.

Yeah, I Think, I think we're, we're ready for that. Mark. I think you'll see Siri be the first, you know, kind of, uh, implementation of that, right?

I think they're getting pretty close to Yeah. Yeah. I believe it.

Yeah. Yeah. I think it's gonna be very interesting.

We're starting to see, I think to your point in the open source world, the mixture of experts and sparse mixture of experts sort of competing with these foundational models. And, and not to mention, to your point, the personal devices that you just described, right? Um, we had, uh, like the humane pens, uh, we, we just recently had the rabbit released.

Now a lot of these are, are still using like, connectivity up into the cloud, but at some point, uh, when you get sort of the hardware that you mentioned and the stack, right? I think one of the biggest things for Nvidia is' Cuda. But I, I anticipate that Triton and some of these other, you know, um, PyTorch two will have sort of, um, open that up so that it won't just be on Nvidia.

There'll be a MD and other players that, to your point, can, um, miniaturize these things and, and have the efficiency that, that Cuda has, you know, done to create this monopoly for Nvidia. Mm-Hmm, for Sure. Anybody else wanna open up with another prediction?

And, uh, we're about three minutes left or so I think That Genesee, and maybe you already are, which is why this is an easy prediction. Um, researchers really jump on this. Um, so for example, I'm currently working on my dissertation and I have been playing a lot, um, ever since Joseph taught us this summer about how to, um, you know, do systematic literature reviews and then how to, um, combine some of those to give new ideas.

And so I, I think that you're gonna see a lot of innovation and research leveraging both the large language models and, and small language models with very domain specific context. Context. Yeah.

We gotta get back together, Robin. And I've been working on that government contracting, um, scenario that you, uh, mentioned to me earlier, uh, last year. Awesome.

Anybody wanna jump in on that or add anything? Mark? Like I'm not touching out with a 10 foot pole just back up.

I may be a consultant, but I don't think I have the stomach for government consulting. Too Many rules and I'm not a rule follower. There are a few rules, quite a few.

Well, with that, I think we're gonna wrap things up. Um, I want s uh, do a shout out to, uh, folks in our audience, Mike, um, uh, Isha, I'm not sure if I can read the name. It's a little small text.

Joel, Amanda, a number of folks have jumped in with, um, uh, comments and questions and that's really helped shape our conversation today. You know, you, you all are such great folks and I appreciate you being on this panel today, uh, interaction with each other as well as you know, your own thoughts and contributions. I will do a panel with any of you or all of you anytime.

So let me, let me know and we can get the band back together and we'll do this again next time. So, um, and by the way, what folks are referring to is the hackathon. Uh, Robin had mentioned, I think maybe Joseph did too.

Techstrong, uh, hosted a hackathon back in August and brought, uh, many of the luminaries and contributors and folks that have helped out in the DevOps world and DevSecOps and security and, and, uh, worked on a number of different projects and some great things came outta that. So, and one of them is great friendships and connections that we either created or, uh, developed further. So that was fantastic.

Well, behalf on behalf of everyone at Techstrong, I want to thank Mark and Robin and Joseph and Josh for being part of this panel. I definitely wanna thank you and the audience for being with us today, being part of the conversation, whether you sat back and listened and kind of absorbed or you jumped in and chat with some ideas and thoughts. And I hope that you have a great rest of the conference today.

Uh, please check out other sessions, some more great stuff coming up. And, uh, mark, Robin, Joseph, Josh, thank you very much. We'll see you all again soon sometime.