AI Hackathon 2023 | AI in Action 2023

Once in about a decade, a seminal technological innovation arrives that fundamentally changes our assumption about what's possible. The latest such innovation is generative ai. There's almost no aspect of business and it that generative AI won't transform, but a Textron group.

We wanted to focus specifically on how it will transform it operation. To that end, we brought together some of the leading lights of the DevOps movement hackathon to determine how best to harness generative AI led by DevOps luminaries such as John Willis, Damon Edwards, Patrick Debois, and our own Mitch Ashley, a total of 15 IT veterans with extensive DevOps experience shared their insights and techniques over the course of three days. Most of the people in this room were involved in the early days of DevOps.

Even though it's different in DevOps, it's gonna have impact in the kind of work that we've done over the last 15 years. And so I think just trying to get, jumpstart that idea and then I think we got the right people in the room to be able to have that initial conversation. You know, I'm really most interested in operationalizing ai.

So, uh, you know, how do we actually put this to use to solve real business problems? And, uh, when I saw the list of folks that were gonna be here, it's people that I've known through the years that, uh, have done just that in many different, uh, contexts. So, you know, as fellow travelers, uh, excited to see what comes, what comes out of, out of all this.

So I always like to learn and, you know, what's better to learn than with more friends, right? We've been in the industry a long time, and it's, uh, it's nice to see new, exciting technology. This is the kind of thing where you put a lot of smart people together who are filled with ideas and we don't know what the outcomes are gonna be.

It's kind of, it's sort of self-directing, and oftentimes an expected outcome isn't what you get some very unexpected results. And, uh, we'll see what those are. Yeah.

Well, actually what happened, I was in DevOps days, Amsterdam and Patrick Debois has been, you know, he was the, you know, a lot of people joke that, well, not joke, he's considered the godfather of DevOps. And Patrick said, I think I want to come over and do a hackathon. Other attendees included Tracy Bannon, Bill Bensing, Alice Chen, Chris Corriere, Joseph Enochs, Mark Hinkle, Alex nar, Shannon Lietz, Stephen Magill, and Robin Yeman.

The overarching goal is to not just define processes, but ultimately build software. But before anyone can write code, there's always a need to establish a common understanding. Look, we're bringing together experts from all over the United States, all over the world actually coming together, and we're all talking about this tidal wave.

We've been working on it before it became a public thing, and now we're coming together and we're gonna put a framing around, its scaffolding around it, and actually help make an impact on what's to come. Uh, I think this is a unique opportunity for us to set some direction, uh, with generative AI and industry. It's very accessible to individual hobbyists right now.

They can just go sign up the credit card. Uh, but how do you bring this technology into a Fortune 500 safely, where you got hundreds of developers working together? And it's reminds me of the early DevOps movement, Just as critical is going to be making those AI advances accessible to IT professionals of all skill levels.

In the big picture, I'm very interested to hear from the, the group here, what they think, how it's going to impact change and, and lead to new ways of working. There's a lot of tech things that are gonna be fun and interesting, but I think the biggest thing is how do we make that jump from the very geeky stuff to the, the domain and the everyday practitioner to help them understand how to leverage this, be safe, be confident with what they're doing. I'm most excited about generative AI and how they bring DevOps thinking to how that impacts companies and the way people do their work.

There's a whole area that's not yet been explored, or words that have not, uh, and phrases that have not been developed yet for that. So what I like to do with this project is figure out how to bridge what I call the geek gap. How do I cross that chasm into the, into the, the normal day, the normal day lingo?

And also just from an industry standpoint, what does it mean? You know, how, how, how is generative AI gonna change the way people do their job? And let's not forget about the need to make sure AI models are secure.

My background is security. So, um, getting, uh, up to speed on all the attack vectors for, uh, for these models. Uh, the new security considerations that open up when you're deploying these models, especially deploying them at scale and over large data sets.

I'm really interested in working together with everyone here to sort of chart that out and figure out best practices, uh, that we can apply as we, as we go and deploy these models. As the saying goes, if you wanna go fast, go alone, but if you wanna go far, we should all go together. I'm really excited about seeing where generative AI can take it, how much more we can do with it.

The most important aspect of any successful journey is knowing where you're going before you set out. There may be twists and turns along the way, but the difference between exploring new territory and simply being lost usually comes down to having a map. So historically, we've always had tool chains and pipelines in our infrastructure, and what we're trying to do is figure out what the tool chain in the pipeline of the data that interacts with our AI looks like, in the same way that we would deploy things to the cloud.

So we have all of our data over here. We have a layer of storage, it's the vector database. Then we go through our middleware, which is the code that we're using, something very popular right now called Lang chain, which instructs the LLM, how to handle that data.

And then over here we're talking about the loop of us developing and deploying that data and tooling around the large language model to run them in, in an enterprise, As is always the case when it comes to any expedition involving it, the best place to start is with the data. And, and one of the things interesting too, I've had a problem trying to understand what's the right mixture of metadata versus embeddings. So there's a nice discussion about, it doesn't make sense to put the metadata in, in every case in the embeddings or sort of yellow m itself.

So thinking about how do you sort of, firstly, if you think about operationalize, like incident is a good example. Just by default it's gonna be the source document of the document you loaded in. But what you really should be doing to be a responsible citizen, if you're sort of doing like what I do, and I'm trying to change my code now to do this, but I, I kind of say because I saved the links anyway, but the point is like, I should be stuffing in the original source.

So now when people do any summary against it, like they're gonna basically say, Hey, I didn't write this. You're getting a summary from Shannon's article. I think at the end I'll summarize where all of the access control elements kind of come in, but the vector stores are definitely one of 'em, right?

The provenance of the data coming in, as you described, once the data is there validating that, okay, this data came in and we checked it for PII and all these constraints, but we need to have an ongoing thing with these vector stores to make sure that it didn't come in somehow. Or if it did, it's tokenized or some securities around what data's sitting inside of the vector stores. And then at the end is sort of this concept of an agent and critic, which comes from generative adversarial networks where you have sort of a feed forward network, which is your last mile, where people are prompt injecting you, or the large language models from a mixture of experts or a single LLM is spitting out data that you've got this critic that can come inbound and say, Hey, we don't respond in, in that fashion, right?

So there's a sort of a emerging thought process of, and fee forward including some of these things. The goal would be, let's say if we're doing, um, question and answers that we can take a corpus of data that has been validated and provenance set up in our, uh, vector db, we can use that as a data source. We can also use the question and answers coming in, in our cache down here at our bottom element here.

Um, or we can use our large language models to generate these question and answer pairs or instruct pairs to fine tune, right? And then pass through this data governance block here, which probably includes, you know, everything if you will. But making sure that that data is clean, making sure that it's secure, understanding the provenance of that data, cleaning it and structuring it so that it can be trained and tuned.

And then taking the foundational model, training that model, validating it, and then red teaming it to make sure that it fits within the confines of, of what our organization believes should be the parameters of security so that it's not hallucinating and things of that nature. Then taking that fine tuned model, putting it into production, including in our enterprise integration, in addition to including it with our retrieval augmented generation for people to potentially do their individual corpus of data or public corpus of data that we can then transform into specific use cases. Of course, whenever there's a large amount of data, there needs to be pipelines to move it.

You're capturing observability, you cannot capture, oh, it was a prompt with that version. It was that the version? No, it's the prompt given the context given the user, given the LLM it was used.

So it becomes like a lot more complex to kind of do the observability at that time. But the cool thing is if we capture all that data, we actually feed it into a new set of data, which we call like, uh, behavioral data, uh, or context data, which could serve as a memory for what the user did. So this is generic knowledge, this is user behavior, and then we can kind of like start mixing both of those together.

So that's kind of the loop we've, we've discussed in, in the pipeline. So the pipeline is just not going to provide, it's kind of like the whole feedback cycle. And once any AI model is created, the next major challenge is naturally defining a set of best DevOps practices to deploy It.

I want to bake off a large language model. And then the next thing is, is once I've baked it off and fine tuned it and made sure it's good, I want to put it into production, which is, I think the big challenge for us, right, as DevSecOps practitioners, is how do we take this model and promote it securely from, um, you know, a loose dev environment, sandbox all the way to customer facing in, in the right way. Traditionally, what I've seen with data scientists is right now they'll build a model and they'll go through, let's say in financial and bias checking and all of these validations, and they'll publish that model with metrics associated with that particular model, and then they'll release it and then the DevOps teams will work that into its production use.

The data scientists are very rarely involved at that point in productionalizing in the very front end of this model. Once it's been sort of, you know, according to hoyle, sort of blessed for sentiment analysis or whether it's gonna give you a loan or not, I think this particular model, um, focuses more on that. Whereas I believe that the data scientists are now gonna have to have more of a dual role in ops because of this hallucination problem that we're talking about.

Because your traditional ops folks are not gonna be able to know why this thing's hallucinating. There's gonna have to be some feedback loop with the data scientists on all the things that you talked about is the data, did we do, did we get a request to remove somebody's data from all of our data? And we gotta do that.

That's gonna be a, a pretty in-depth thing that's gonna have to go throughout the pipeline. As always easier said than done. You can't know all the things.

Like, it's literally impossible to expect a developer to be be on top of all the security, what model they're running, what cloud. They're not gonna be an expert on all this stuff. Legal, legal, Ve load is a thing.

It's, It's, it's too much of a work. Yeah. The reality is AI is gonna help with some of this to simplify, actually more accessible.

So the question is how do we get back to a place where folks can, if they need to or want to, but we've, we've got so many complicated domains at this point. We have wordss that mean the same thing, duplicated and differentiated and ad nauseum. Honestly, I feel like at this point there's so many duplicate words out there that we've actually gotten to the point where it doesn't help anyone.

Finally, when it comes to ai, security and safety are always going to be a paramount concern. This is all the sort of undesirable effects that aren't about leaking sensitive data, but still cause harm. So like things like bias in the model.

If you're having an AI agent that's acting in an autonomous manner, and so it's making decisions and performing actions, uh, automatically without a human in the loop, then there's potential for undesirable effects. And you would wanna try and get a handle on, uh, you know, the scope of actions that it might take and whether any of those violate some, some notion of, uh, what's desirable. And then resilience is really about predictable behavior.

That was sort of the, the high level description of that that we came up with. And so hallucinations falls in that category. 'cause hallucinations are unpredictable.

That's sort of almost the definition of a hallucination, right? It's like, uh, this isn't, you know, the model's not acting how I would expect, right? I gave it a prompt that seemed reasonable and it came up with something out of left field in terms of methods for defending against these.

Uh, the first one was just setting reasonable expectations. I feel like people, uh, you know, we struggle with having, say developers adopt a security mindset when they're coding so that, you know, we can get more secure software. Like, we're so much farther from having that for deployment of ai, you know, like having a correct mental model for, uh, what does the threat space look like, what's possible from an attacker perspective.

So, um, things like, uh, the fact that, uh, data fed to a model, whether it via training, fine tuning or prompting should be assumed to be shared with anyone that has access to the model, right? Given some of the, uh, results around even black box methods for extracting training data. Like, you should just have that as a baseline assumption.

And if you're not comfortable with that, then you should sort of reevaluate how you're deploying, uh, AI and what you're feeding into it. Whatever you come up with in terms of comfort level for deploying these things. Um, you know, governance, uh, is the method for ensuring that you're doing, you're doing what you say you're gonna do, right?

You're saying we're comfortable deploying this, you know, with these controls in place with this sort of architecture, uh, with these checks in the pipeline. You know, governance is all about making sure that happens. And then red teaming is about, uh, detecting unanticipated vulnerabilities.

You've done all these protections, you've implemented your best practices, you've come up with, you know, your comfort comfortable deployment scenario, and then, uh, the red team tries to sort of break all of that and say, oh, well you didn't think about this, right? You didn't think about a, a user interacting with it in this novel way. Um, and then sort of, I think that really one of the key things that we wanna spend a lot of time developing, uh, throughout the week and as we write this up, is defense in depth and what that means.

You know, that's a core concept in software security. I think it's a core concept here as well. And so what does that look like for these AI models?

Um, and so, you know, things like having points of control, you know, deploying controls, but also testing and monitoring, uh, restricting access. You know, what, what sort of API are you making available to your model? How permissive is that API?

Does it give you, you know, the top five results? Does it give you the probability of the top result or does it just give you the top result? Like those matter from an attacker perspective, The challenge is the technologies being relied on to build AI models are still immature.

Well, I also think it's important for us to realize that these models are really primitive parts. Yeah. And that the technology's really not ready for what was released to the world.

Yeah. And so if you think about it, like you're gonna use chat GPT as part of your assistive technology for, you know, dialoguing with a customer who doesn't know how to talk to A GPT, Ultimately, AI is going to be all about establishing reliability and trust. You know, to your point, Ben, or you know about things like the word security, the word safety.

What are we really trying to get to? And, and I think it's, you know, about value, actually. What value are we trying to instill, right?

Because risk is, you know, they talk about it as a cyber insurance problem. Value is a whole different problem, right? And I, I think of it now as a resilience word, I think of a lot of those words now co merging into we're actually trying to create durable, resilient, you know, customer advantage into our products and capabilities.

Let's say you have to give permissions to 20,000 documents. The LLM is suggesting whatever, you know, needs to be done. You see like, it's the trade off.

Am I gonna press 20,000 times? Yes or no? So you verify a sample, you say yes, and then go, uh, and after a while you're just saying go, because you know, you start trusting the system, right?

Uh, and that's kind of like, yeah, people want to have the human in the loop, but as you know, as soon as you start showing some things that they start trusting, they're very fast that giving up any of those. And it's the same thing with the pipeline. I keep telling like, people trust the pipeline because most of the time it's working.

So it builds up that trust. But in like, in essence, it could fail as many times as one. Exactly.

That's why it was designed. Uh, so yeah, they're very easy to give up that human In loop. It's really interesting to see that a customer just isn't gonna care as much about all of the things they're gonna care about.

How do I leverage an AI to do everything for me? 'cause that's gonna be their interface going forward. And I think that's the, the most interesting piece of doing a worldly map on this is what is the customer gonna care about?

And you know, what the customer's gonna care about, they're gonna care about, Hey, I wanna use something from a group of people that represents the best knowledge around my specific domain. And if it's flawed, I'm gonna have to have a way to tune that flaw out. And the reality is, when you looked at generative AI and Chad GPT coming in and all these things, they trained it on a bunch of, I hate to say it, junk data.

And you're getting things out of it that are more junky than you'd like, right? So hallucinations and all this stuff. But if you put good data in and you put real experience and real information into these things, they become super valuable, especially when they're talking about access controls being baked into results.

Because that means that you're gonna have to force the model itself to start to be cognizant of cognizant of the access and the consent as part of its work. And where's that coming at? Models shouldn't have to have that type of p and I think there's a layer That sits, but the data that if you're going after transactional data, transactional data does already carry some level of consent.

But I think that consent's gonna go deeper. I I don't think you're gonna be able to like just give away your data anymore at, I think knowledge is going to either become part of your PII or it's not. I Really want to drill down on this because when you start using AI to support functional roles in an organization, that's basically what this is, right?

You're abstracting part of yourself to live on. All these things are like anecdotal testing and then quantifying whether the results are correct or not. That's kind of a, a big challenge right now.

So, you know, we live through one model, two models, but now you see everybody trying to figure out, because the manual labor of kind of working through what the model wants is just, uh, very costly. Yeah. Yeah.

It's not, it's statistical. And to your point, the measuring stick, as long as it's kind of quantitative and equal throughout all the players, it hopefully it's statistically wrong or right consistently. But to your point, we've gotta figure out, um, multiple mechanisms on how to rate these things.

And there could be multiple rating systems on did this, you know, how, how closely did this align to our policy or Right. Should it have missed this or missed that? They're still humans crafting the, the prompts.

We're still engineering the prompts. Yes, there are frameworks now too that are very fledgling, that are helping us to create better prompts, but it's still, we're not yet feeding it. You can get 200 prompts on Nsa.

Yeah. But, but it's not yet feeding. Right?

We're not directly influencing and having one framework influence another framework quite yet. We're, we're right at that precipice right now. It is time to set off on the next phase of the journey.

I think we're on a great glide path. We've got a lot of interesting ideas, and we really still have to, I'll do a lot more work to pull this all together and make it real for industry. We are on to the second full day of the operationalizing AI hackathon, and the teams are starting to coalesce on the mission.

You know, it's been really, really awesome to be here. And the project is doing really well. I think that being part of a group that's talking about AI thought leaders that are using these products, doing stuff with 'em, uh, it's really helpful because you get a perspective in a community to be able to understand more clearly what some other folks are doing, or even innovation opportunities.

There's A lot of people with different backgrounds, different knowledge. I think the first day was about, you know, a lot of setting the stage and people getting to know like, you know, what are you interested for? Like, what's your knowledge?

What, how do we bring this together? And so now the question here for today is how can we bring this all together into a outcome for, uh, people to use? It's now time to get to work.

Right now I'm getting my, uh, new laptop set up with Python. I've also synchronized with the GitHub, um, the Gaia Hackathon project. Um, Patrick's working on stepping through a number of different, uh, Python, uh, scripts.

He's kind of a tutorial in nature. I'm also taking side notes at the same time, so then we can merge them together and be able to help other people with this. And what I'm really looking to do is I'm focused on reproducibility at this point, and that puts us in a position to start, uh, benchmarking this with respect to hardware.

I think we're addressing, uh, a, you know, the broader population and technologists, how do we make this welcoming to people who are less technical, uh, make it accessible for, uh, folks in organizations to augment their skill set, but also with the more advanced crowd, getting into the details on performance, Of course, the best part of most journeys are the discoveries. What I tried to do in the examples is like debunking kind of this fuzziness and, and you would see a lot of the tutorials saying like, you know, Hey, I use this call, blah, I got this return. I was like, but what's it doing?

Like, explain this to me. And that's, I think what took me also a long time to understand like, what's now going over that wire? Like nothing.

Like there's nothing special. And it's kind of like that. What About the caching?

Tell 'em about the, the way you could do the private caching so that you could get faster Feedback. So one of the examples, like of the show was, uh, like the local development setup. So a local LLM together with a local embedding model, but that, that's slow right?

On your laptop. So we, we we're like putting the caching layer in there so you would get a result. But then I also show that if you're asking a question, uh, typical caching would be I asking question, but it gives based on the text, the result, right?

But I, I used embeddings to kind of show that if you're doing similar questions with a kind of small variations in the text, it will still find that in the cache. So it kind of like using this, uh, as well. Eventually, however the destination starts to come into view, It's really interesting perspective on runtime and how we could bring the governance to security from compliance.

And so like answering questions or being able to do, like, can we like pragmatically rebuild a model, call it like a gizmo or whatever. Um, if source X is bad, so say we go build something and like one of the sources of the 10,000 are used, we find out as erroneous or bad basically being able to say, okay, now we can rebuild this. We've identified it so we can sort of rebuild it.

If you think about like sort of rebuildable software, you think about some of the, How are you gonna determine it was that source that was bad? We'll, we'll show you, oh, I'm curious What, no, like I, someone, someone's, someone's gonna identify like, this won't say this a bad source, but once you've said, this source is bad, we'll I'll do that. Oh, you're gonna use feedback to basically originate that.

Or like, we'll show you. Go back. Um, so as you think about a verification process, we run against in sources as mini sources.

We sign the source, like we sign software once we verify it. So now we have different, we have these signatures and those signatures represent. Um, and then as you get towards the model, the model itself are inside this little middleware.

Um, we said the model here, but uh, we sort of updated middleware. It verifies the analysis has been complete and then it actually looks at the signatures. And so it's like this two part verification to say it is what I'm ingesting actually been, um, something that's in, in the repo itself.

And that's how you sort of use that transparency. Um, verification considerations. Can this target, you know, can we verify this target can be used?

And again, at the very beginning we talked there, oh, direct versus indirect predicate. So we talk about the privacy predicates. There's a predicate that says, Alex can use this specific Google doc, but then you have indirect that says, Hey, this model can use Google docs that are by this organization, you know, that have metadata.

So you're not directly saying this one, but you're saying these types. And so as we're getting down to describing what this is, and as we're building to is a document and an approach on how somebody could implement this in a, a fairly, uh, gen generic model of how you control this and they can be applied at blood time. Um, one, the thing here we talk about architectural gap is the, basically can the LLM or the gizmo, the model report information about itself, such as, um, and you go into this, 'cause you had a really good idea about this one.

Well, I mean, it was just basically nature at large. You know, there's gonna be a lot of different, uh, developers that have to different business cases, creating their own LLMs, their own kind of gizmo AppSec. And, uh, so in the large, like how do you discover these and how do you, you know, match 'em together?

How do you create reusability? So, you know, the, I you saw a little bit of that thinking about a repo up there. So, you know, I think you could think of that repo is also sort of an app source you can consume from it.

And it's all, you know, based on the kind of metadata that that goes with these artifacts, you know, maybe through manifests and that, you know, as you load things, it starts to advertise itself in a directory space. So, um, yeah, we're just calling that out as a proposal slash gap that we hadn't really talked about earlier. Yeah.

Since we're collecting on this information, you could, it could self-report and say, here's my features, my functions, and then other people can self-service. So these definitely seem like great best practices for, you know, designing a system with, uh, security and governance in mind. I'm curious, like when it comes to these controls and the predicates and so forth, how much of it do you think, uh, needs to be sort of accustomed to this use case versus could be implemented using existing access control scheme?

So like, if you're pulling data from, you know, S3, like you have the whole policy language there to control access and you know, you have policies on, on Google as well in terms of like, you know, groups that have access to particular folders of data, right? Is it really like a methodology for mapping your policy onto the access control scheme for your data source? Or do you imagine this being a separate agent that you know you need to run?

It's, so I haven't thought about it like that, but let me, let me go off the cuff real quick. So I think of this as a separate agent and think of like this privacy predicate as a wrapper for something like that. So does this person have, if we're talking about person source target, does this person have access to this material in this, in this bucket?

So as you think about it, because you have different systems and sources you're pulling from, the problem we're trying to solve is you have a lot of different untrusted areas. You're not always pulling from one, but it could be a ton of different things. How do you provide sort of a unified description of what you can do if it's S3, Google's version of it, and all that type of stuff.

And so what I looked at this as sort of that standard that, uh, a one homogenous slayer that can consider all the heterogeneous, um, source, uh, storage areas that you could get your sources from such that you, you may have, like, it would basically say you may have some hard stops with S3, but even in there that could be verified. One thing we covered was the difference between the first line and second line of defense. And a lot of this, if you guys are familiar with the se first line of defense, the three lines of defense, a lot of this is automating the second line of defense.

So while the, like the first line may be S3 says, you know, this type of person can't have access to this. The second line is validating sort what the first line's doing. So we're taking that similar concept here and make making sure that, and applying that holistically.

So what we can do is we can report the privacy predicates how we, how we did this. Yeah. Yeah.

I'd say one thing to keep in mind as you sort of build that out and, and market it to people is like, uh, potential pushback around like, wait, so now I have to maintain two sets of access control policies and make sure they're consistent and everything. You know, like I, I mean I've lived in that world. I, I guess my first reaction is, I think this is more at first best practice and its capability sets that maybe you could, uh, approximate with existing tools and maybe vendors sort of converge on something that, that really spells us out.

Oh, this was, we started documenting something out. So we think about our gizmos, like the extensions and lms, we're using a lot of the gizmos. 'cause we think that you have users or other autonomous agents, we use gizmos.

You have this dependency and we're starting to, this is where we start to separate runtime from build time, which actually came more into this. And so as we start talking about authorization and privacy at the top, right, proved to me X there's two things that we're thinking about here. Like user or what Alex was bringing in was like, a lot of this stuff could be autonomous agents.

So I didn't wanna show gizmos talking to gizmos. Well, it technically could be a gizmo. Alex really brought in the runtime perspective and I was really focused on the build time.

And so I thought this was interesting of like, hey, how do we prove that X request did not use y type of data? So if somebody comes in and says, eight months ago, I wanna audit your stuff and whatever it is, this couldn't use this private type data, how could we prove that? And why did that come up?

I'm working with a couple of early stage companies that are building digital agents. So a a couple of early stage companies that are building digital assistance, I should say, which are basically agents. And I'm seeing operational lease questions come up from some of their prospects on this kind of thing.

So I'm just like, it's already there. Let's just put it in this picture. One way you could prove out this model is like, I think one of the hard things of today is access to files.

Some of the convention, like I ask, where's the transcripts? Like, oh, it's in the drive. Great.

So who has access to that? Everybody care. Should everybody have access to it?

I don't know. But you could put in a model that says it's explicit versus implicit. The one I thought was the actions too.

So we were talking about the digital assistance. Assistance may do more than just provide information. They may actually start taking actions on your behalf.

And so when you start to ask like, prove to me the action was what was expected, right? Not that it was wrong or right, what was, what was expected? And then tracing that back down to source material.

So as we think about two layers of authorization, they're sort of like, the agents is like, can I use these gizmos? Right? A lot of these may be out there, so you have, can I use these gizmos?

But then you have this model or these gizmos at this model gateway, can these gizmos use these other resources, which could be other large or small language models or other gizmos inside of themselves. So how does that prove that the action the gizmo took was expected? I think that's more, that's not shown in this diagram.

I think it's more just, okay, I think of that more as a kind of new bug. Yeah. So that's gonna go through the C basically.

Don't even know exactly how you, I mean, unless somebody exposes, Are you talking about is there heart back for the kinds of actions agents can take? Is that what you're asking? I'm not, not, Uh, so just how do I know that the action was expected?

The gizmo action that you selected? You know, I wonder if we restate this. So like of an action taken, like as we're looking at this, actually, lemme go to this sort of slide here, what questions we can answer with this.

So as you start to look at this as a graph of your ultimate sources, the models that built based upon versions, and then as it starts to go up, we can now say whether this user or autonomous agent, we can say, based on what happened with them, we can record what what they did. We can now trace it all the way back to the sources over time. And this gives us the ability to answer four types of questions.

A top down question like, is this model compliant? A bottoms up question such as what models have been trained on this source? Also, we talk about depth and breadth since it's, since it's graph.

Now you have depth questions of was this model trained on a specific source? And you have breadth questions of like, was this, were these models trained on this source? And so as you start to answer those types of questions, and we talk about governance and compliance and security, we're, we're bringing this all into an ultimate safety.

So this bi-directional traceability thing that, that you're basically pointing out is one of the reasons it's so interesting to me from, from more of a system perspective. Most cyber physical systems also require this bi-directional traceability. One of the things that I'm really trying to show is that you can use agile and DevOps on cyber physical systems.

One of the biggest pushbacks from the community, the, the safety, let's say, um, the FAA or you know, air force or whatever, is how do you prove bidirectional traceability? I wonder if there is a way to use something similar as we're getting requirements to be able to prove that These are similar ideas as we're coming out. It's basically the underlying philosophy is this is you have some known state that you can cryptographically prove and you've basically versioned it.

Yep. And then you're basically saying, I know whether it's these nodes, whatever those nodes are, I know the state of this node at this point in time. Yep.

And I do know. And so then what you start to do is you basically make relationships as a first class citizen there. This node contains this node or these nodes of these versions.

And so, um, and the, the way you have to do it autonomously automatically to actually have it be effective and efficient, what we're working on right now is a piece, a simple piece of software that your source target and predicates where we can basically load a list of sources by URLs, create a list of predicates and model different targets. So we can have a private and public target and then we can predic then we can put additional sources in there and classify 'em as how we wanna classify it and prove that says, hey, you can have this or not. So the idea is we could then plug into sort of what you're doing and then also through the pipeline to provide that layer at these different pieces.

So when it goes to build the model, it basically goes through our, our middleware, we'll call that middleware and you request a hundred sources and we only return to you what you have access to. And then we actually generate a report of what we've returned to you. And so then you can start to do that.

There will be bumps along the way. Oh, I think you're in the right direction. Directionally this is like fan freaking fantastic.

Uh, I just think that for the world to catch up with where this would go is, um, a hurdle. So the question is like how to enact it in a way that could have lineage and some of these capabilities that are less than multi-billion dollar company level would be really useful. Oh yeah.

It's a comment. So like if they're gonna bring AI into their company, could they start now with better practice? Like, hey, if you're going to give everybody in your company, you know, Google Drive or OneDrive or any of those things, you know what things have to be applied to their data for it to be able to be consumed at the AI level and what lineage has to be put in place.

Could there be a set of best practices for how they would enable that capability? There's gotta be a crawl, walk around. And that's what I'm saying, the crawl perspective.

I think you're right. Best practice. So like, hey, for anything that you're gonna contribute to an AI model, like there's a AI folder in your like drive if you will, if you're gonna, you know, be an analyst.

If you're gonna bring in big, large data, is it that, you know, something in there has the RAC model associated with it properly? Like I think you've got some of those that if you were to put those in place, those are the Yeah, that's the influence that has to happen in the industry for this to really take off. Is it, you actually have to go back and get everybody to agree in sort of how they assemble some governance at those layers.

Because honestly, most of those providers are like, here's your drive, good luck. And that's just not, that's, it's unfortunate because it's gotten us into a dirtier place to be and that's gonna cause us to have to spend a lot more investment. So I, I think it's the right thing.

I think for all the privacy of Mavens would love this. And I think it's a really, really powerful, you know, process that you put in. But I, I think it's gonna be practically challenging to implement unless you come up with some strategy for the implementation.

But the end is in sight. We're doing OI hackathon, we're actually more of talking about some useful, we're we're trying to actually build value. That's, I put this thing together, right?

And I think it's like the roadmap for building value with AI is actually what we're gonna put together and it's gonna make it so people could not have to do this for themselves is super valuable. Like, so we're talking about operationalizing ai, talking about OI, so I feel like this could be a proposed term as we're going through it. And, And so as I sat here and I just as a look and every teams, I was thinking last night and this morning like, hold on, we actually have all the pieces that we can put together with what Joseph's provided as a basis and then hitting on the concerns.

And so really why, um, the deliverable is a cos it needs to be cohesive. Um, and if it's well done, this is, this is my assertion. If it's well done, we have an end-to-end approach that technical and most importantly non-technical because a lot of us are technical first, non-technical people can reference as their gold standard to include as their aspects as they begin their journey.

So my mind is, we're gonna talk to a lot of people talk to folks, tech will take this, but they need to bring the non-tech along. So when a company says, Hey, we want to do this, they can basically go to whatever we call the output of this and be like, look at what they did. You can take, and with all the world's foundational models take, you could specify.

So any company could take what we do and they could start to build something for themselves. Now it's just a matter of setting expectations. It's a hackathon, right?

So which means like, we're just trying to get things like kind of structures set out. So I don't necessarily always expect a full product at the end of a hackathon, so to speak. So I guess it depends on what your expectation of the result of a hackathon is.

So if it's something done, all that, I don't think so that's not good expectation to have is what I would say. But if it's like just more or less like an outline MVP or like a structure for something, then I think we're doing great. 'cause we're working towards a Structure.

It's been three days of intense effort that in addition to code that can be shared, has deepened everyone's understanding of the challenges and opportunities ahead. For now, however, it's time to wrap things up. So It's great.

It's, uh, three days we're wrapping it up. We were trying to find our way. We worked on a lot of stuff.

I think in the end we have some good documentation. We definitely have a ton of really good code that we're gonna be putting out. And we've got some of it already out there.

We're gonna be making it public, cleaning up, document it, write some blogs. We may create a discord channel. Uh, but so you'll see what we've done and we're gonna try to build this, this is like a community.

We're gonna build like what the DevOps people and infrastructure people of how we need to look at the growth of generative ai. It primarily in the enterprise large, you know, critical safety, um, high consequence businesses. Well, so, and yeah, it was just a blast.

Well, I'm hoping that, um, in the next six months that, that we can really develop some sort of standard methodologies of promoting things into production. And if we can do that, then I, I think it'll be a big sense of, of safety for the people at large enterprises who want to take this technology and adopt it. I think there's a lot of fear in the world.

So in six months, I'm hoping that at least there's sort of a codified sort of set of best practices to promote a foundational model, um, from its, you know, state and development and all of the surrounding APIs and promoting that into a production use case in a safe way. And that we've sort of developed a framework that we can all agree on will get us from, again, the beginning phases of development to operations. Um, the, the other, the thing that that really scares me, um, if you will, is that how fast is the technology gonna change though in the next six months?

That's the only, um, thing I'm concerned about because things are moving so fast. So it's a moving target for us. There are of course, some key lessons that have already been learned, I would say to engineers, it's like, start thinking, uh, like find your own pet project, whether that's, uh, kind of your, your personal hobby or anything there.

I I, I strongly believe that by doing it, you'll start on more understanding the behavior. Uh, and we know we need more people to kind of like just playing with it. Um, obviously there's downsides in security and issues that we need to avoid, but I think the more people will tinker for it, they're, they're, um, it's actually gonna improve.

So I'm, I'm very hopeful on that. Uh, and don't let it scare you that it's like all you think, it's all about mathematics and all that stuff. Those ages have been gone.

It's, it's all about integration and making it more accessible for people to use the technology. So I think anybody could start right Now. The most important advice, as always is to not forget your DevSecOps fundamentals.

Uh, this is all the wild west, uh, and everyone's really excited, but we need to remember our fundamentals today, now that folks have code checked into repo. Uh, even with all, all the expertise we have in the building, uh, running quality scans, dinging on code coverage, we don't have testing set up, um, how we're doing secrets management. Uh, even if it's in a, a Python notebook, how, how production ready is a, a Jupyter notebook versus actually breaking it down into a script, uh, and, and what the audience of it is and be.

That is where my warning is, is just there's so much that is proof of concepts hasn't really hit a minimum viable state for production use. And, uh, a lot of folks new to this space, uh, aren't aware of that. Just remember best practices is, uh, be cautious of how quickly you move to production, uh, with one of these tools.

Yeah, I would suggest, uh, that they get familiar with, um, some of the attacks and vulnerabilities that are unique to these systems. Um, obviously there's traditional security principles that continue to apply to any software based system. Um, but there's some attack vectors that are unique, uh, to these AI based systems or to generative AI in particular.

And so it's kind of a blend of old principles, but new ways of thinking. And so some of the old principles are, you know, how do you manage change and how do you, um, bring data more into the process. Uh, so that's sort of like traditional ways of thinking, but new ways of thinking really comes down to a new software development paradigm.

And, uh, that's a new game. And you know, and that's an exciting place to be. And some of the folks here came already, you know, with some skills, uh, learning how to use these, these AI development tools.

First off, look at it like you're creating another piece of software. Um, bring rigor to it to start with and definitely start small, but find a real business case to start with. So there's a lot of fun things.

You hear the lang chain, you know, Dali, yo llama, all the things and there's tons of tools and more coming out and it can probably be very cumbersome to figure that all out, uh, to start with, especially how early on everything is and how fast the whole industry around generative AI and AI is changing. Uh, but get started finding business use case. Find a something that is probabilistic in nature.

So something that, you know, I can't go to a piece of paper and say, does this value equal this value? But something that's probabilistic and identify what and how you wanna solve it, and then start to work your way backwards and then only integrate the software you need to integrate, uh, to achieve that outcome. Because you will lose yourself in trying to play with all the nude nice, shiny things.

Um, and you'll most likely not solve what you're trying to solve. I had a confirmation that this is an ocean. There's so much for us to learn.

We are very much at the beginning of this. It's embryonic. We have to look across all the different dimensions, all the things that we've learned in the past.

We can't forget those. But now we have to figure out how to apply it to leveraging generative AI to help us build software, but also to give us really cool new end results. And remember, AI is a journey that we're all going to take together.

My best advice for other people building, um, AI technology is just to be open-minded, um, and not to be super set in their, in your way about how you're gonna implement it or anything. As of yet, the technology I think is still quite new. Um, there's more coming every day.

So if you're not, if you're not set and if you're open-minded, you'll be able to make best use of what's coming in the future. Don't com be complacent and think that what you learned last week is still gonna be relevant next week because things are changing at a faster pace than anything else in technology. So keep on learning, keep up with the trends, um, that things aren't gonna settle down for a while.

Find and create communities of people that also are exploring AI and understanding it. And I think once you get a little bit more depth, you don't have to be an expert, but that's gonna inform, okay, let's think about our business and our products or what I in, in my life or my consulting practice or whatever it might be. I think you'll have a much richer understanding to be able to kind of go farther faster.

And so what I would say is if you are learning AI or even considering learning, um, reach out to, you know, the wider network. It's very sharing and, and just start, right? Don't, don't be intimidated because actually when you break it down into its basics, you can actually accomplish something.

You know, begin with hello world and, and keep growing.