Artificial Intelligence and Cybersecurity | AI in Action 2023

Hello, my name is Caroline Wong, and I'm so delighted to be here today with my friend and colleague, Adam Lundquist. Today we're talking about the implications of artificial intelligence for cybersecurity. These are significant and multidimensional.

And today what we're gonna do is we're gonna talk about risks, predictions, and recommendations. Adam and I are leaders at the security company Cobalt, and we are going to be highlighting the most likely changes to the cybersecurity testing landscape. Again, my name is Caroline, and I'm Cobalt's Chief Strategy Officer.

I've been working in the cybersecurity field since 2005. I got started at eBay and at Zynga, I'm the author of Security Metrics, a beginner's Guide. I host the podcast, humans of InfoSec, and I teach cybersecurity courses on LinkedIn learning.

Adam Lundquist is the director of engineering at Cobalt. With a focus on the nexus of AI and security testing, Adam leads our data and infrastructure teams championing the integration of state-of-the-art AI technologies, and amplifying the capabilities of Cobalt's pen tester community. Adam's profound understanding of the tech landscape is shaped by his dynamic evolution from developer to director over the last, more than two decades beyond Cobalt.

Adam is the founder of playbox, a platform dedicated to elevating development teams. His vast experience marked by additional roles at Nexus Group and Urban Mobility Innovations underscores his deep experience in AI security and software engineering. Adam, it's super great to be here with you today.

Mm-Hmm. Adam, can you tell us a little bit about your perspective on the current state of the art, artificial intelligence, and some of the major impacts? Oh, yes.

Uh, this is super exciting. Um, and, you know, I love to talk about AI and these things. So, um, if we look back just a little bit, uh, AI has made a huge advancement over the last year, uh, also over the couple of years.

But, um, what we have seen over the last year has been chat, GPT by open ai, which was a huge step forward, right? It knows a lot, even surpassing humans in knowledge. Um, and then with GPT four, we have seen big improvements, uh, in particular, I would say, in its emergent skills like, uh, reasoning and planning.

But I think the biggest thing over the year is probably the multimodal AI that can understand and generate both images and written language. This is a really fast and amazing development, um, and these new AI models, they are leading to really cool new things. So if you add a large language model to a software system and you equip it with tools, we get something that is called a autonomous agent.

We've seen this, uh, being, developing over this year, like for example, out the GPT, baby ADI and just recently released by OpenAI, the GPTs. These systems are designed to fulfill tasks and needs for people. It's important to remember that AI doesn't always get things perfectly right.

We know it, uh, it hallucinates makes mistakes. It's not really good in math. Um, and to make this work well, um, the AI need guardrails.

They, they need clear instructions. You need to set the stage. When you communicate with an ai, you need to give it the context, and you need to, uh, be careful with error handling if you are, uh, working with the results out of it, right?

But even with these limits, they can do a lots of things on their own, on their own, and that's where we are right now. The progress in AI this year, uh, or the past year has been very exciting. And I think for, for me and many others, actually overwhelming, we have seen super big technical, uh, advances, but we're still waiting to see how they changed society.

Previous year has been lots about the new AI fundamental models and simple applications built upon them. And I think what we'll see over the upcoming year will be these autonomous agents, uh, see them changing the way we do many tasks. Soon, some tasks will be performed almost automated in a fraction of the time.

And I think in some areas, the productivity can really explode and, uh, be really, we, we can achieve a lot, right? But the pace of this change, it holds for regulation, right? Um, there are, there are attempts at this.

Uh, and, uh, the primary focus, I would say on these regulations have been on the worst case scenarios. Uh, they are super important to, uh, regulate and, uh, I, I really appreciate those initiatives taking place, but I think we also need to think about the short term challenges. What we need today is a thoughtful approach to how the advantages of AI are attribute or distributed across, uh, society.

So yeah, I think we have lots of things that have happened. We are in the middle of something quite big. Wow.

You know, Adam, I feel so fortunate that I've been able to learn a lot about you and AI and what you have to think about it. Um, and I'm delighted that today we are here with the tech strong community at AI in action, sharing it even more broadly. Adam, tell me, I've heard about this concept, which is human in the loop.

And so what is this and what does it have to do with artificial intelligence? Mm-Hmm. Yeah, human in the loop.

It's a really important concept when we use and think about ai. So it means that even those AI is smart and can do a lot, we still need people to make the final call, uh, and keep eye on things. So today, AI isn't perfect.

We need the safeguards. Uh, it often makes mistakes get things wrong, uh, hallucinates. So we need humans to check the AI work today, right?

To, to, to make sure it gets things right. But this is the concept that, uh, will stick because moving forward, even when AI gets better, we still need the people involved. And, uh, take for example, national Defense systems.

They will use ai, they probably do today, but, um, they will use AI to quickly spot dangers if we want the person to make a big decision like defending against the threat. Um, this is important because a mistake in these situations could be really serious. So, and if we, we want to trust humans, we do trust humans, and we trust them to be more careful and thoughtful than an ai.

So therefore, having a human in the loop, it's about making sure that AI is used safely and responsibly. It's not about preventing mistakes, which might be a challenge today, but perhaps not in the future. Uh, but it's also about making sure that we use it in a way that is effective and ethically sound.

Actually, I really like that. You know, Adam, just before we began our session today, we were speaking with our friend Brian, and Brian is actually using some software that helps him using AI to modify the production of his audio engineering as a musician. You know, and while I think it's a vastly different use case from defense systems, music requires human judgment, human creativity, you know, and this is I think also where the human in the loop comes in, um, for these many things, you know, for ethics, you know, for judgment, uh, for, for art, even Adam, you know, our audience today at Techstrong, it's primarily folks who are very interested in different things about software development.

And I'd love to know from your perspective, what are the possible benefits and advantages of using AI when it comes to modern software development? Yeah, sure. So using AI in software development has a lots of benefits, right?

Uh, it's like when AI helps me to write a book by filling in the details after I give the main ideas, for example, GitHub co-pilot, it can suggest a couple of co, uh, code lines more or less automatically while I am working. That's quite cool, actually. It's, it takes the context and generates code, what it believes I want to type.

That's super cool. Uh, small scale, but super cool. I use, uh, myself, copilot and g PT four quite a lot.

Uh, sometimes G PT four is better, sometimes copilot is more convenient. I really like source graph code. Uh, it's another tool, uh, similar to, uh, GitHub copilot because it knows the entire code base, so I can actually ask it to figure out complex topics within my entire code base.

So if you are a software engineer and listen to this, you must definitely work with the tools and get used to them. So these tools help me a lot. Actually, I was a developer long time ago.

I am not so much today, but, uh, they helped me to transform me from a mediocre developer to a senior one actually. So for me, the return of investment for these tools, it's amazing. And, um, but AI can do much more than just writing code, right?

It can also fix bugs, it can write tests for you. It can spot problems in the code, uh, suggest solutions to the problems, and it can even review changes. This helps, uh, me and others to develop code faster.

And in particular, I think from my perspective, it helps me focus on the more interesting parts of my job, which is actually building cool applications. So, and I keep telling my development teams for years that they need to focus on design and the process for continuous discovery when building products. So they, they need to know what they are building, and they need to do it in a sound way, right?

The main goal should be building the right products, not just writing good code, which is also important, but, um, that is what I have said before, the AI hype to DAO today, this is so much more urgent. So I'm convinced that software developers, they will turn into software builders where the job moves from coding to actually guiding and AI figuring out what the software needs to do, especially in smaller companies who transform a little bit faster than others. I think the role of developers, product managers and designers, they will blend into one more or less the software builder.

This is a transition and a mindset change. It might take time, but in the end, we will make better products faster. And that's super exciting.

You know, Adam, I have to agree with you on some points, and I must disagree with you on one point. And here's the thing. When we are speaking for a group like this, uh, it is often that the people speaking on a panel or in a fireside chat, they always agree with each other.

And I agree with so much of what you're saying, but what I like to do is I like to be a little bit controversial. I like to mix things up. And there is one thing that I strongly disagree with you on.

You know, you said AI helps you to transform from a mediocre engineer to a senior engineer. And I think this is totally wrong. You know, perhaps it allows you to change from a senior engineer to an extraordinary engineer.

Um, but the fact that you described yourself as a mediocre engineer, you know, this is something that I have to put my foot down and I have to say something about it. Thank you. Wonderful.

You without ai, I think That's good. We are, of course, from Cobalt as a cybersecurity company. We are actually perhaps the most cybersecurity focused session at today's AI and action virtual conference.

So let's talk a little bit about some of the different threats that folks should be thinking about. Yeah. So, um, in the end, I mean, cybersecurity, it's about reducing risks and threats, right?

So I think there are two types of threats coming from ai. The first one is that it, AI makes it easier for hackers to do their job. They can make really convincing fake images and texts, which is tough, even for forensic experts to spot as fake.

And AI also helps hackers to target people more accurately through emails and social media. Uh, I think of spearfishing, uh, really targeted, uh, not just spam sending to everyone. Um, and because of this, we really need to be careful about what we believe online.

I think that will change over in the near future, I think, but also hacking tools, um, supported by ai, they are getting stronger. And, uh, so-called script key, this, they can use them by just talking to an AI program and hack a, um, a target of some kind. This means even people with little hacking knowledge can be very dangerous.

And, um, beyond that, I mean, a compute virus may transform itself using AI and go undetected from virus scanners. AI botnets may be able to take over more computers and to deal with this, we need to improve the security posture everywhere, and we need to build better protective tools, uh, and stay ahead of the hackers. So that's the first one.

AI makes hackers stronger, right? The second one is that using AI in our own software products that brings new threats. So today everyone is talking about using ai.

Uh, everyone has respect for the risks with AI for good reasons. So traditional security measures aren't always enough. We're facing new problems like supply chain attacks.

We have had that in the software all the time. We solve that by s om and, uh, other approaches. Now we have that much more in data.

So we have data supply chain, uh, attacks that would be an AI bomb instead. I think that that will evolve. And, um, these are real, these, uh, threats.

There were, there are some researchers at the University of, uh, Chicago, it's really cool. They have developed a method to add some hidden pixels to a, uh, image, which is then used in training, and that poisons the ai. So by adding hidden pixels into images, they were able to poison an AI to trick it, to detect a dog as a cat instead.

Imagine that. It, it's kind of crazy. Um, I hope and believe that open AI and the other big players are one step ahead and kind of know of this, but this small difference between a cat and dog, that would probably not be the target of a hacker.

Uh, they would have other goals, but, so we need to protect the data and the AI models themselves. And later when AI is actually working, that's, uh, something we call inference time. Inference time, yeah.

Uh, um, it can face attacks just like regular software. So, uh, people might trick the AI to get, uh, get it to leak secret information with specific prompts. Usually that's called prompt hacking or jailbreaking.

It may be compared perhaps to SQL injection where you extract or, um, change data. So if you are using these, uh, off the shelf models like GPT four, um, from open ai, then the vendor usually handles these threats and you can lean back and feel safe, but as soon as you even start fine tuning the AI model, or, or you add your own data to it, then you need to be extra careful with security, right? So if you want to dive deep into that, uh, Ovass has, uh, given out a ovass top 10 AI risks, uh, list, which is awesome resource for understanding these kind of things that were the two risks.

But I actually have another bonus risk. Um, the problem is that many people making these AI products, they're still learning. So software developers, they are probably new to AI and data scientists, they are used to AI or machine learning, but they don't know much about building secure applications.

So the lack of this experience can lead to even more security problems, right? So we are opening up another challenge here, and since AI development is moving super fast with new tools coming out all the time, we should give people a little bit more time to learn and keep up with the fast changes in both AI and cybersecurity. So yeah, I would say AI certainly puts, uh, cybersecurity business a bit under pressure.

It's, it's a lot to think about. Um, and of course, we are leaders at Cobalt. Cobalt is a cybersecurity company that leverages a community to do all sorts of security testing.

And naturally, we are often talking about and thinking about what will be the impact of AI on cybersecurity in terms of how solutions and services are going to provided, and how can AI be used to perform cybersecurity activities more effectively. What do you think about all this? I think just fundamentally, I think, uh, we will see AI supporting in all sorts of knowledge work, whether it's an author writing a book, a software developing a new application, or a penetration tester, uh, researching vulnerabilities, all solutions will become more powerful, and that will help people to become more productive, I think.

Um, but if we take some examples, which I find can quite interesting. So, um, if we look at reverse engineering of the virus, this is a typical task probably at McAfee. Um, and the, the engineers there, or researchers, they would decompile the virus and try to make sense outta the scrambled code or assemble they get there, which is a very tough task.

Uh, I am, I'm impressed by these people. The thing is that a large language model can make sense out of that because that is language. The scrambled code is still language, assembler is still language.

So the LLM can help these people to actually understand what they have de compiled. Um, another example is, um, intrusion detection, um, ideas, intrusion detection systems. Uh, they have always, or for a long time, used machine learning to, to achieve what they need to, but they are, they have the chance to get even much better, uh, automation and false positive detection.

Uh, they are typical topics that will save efforts for, for all people operating these systems. Right systems. I, I have a small anecdote that's kind of interesting.

So this spring I was playing a little bit with, um, with chat g pt, it was, and I was thinking, can chat GPT act as a intrusion detection system? Uh, which was, in my opinion in that moment quite farfetched. But, uh, what I did was to essentially take an HTT P response from a request.

So request response pair, and gave it to GPT four and asked, does this, uh, or is this, um, potentially an exploit of a security vulnerability? I did that for quite a large list of HCTP requests and responses, and it was able to identify when it was, for example, an SQL injection attack, you know, that that's something that's an HP request with a response, and it was able to identify that. I am not saying that an LLM like GPT four will be better than the intrusion detection systems of today.

That would be far, far too farfetched. But it's kind of interesting to show how the capabilities of a large language model, it's, um, mind blowing, sometimes it's mind blowing. So yeah, that was just two examples.

I, I think in summary, security professionals will become much more efficient in their work, but we shouldn't forget that also, the adversaries are getting better. Yeah, so interesting. Technology just makes things easier for all of us if you're the good people or the bad people.

Um, you know, Adam, you kind of touched on something which is you do expect that AI is gonna have an impact, particularly on knowledge-based work. And certainly whenever new stuff comes along to disrupt an industry or a particular way of working, you know, folks have different reactions to it. Um, some folks, uh, have fear associated with it.

You know, uh, you spoke a bit in, in our earlier talk about, uh, regulation and the role of regulation. You know, some folks are coming out with policies that say, this is not allowed. Uh, and others will embrace it, you know, and take it on with some amount of uncertainty and caution, uh, as they explore the newest technologies.

And so what's your particular stance on AI and cybersecurity, and how has that changed what you've been working on at Cobalt? Yeah, perhaps I should start by, uh, saying that I, I totally understand why businesses are cautious and, uh, want to build these policies to reduce the risks. For example, there are, there are tricky legal issues with ai, uh, like who owns the copyright when an AI is trained using books.

Also, when AI like, uh, chatt PT learns from what you say or chat with it, that could accidentally leak information, you, you give it some information because you just want to know something, and that is leaked. Uh, there, there have has been evidence of that already in chatt pt. Um, and that's why some businesses are slow to adapt AI due to these legal and security transfers.

But the good thing is that there are different AI tools out there that can perhaps match the company's risk appetite. So at Cobalt, we are a company that tests other company's security, and so we are really careful about data leaks. So we choose an AI solution that mitigates this perfectly well.

So every company needs to find AI tools that suit them and their risk appetite. Um, but avoiding AI completely, that's a bad idea. So, um, AI is moving super fast, and if you don't adopt quickly, then you will be helplessly left behind.

But another aspect actually of this is, um, you know, it's a change. There's a lot of discussion about the AI apocalypse and, uh, this can make people paralyzed. Uh, in fact, actually I spent quite a lot of time thinking about that as well.

And I, I was, when I really spent my, uh, mind in that topic, uh, I was also very scared. And, uh, it took a while. But in my face of recovery from that, uh, fear, uh, I realized that I have to impact the outcome of AI in some positive way.

And, uh, there are material risks with AI if we don't watch out, but we are still in the driving seat. And, uh, however, little Adam doesn't have any impact on politics or, uh, nobody will listen to me if I say we should pause the AI development. Um, but what I can do is to actually join this revolution.

I can only have an impact if I am actually at the frontier actually driving it. Um, participating in where decisions are taken me in my situation. I have the luck to work with cybersecurity, which I think is an area that really matters when it comes to ai.

So, um, but I think if you are keen about where we are heading, then the only way is to, to, to, um, be proactive. And that's where I am today. If we just look at Cobalt as a pen testing company, we, we help businesses to secure themselves actually by uncovering vulnerabilities and, uh, with stronger hackers or the adversaries.

Uh, the threats increase actually very fast right now. Um, and we also help them to discover vulnerabilities in their AI powered applications, in their own applications, which most companies do. So, so at Cobalt, I think my role is very much to help our team to use AI to improve our services, and in the end, keep the clients safe.

I think, Adam, I can't believe we are already approaching the end of our time for our session together today. I have one last question for you, and it has to do with manual pen testing. This is of course what Cobalt is most known for.

And as our director of engineering, I'm so curious to know, what do you think will be the impact, how AI may change the way in which this type of work is conducted in the future? Yeah, yeah. So I think AI will be able to assist us in solving tasks all over the board, all across the board.

In the beginning it'll only be simple tasks, but over time, I think we'll be able to make AI supporting us in ever more complex tasks. Pen testing requires extensive skills, and it's, uh, really thorough detective work to run a pen test. Our pen testers are extremely bright and it, it'll surely take, um, long time or it will take time until AI can really help them a lot.

But I, I do believe that AI can assist pen testers in some areas already today. And, uh, that's why me and our data team are working primarily on exploring and trying out AI solutions for this. So each pen test is different.

For example, testing web applications. That involves quite a lot of manual work, um, to find out the nitty gritty details and finding the vulnerabilities. But every such pen test starts with the reconnaissance and the scanning phase with the pen tester.

Try to learn as much as possible about the application. And this part is today already helped by tools and, uh, these tools can be made stronger by AI and it can be managed by ai. I think it's today a little bit too early to say when such a product can be launched, but we have made awesome progress and we indeed see great potential here.

Looking a little bit further into the future, I envision a future where pen testers fly the pen test, just like a pilot with its copilot and the pen tester, it owns the decisions and takes the, uh, concrete actions, uh, when it really matters, when it can, uh, have a negative impact on the target. But the copilot else takes most of the easier tasks. I think that will be like, yeah, just like a pilot.

I think that will be the future. Yeah, Absolutely incredible. Adam, I'm always delighted when I get to spend some time with you, and to be here with the tech strong community, uh, is an extra special treat.

Uh, thank you so much. I've enjoyed it, uh, and really, really appreciate you sharing your perspective with me and with us. Thank you, Caroline.

It was wonderful talking to you again.