What is Generative AI (GenAI) in cybersecurity?

What role does generative AI play in cybersecurity?

Generative AI is software that analyzes existing data and outputs synthetic data based on it. By consuming large amounts of data and processing it using a neural network running deep learning models, generative AI can create text, programming code, images, and even videos, all based on the input it was trained on.

By training generative AI on data about specific IT infrastructure and from cybersecurity platforms, including information on attempted and successful hacks and breaches, it can be used to predict, detect, and automatically remediate cyber threats. There is a flip side to this, however: the same data can be used to train generative AI on how to craft new malware threats, or to identify potential attack vectors that can be exploited.

Generative AI usage outside of cybersecurity also presents a significant security and privacy risk: generative AI consumes large amounts of often sensitive data, which it may then leak. Like any code, the underlying models and software may also provide a potential attack vector to organizations that do not properly secure their AI workflows.

How can generative AI be used in cybersecurity?

Generative AI is currently used to provide the following functionality in cloud cybersecurity platforms:

• Generating security strategies and policies: By scanning your infrastructure, code, and configurations as well as being supplied with your operational requirements, AI can identify vulnerabilities and generate security strategies tailored to your specific IT environment. Automatically generated policies, reviewed by your security team, provide a more comprehensive set of rules with less potential gaps than solely relying on manual policy creation.
• Automated incident detection and response: Ongoing activity from processes can be monitored for behaviors that may indicate an attack in progress, and scripts to mitigate these actions can be automatically written and deployed. Potentially compromised systems and processes can be isolated, buying security teams valuable time to properly assess, categorize, and fully respond to security incidents.
• Detecting malicious behavior: Users themselves are a common attack vector. They may be tricked into running scripts or performing other actions that open the door for an attacker to gain access. AI can detect these behaviors, even going as far as to parse user input for commands that may have negative cybersecurity implications.
• Protecting against social engineering attacks: Phishing is a common way to convince users to perform malicious actions or run malicious code. An email instructing a user to open a path for attack that looks convincing to human eyes may not be able to fool AI detection tools.
• Privacy protection: If sensitive data (including personally identifiable information) is required for a specific purpose, any information irrelevant to a specific task can be automatically masked or redacted. If data is required for testing or training, synthetic data can be generated using AI so that exercises, testing, and debugging tasks are as close to real-world scenarios as possible without undermining the protection of real users’ information.
• Training: Gen AI can provide realistic and dynamic scenarios that help improve the decision-making skills of IT security professionals.

Benefits of generative AI in cybersecurity

The benefits of generative AI in cybersecurity include increased automation, more proactive protection (including a greater security posture in the cloud), and improved protection against novel threats that are not yet fully understood.

Cloud security platforms (including Cloud Native Application Protection Platforms) that integrate generative AI significantly reduce the time it takes for security teams to respond to security incidents. Generative AI’s ability to summarize data from different sources and identify trends also greatly improves the efficiency of reporting. Difficult-to-understand programmatic output can be collated and formatted before it is sent to stakeholders, so that the meaning of the data can be quickly understood and acted on.

What risks does generative AI pose to cybersecurity?

The risks posed by generative AI to your infrastructure, workloads, and data include:

• Targeted phishing campaigns: Public information can be consumed by a large language model (LLM) and then used to generate convincing emails targeting your employees, either to directly convince them to hand over money or information, or to take actions to assist in an attack.
• Tailored malicious code: Generative AI can help an attacker write malware that isn’t familiar to antivirus.
• Exploit discovery: The same process of using generative AI to identify potential weaknesses in your infrastructure so that you can fix them, can be used to find and exploit them.
• Deepfakes and impersonation: Generative AI isn’t limited to generating text and pictures — it can create audio and video based on a person’s social media presence that looks and sounds just like them. This can be used to trick colleagues into helping bypass cybersecurity measures, transfer money, or perform other actions.
• Data leaks through public generative AI products: Not all threats are the result of a malicious party. Employees who are not familiar with how generative AI works can unintentionally divulge sensitive data in their prompts to public APIs, which may then appear in its later responses to other users.

An often overlooked aspect of AI in cybersecurity is the potential overreliance on it. AI-generated output (for strategies, security policies, scripts, etc.) must be reviewed by a team member who fully understands what is being secured and how it is used. Otherwise, misconfiguration may occur, leaving exploitable gaps, or overly strict security measures (that prevent users from accessing the resources they need) may be bypassed or disabled.

Securing generative AI applications

As the adoption and integration of AI into almost every product and service increases, securing generative AI tools as an attack vector is vital in modern IT environments.

Visibility and training is key to solving this problem. Maintaining oversight over AI tools will identify potential threat vectors, while training your users so that they understand how public generative AI tools work will reduce the incidences of them disclosing information to them.

If you develop your own AI tools (for internal use or for public consumption), there are additional threats that you must protect against. For example, prompt injection allows attackers to coax sensitive information from your models or even run their own code on your systems, while LLMjacking using stolen credentials for public APIs is an increasing threat. LLMjacking is a growing issue for businesses that rely on public generative AI platforms, and is actively being used to steal resources and poison AI models (inserting deliberately malformed or dangerous data), as well as other malicious actions.

If you provide AI services to others, you may also be liable for harmful output, which can turn AI model poisoning from an annoyance to something with real world impacts with potential legal liability.

AI, cybersecurity, and the future

The arms race between hackers and cybersecurity professionals has been amplified with the arrival of AI. AI-powered attacks will continue to be more complex and targeted, and require much less time to develop against newly identified vulnerabilities. Cybersecurity tools will match this with their own rapidly evolving generative AI applications.

Only by adopting IT security tools that integrate AI functionality will you be able to achieve the best possible protection against this modern, fast-moving threat landscape.

Security for AI

AI workload security provided by Sysdig automatically tags common AI platforms (including OpenAI, TensorFlow, and Anthropic) in your IT environment. It can uncover hidden attack paths, and assess activity to visualize the risks across your infrastructure and provide real-time protection.

The Sysdig cloud security platform can also help to expose risks such as misconfiguration, suspicious activity, and potential public exposure of sensitive information.

AI for Security

Sysdig Sage^TM, Sysdig’s AI-powered cloud security analyst uses multi-step correlation, contextual awareness, and guided response capabilities to help security professionals understand what’s happening in their environment and respond adequately.

Strong generative AI can provide responders with insight generation, decision-making, and adaptive problem-solving, helping to accelerate human response and keep up with the evolving cybersecurity threat landscape.