What is OWASP Top 10 for LLM applications

What – Or Who – Is OWASP?

As mentioned in their intro, OWASP’s work falls into three categories:

• Tools and Resources
• Community and Networking
• Education & Training

OWASP Top 10 for Kubernetes

In a previous post, we talked about the need for a dedicated OWASP Top 10 for the cloud-native container orchestration platform – Kubernetes. The below controls focus entirely on the orchestration-level of deploying and security web applications in production using Kubernetes. 

This shouldn’t be confused with the broader OWASP Top 10 for Cloud-Native Applications that delves into the extended use-cases of securing the cloud provider, the associated cloud services, asset management within these services, and any potential injection flaws in the application layer, cloud events or cloud services.

The most recent version of the Kubernetes OWASP Top 10, which received some minor updates in 2024, includes the following:

K01: Insecure Workload Configurations

K02: Supply Chain Vulnerabilities

K03: Overly Permissive RBAC Configurations

K04: Lack of Centralized Policy Enforcement

K05: Inadequate Logging and Monitoring

K06: Broken Authentication Mechanisms

K07: Missing Network Segmentation Controls

K08: Secrets Management Failures

K09: Misconfigured Cluster Components

K10: Outdated and Vulnerable Kubernetes Components

Since LLM workloads are often deployed into Kubernetes clusters to avail of the benefits of scalability within container orchestration, there will often be overlap between the best practices specific to LLM-based workloads, and the environments they are running in, such as containers, Kubernetes and the cloud. This lends itself well to forming an OWASP Top 10 for LLMs.

The Urgency of Securing AI Workloads for CISOs

LEARN MORE

OWASP Top 10 for LLMs

The OWASP Top 10 for LLMs(Large Language Models) provides a prioritized list of the most common areas that should be addressed by developers, data scientists, and security experts who have now been tasked with designing and building applications and plugins which leverage LLM-based AI technologies. Below, we’ll list these areas and give a brief description of what each one entails.

LLM01: Prompt Injection

The Prompt Injection Vulnerability occurs when an attacker manipulates an LLM through crafted inputs, either by directly overriding the system prompt (known as “jailbreaking“) or indirectly via external sources like websites or files. This allows the attacker to control the LLM’s behavior, potentially leading to data leaks, unauthorized plugin use, or social engineering attacks. In advanced cases, the LLM may unknowingly aid the attacker by bypassing standard safeguards, furthering their objectives without alerting the user or triggering security measures.

LLM02: Insecure Output Handling

When outputs generated by the LLM are not properly validated or sanitized before being passed to downstream systems, the program potentially exposes vulnerabilities to the business. Since prompt inputs can control LLM-generated content, this risk is similar to giving users indirect access to additional functionality. Exploiting insecure output handling can lead to attacks like XSS, CSRF, SSRF, privilege escalation, or remote code execution. The impact is heightened when LLMs have excessive privileges, when applications are vulnerable to indirect prompt injections, or when third-party plugins fail to validate inputs properly. For example, LLM output entered directly into a system shell, or functions such as exec or eval could result in remote code execution.

LLM03: Training Data Poisoning

Data poisoning in ML refers to the manipulation of training data or datasets used in fine-tuning or embedding processes, which can introduce potential backdoors, or biases that compromise the model’s security, effectiveness, and ethical behavior. Poisoned data can result in harmful outputs, performance degradation, or even reputational damage. This attack impacts the model’s ability to produce accurate predictions and is particularly risky when external data sources, which may contain falsified or biased content, are used in training, making it an integrity attack on the model.

LLM04: Model Denial of Service

Attackers can potentially exploit your LLM by consuming excessive resources, degrading service quality for users and incurring high costs – no different to a DoS/DDoS attack on other web applications. In the case of LLM-based web apps, the rising security concern involves manipulating the LLM’s context window, which is the maximum length of text it can handle. This can lead to unpredictable behaviour with the program. As LLMs are increasingly used in resource-intensive applications, many developers remain unaware of this vulnerability, which stems from the model’s architecture and defines how much input and output the model can process at once.

LLM05: Supply Chain Vulnerabilities

The supply chain for LLMs is vulnerable to attacks no different to the Kubernetes supply chain components. Outdated supply chain components that are abused can compromise the integrity of those training models. Aside from leading to biased outcomes, the organisation could suffer security breaches or system failure directly from these abused vulnerabilities. Unlike traditional software vulnerabilities, ML programs introduce risks from pre-trained models and third-party data, which can be tampered with or poisoned. Additionally, LLM plugin extensions pose further risks, as outlined in LLM07 – Insecure Plugin Design, which highlights the need for secure plugin development and thorough evaluation of third-party plugins.

LLM06: Sensitive Information Disclosure

LLM applications can inadvertently expose sensitive information, proprietary algorithms, or confidential details through their output, leading to security breaches and privacy violations. To mitigate this risk, proper data sanitization should be performed to prevent sensitive user data from entering training models. Clear Terms of Use should inform users about data processing and opt-out options. Since both input to and output from LLMs cannot be fully trusted, adding restrictions within system prompts can help reduce data leaks, though these measures may not always prevent sensitive information disclosure due to the unpredictable nature of LLMs.

LLM07: Insecure Plugin Design

LLM plugins are extensions automatically triggered during user interactions, often without the application’s control, especially when hosted by third parties. These plugins can accept free-text inputs without validation, allowing attackers to send malicious requests, potentially leading to data breaches, remote code execution, or privilege escalation. The risk increases when plugins lack sufficient access controls and fail to verify authorization, blindly trusting inputs from other plugins or users. This vulnerability specifically addresses LLM plugin creation, as distinct from third-party plugin risks.

LLM08: Excessive Agency

This happens when the LLM-based system is granted too much functionality, permissions, or autonomy, allowing it to take damaging actions in response to unexpected or ambiguous outputs. This can be caused by issues such as hallucinations, prompt injections, malicious plugins, or poorly engineered prompts. Unlike Insecure Output Handling, which focuses on scrutinizing LLM outputs, Excessive Agency stems from the system’s ability to perform actions with insufficient oversight. The impact can range from breaches in confidentiality to system integrity and availability, depending on which systems the LLM interacts with. Sysdig documented how to abuse Excessive Agency in a real-world web application.

LLM09: Overreliance

Overreliance on a pre-trained model can lead to significant issues when it generates authoritative-sounding but incorrect or unsafe information, a problem known as confabulation. For example, if the pre-trained model produced flawed source code that was integrated into a different web application without proper review, it can introduce glaring security vulnerabilities. This risk kind of highlights the obvious need for rigorous oversight, continuous validation mechanisms, and clearly-defined disclaimers about potential inaccuracies.

LLM10: Model Theft

Unauthorized access and theft of LLM models by malicious actors pose a serious security risk, as these models represent valuable intellectual property. When LLM models are compromised, copied, or their weights and parameters are extracted, it can lead to significant damage to your brands reputation, or worse yet, the attackers expose your models’ sensitive data to the public. To mitigate these risks, organizations can enforce better access controls, as well as encryption for traffic in transit – but also at rest. Like all of the other controls listed in the OWASP Top 10 for LLMs, continuous monitoring is needed to properly identify drift in your LLM models, and ultimately to safeguard your intellectual property.

The Growing Dangers of LLMjacking: Evolving Tactics and Evading Sanctions

LEARN MORE

Security & Governance Checklist

The OWASP team has introduced a Security & Governance Checklist to assist Chief Information Security Officers (CISOs) and other security leaders in managing Generative AI (Gen AI) technology within their organizations. As many are concerned about their preparedness for this emerging technology, the OWASP Top 10 for LLM Applications Checklist offers guidance across executive, technical, cybersecurity, privacy, compliance, and legal domains. This resource is designed for those seeking to navigate the rapid advancements in AI while mitigating risks associated with insecure implementations. It helps leaders and teams understand the risks and benefits of LLMs and develop strategies to enhance their defenses as the industry is still in a Race for robust AI governance to address the evolving threat landscape.

Next Steps

With a clear understanding of OWASP Top 10 for LLM applications, you’re ready to advance your AI workload security efforts. To continue your journey, consider engaging with the OWASP community. Join the OWASP Slack working group by signing up and participating in the #project-top10-for-llm channel to stay updated and collaborate with others in the field.

Also, read our blog on LLMjacking – an attack targeting LLMs discovered by our Threat Research Team.