What is LLM security?

Prompt injection

A user engineers an LLM prompt that causes unintended behavior within the model, such as sensitive data sharing, infrastructure disclosure, elevated LLM access, arbitrary command execution, decision manipulation, and content manipulation. This can be accidental or intentional. Injection can also happen directly through the user prompt, or indirectly through attached files and websites.

Limit model behavior and instructional inputs. Specify and validate output formats. Filter inputs and outputs to avoid sharing sensitive information inadvertently, while evaluating contextual relevance. Require manual human approval for actions that require elevated privileges. Handle API token use in code and enforce access control based on least privilege. Limit external content to avoid malicious inputs. Penetration test your models to ensure resilience against attack.

Sensitive information disclosure

Confidential business intelligence data, personally identifiable information (PII), sensitive files, and security credentials are exposed due to hacking, unauthorized access, and breaches. Users who include such data in their prompts put this information at risk.

Sanitize data to prevent user PII from entering the training data pool. Publish clear terms of use to prevent accidental disclosure while prompting. Limit content outputs to include specific data types.

Supply chain

LLMs and supporting infrastructure components from third parties can introduce vulnerabilities into the system. LLMs that partially or completely rely on external models face additional risks.

Carefully vet external tools and components before incorporating them. Ensure that data sources and suppliers are trustworthy. Continually maintain a software bill of materials (SBOM) to account for your tools and better understand how they mesh. Audit software solutions to prevent licensing and compliance issues. Create policies for security patching and periodic updates.

Data and model poisoning

Critical LLM data, collected and processed before or during model training, can undermine LLM security. Attackers can tamper with existing data or feed bad data into the model to produce outputs with less contextual value. Tampering can also introduce vulnerabilities and backdoors that compromise the LLM.

Monitor data uptake and transformations on the backend. Carefully vet all data sources, and especially those from external providers (which can contain malicious content). Sandbox your models to prevent exposure to poisoned data sources. Use version control for easy reversion. Implement thresholds and testing methods to highlight suspicious behavior.

Improper output handling

Improperly validated, sanitized, and managed output can pass faulty outputs to other downstream systems. This can lead to unauthorized privilege escalation, code execution, poor input validation, poor output visibility, security measure ineffectiveness, and vulnerability to prompt injection.

Apply zero-trust principles within the model to avoid unauthorized system actions. Validate inputs and responses originating from within the model itself. Properly encode LLM outputs that’s context-aware. Protect internal databases with parameterized queries and prepared statements. Enforce strict content policies. Log and monitor everything for quicker troubleshooting and security responses.

Excessive agency

Giving the LLM unchecked autonomy to make decisions, call functions, and communicate with external systems can make the model less effective, undermines privacy, and can introduce vulnerabilities.

Minimize the quantity of embedded extensions. Limit available functions to prevent excess agency across the model, and restrict those functions to specific tasks. Don’t just give an LLM the ability to define which actions are allowed and off-limits. Sanitize LLM inputs and outputs.

System prompt leakage

Instructions that influence how an LLM behaves or responds to users can contain secrets or sensitive backend details. It’s possible to accidentally leak credentials, system architecture, API keys, tokens, and more to users who submit malicious requests.

Isolate sensitive data from system prompts to avoid accidental leakage. Use external mechanisms to help guide LLM behavior instead of relying on system prompts (which hackers can manipulate). Create and enforce independent guardrails to ensure compliance without impacting accuracy. Separate security measures from the LLM itself and use individualized agents to perform specific tasks.

Vector and embedding weaknesses

RAG-enabled LLMs are vulnerable to improper creation, storage, and retrieval of embeddings and vectors. This makes it easier to access private data, alter LLM outputs, and inject damaging content.

Implement RBAC and similar fine-grained access controls, paired with vectors and embeddings that are permissions-aware. Create systems for reliable data validation and authenticate sources before feeding new data into the model. Tag and classify data within combined data pools to reduce mismatches and closely control access. Monitor and log everything.

Misinformation

LLMs can present false information or hallucinations as objectively or factually correct. The model’s responses can be quite inaccurate, impacting users who rely too much on LLM-generated responses to guide their decision making.

Use retrieval-augmented generation (RAG) to verify model responses and incorporate data only from trusted sources. Fine-tune models to improve accuracy while adding automatic response validation. Encourage fact checking and verification with LLM users. Integrate content filters, clearly label all AI-derived content, and be honest about anticipated response accuracy. Educate users to think critically while evaluating the usefulness of an LLM’s outputs.

Unbounded consumption

The LLM may enable users to perform too many inferences without controlling that behavior. Excessive use or abuse can consume too many backend resources (compute and memory) and increase operational costs.

Validate all inputs and enforce strict limits for prompt sizing. Restrict API responses to only provide essential information within responses. Perform dynamic resource management to scale appropriately with demand, while limiting resource use. Enforce rate limits to stop abuse. Implement timeouts and throttling mechanisms to boost performance (and improve UX). Limit how the LLM can access internal services, network resources, and APIs.