-
Notifications
You must be signed in to change notification settings - Fork 49
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* Added Prompt Injection template, recommendation, guidance * LLM Output Handling, Training Data Poisoning, and Excessive Agency/Permission Manipulation are all to be added shortly
- Loading branch information
Showing
3 changed files
with
44 additions
and
0 deletions.
There are no files selected for viewing
5 changes: 5 additions & 0 deletions
5
...s/description/ai_application_security/llm_security/prompt_injection/guidance.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
# Guidance | ||
|
||
Provide a step-by-step walkthrough with a screenshot on how you exploited the vulnerability. This will speed triage time and result in faster rewards. Please include specific details on where you identified the cryptographic weakness, how you identified it, and what actions you were able to perform as a result. | ||
|
||
Attempt to escalate the vulnerability to perform additional actions. If this is possible, provide a full Proof of Concept (PoC). |
13 changes: 13 additions & 0 deletions
13
...iption/ai_application_security/llm_security/prompt_injection/recommendations.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,13 @@ | ||
# Recommendation(s) | ||
|
||
There is no single technique to prevent prompt injection from occurring. However, implementing the following defensive measures within the LLM application can prevent and limit the impact of prompt injection: | ||
|
||
- Use privilege controls for access to backend systems or when performing privileged operations. Apply the principle of least privilege to restrict the LLM's access to backend systems to that which is strictly necessary for its intended functionality. | ||
- For privileged operations, require user interaction to approve any authorized action that would be performed on behalf of them. | ||
- Treat user input, external input, and the LLM as untrusted input sources. | ||
- Establish trust boundaries between external sources, the LLM, any plugins, and any neighboring systems. | ||
|
||
For more information, refer to the following resources: | ||
|
||
- <https://owasp.org/www-project-top-10-for-large-language-model-applications/> | ||
- <https://aivillage.org/large%20language%20models/threat-modeling-llm/> |
26 changes: 26 additions & 0 deletions
26
...s/description/ai_application_security/llm_security/prompt_injection/template.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,26 @@ | ||
# Prompt Injection | ||
|
||
## Overview of the Vulnerability | ||
|
||
Prompt injection occurs when an attacker crafts a malicious prompt that manipulates a large language model (LLM) into executing unintended actions. The LLM has a lack of segregation between user input and the data within the LLM. This can allow an attacker to inject malicious prompts into an LLM which bypass safeguards and gain unauthorized access to data. | ||
|
||
## Business Impact | ||
|
||
This vulnerability can lead to reputational and financial damage of the company due an attacker gaining access to unauthorized data or compromising the decision-making of the LLM, which would also impact customers' trust. The severity of the impact to the business is dependent on the sensitivity of the accessible data being transmitted by the application. | ||
|
||
## Steps to Reproduce | ||
|
||
1. Navigate to the following URL: | ||
1. Inject the following prompt into the LLM: | ||
|
||
```prompt | ||
{malicious prompt} | ||
``` | ||
|
||
1. Observe that the LLM returns sensitive data | ||
|
||
## Proof of Concept (PoC) | ||
|
||
The screenshot(s) below demonstrate(s) the vulnerability: | ||
> | ||
> {{screenshot}} |