Prompt Injection Testing: A Practical Red-Team Workflow for LLM Applications
Prompt injection testing helps you find where an LLM application may follow hostile or irrelevant instructions instead of its intended task. This practical red-team workflow covers direct and indirect attacks, untrusted documents, tool-call manipulation, logging, severity ratings, remediation, and a repeatable release-gate process.
Overview
Prompt injection is a reliability and security problem in which content supplied to a model changes how it interprets its instructions. The content may come directly from a user, or indirectly from a document, web page, email, database record, retrieved chunk, tool result, or other external source. The risk becomes more significant when an LLM application can access sensitive information, call tools, modify records, send messages, or trigger automation.
A strong test process does not assume that a system prompt alone can separate trusted instructions from untrusted data. Instead, it examines the complete application path: input handling, prompt construction, retrieval, model behavior, output parsing, authorization, tool execution, logging, and human approval. The objective is not to prove that injection is impossible. It is to identify unsafe failure modes, reduce their impact, and verify that defenses continue to work after changes.
Use this workflow before an initial launch and whenever you change the model, system prompt, retrieval pipeline, connected data, tools, permissions, or orchestration logic. For broader release discipline, pair it with an LLM prompt testing and evaluation framework and your application’s normal CI/CD process.
What to track
1. Map trust boundaries and high-impact actions
Begin by listing every instruction and data source that reaches the model. Mark each item as trusted, user-controlled, externally retrieved, or generated by another model. Then document what the model can do after producing an output.
- Trusted instructions: system messages, fixed developer rules, application policies, and hard-coded workflow constraints.
- Untrusted content: user messages, uploaded files, retrieved passages, web content, emails, support tickets, and tool results.
- Decision points: routing, classification, retrieval selection, approval requests, and escalation.
- High-impact actions: reading private data, changing records, issuing refunds, sending messages, executing code, or calling external services.
For each tool, record the allowed arguments, user authorization requirements, data scope, and whether the action can be reversed. A model should not be the only control deciding whether a sensitive action is permitted.
2. Test direct and indirect injection
Direct injection places the attack in the user’s message. Examples include requests to ignore previous instructions, reveal hidden context, change the task, or perform an unrelated tool action. Test these cases in normal language as well as with content that attempts to create urgency, authority, or confusion.
Indirect injection places the attack in content the application retrieves or processes. A document might contain instructions aimed at the assistant rather than the user. A web page, email, or database field may attempt to redirect the model, request secret disclosure, or alter a tool call. Test each ingestion route separately because filtering one source does not automatically protect another.
3. Track instruction-hierarchy failures
Record whether the model treats untrusted text as data or as a higher-priority instruction. Useful test cases ask the application to summarize, classify, quote, translate, or extract content that contains phrases such as “ignore the task” or “send the following information.” The expected result is that the application completes the assigned operation without adopting the embedded instruction.
Do not treat a refusal by itself as a complete pass. A robust result should also preserve the required output format, avoid exposing hidden context, and avoid taking unauthorized actions.
4. Test tool-call manipulation
When an application supports function calling or agent workflows, evaluate both the model’s proposed call and the server-side executor. Try to influence tool names, arguments, recipients, filters, destinations, quantities, and approval states. Include cases where retrieved text tells the model to call a tool or where a tool result contains a second instruction.
The application should validate arguments against a schema, enforce authorization outside the model, restrict access to the minimum necessary data, and require confirmation for consequential actions where appropriate. Tool-call tests belong in regression suites because a prompt or model update can change the model’s preferred action even when the tool definitions remain unchanged.
Attack-pattern comparison
| Pattern | Typical entry point | What to observe | Primary control |
|---|---|---|---|
| Direct injection | User message | Instruction override, secret requests, task diversion | Clear task boundaries, output checks, authorization |
| Indirect injection | Retrieved or uploaded content | Model follows document instructions instead of processing content | Source labeling, isolation, retrieval and output evaluation |
| Tool manipulation | User text, document, or tool result | Unsafe arguments, unauthorized destination, excessive scope | Server-side validation, least privilege, approval gates |
| Context disclosure | Prompt or conversation request | Exposure of hidden instructions, credentials, or private context | Secret isolation, redaction, access controls, monitoring |
5. Use a consistent severity rating
Rate findings according to realistic impact, not just how unusual the attack looks. A useful internal scale is:
- Critical: injection can reliably cause unauthorized high-impact actions or broad sensitive-data exposure.
- High: a practical attack can cross a meaningful permission boundary, affect external users, or trigger costly or difficult-to-reverse behavior.
- Medium: the attack changes output, reveals limited internal context, or creates workflow errors without a confirmed high-impact action.
- Low: the attack causes cosmetic drift or a contained refusal without affecting data, permissions, or downstream behavior.
For every finding, record exploitability, affected users or data, reversibility, required access, and whether a human approval step can stop the action. Severity should reflect the application’s permissions and business context.
Reusable test-case template
Test ID:
Application path:
Trust boundary tested:
Attack source: direct user | retrieved content | tool result | other
Input or fixture:
Expected behavior:
Observed response:
Tool call produced: yes/no
Sensitive data exposed: yes/no
Unauthorized action completed: yes/no
Severity:
Evidence and trace ID:
Remediation owner:
Regression test added: yes/no
Cadence and checkpoints
Run a full prompt injection review before production release, then use smaller recurring checks. A quarterly review is a practical baseline for applications that change gradually; use a shorter interval for systems with sensitive data, broad tool access, or frequent model and retrieval changes.
Before each release
- Run the direct-injection, indirect-injection, and tool-call regression suites.
- Compare results with the previous model, prompt, and application version.
- Verify that logs capture the relevant request, retrieved source identifiers, model output, tool proposal, executor decision, and final outcome without storing unnecessary secrets.
- Confirm that failed tests block deployment when they involve unauthorized data access or high-impact actions.
Monthly or quarterly review
Review production traces for unexpected refusals, unusual tool calls, repeated prompt patterns, retrieval sources that contain instructions, and actions requiring manual correction. Sample failures and add representative cases to the regression set. Recheck permissions and data-source assumptions rather than focusing only on prompt wording.
Change-triggered review
Repeat relevant tests after changing the model provider or version, system instructions, retrieval sources, chunking, tool definitions, agent routing, authentication, output parser, or approval flow. A small change in one layer can alter behavior in another, so retain the exact test fixture and application configuration used for comparison.
How to interpret changes
Compare more than pass and fail counts. Track which attack classes succeed, which tools are proposed, how often the application asks for clarification, and whether a successful injection reaches an external side effect. A higher refusal rate may not indicate improvement if legitimate tasks are also being blocked. Conversely, fluent answers are not evidence that the system is safe if the executor accepts unchecked arguments.
When a test fails, isolate the layer responsible:
- Prompt construction: Is untrusted content clearly separated and labeled as data?
- Model behavior: Does the model follow embedded instructions or disclose protected context?
- Output handling: Does parsing accept unexpected fields, destinations, or commands?
- Authorization: Can the server reject an action even when the model requests it?
- Observability: Can the team reconstruct what happened without exposing more sensitive data?
Prefer layered remediation. Prompt changes may improve instruction following, but they should be combined with least-privilege tools, strict schemas, independent authorization, source-aware retrieval, rate limits, human approval for consequential actions, and safe failure behavior. For retrieval-heavy applications, also review the RAG knowledge base maintenance workflow.
When to revisit
Keep this article and your test suite as living release material. Revisit it on a monthly or quarterly cadence, after a prompt injection incident, and whenever recurring data points change. The most important triggers are a new model, altered system prompt, new document or web source, expanded tool permissions, changed agent orchestration, new parser, or a production finding involving unexpected instructions.
Release-gate checklist
- All known direct and indirect injection cases have expected outcomes.
- Untrusted documents, web pages, emails, and tool results are tested as separate sources.
- Tool arguments are validated and authorized outside the model.
- High-impact actions have an approval or reversible execution path.
- Severity, evidence, ownership, and regression status are recorded for every finding.
- Logs support investigation while limiting unnecessary sensitive-data retention.
- Failed critical or high-severity tests prevent release until remediated or formally accepted.
- The test suite is versioned with the prompt, model configuration, retrieval setup, and tool definitions.
After remediation, rerun the original failing case, nearby variations, and a representative legitimate task. A fix is ready for release only when it reduces the unsafe behavior without breaking the application’s intended workflow. Keep the successful regression cases in CI so prompt injection testing becomes part of ongoing LLM application reliability rather than a one-time security exercise.