LLM hallucinations are usually symptoms of a failure somewhere in the application pipeline, not simply a model problem. This diagnostic workflow helps developers trace unsupported or incorrect answers to prompt ambiguity, missing context, retrieval defects, tool failures, model limits, or evaluation gaps—and then choose a targeted fix instead of rewriting prompts at random.
Overview
A hallucination is an answer that sounds plausible but is unsupported by the information the application should use. The response may contain an invented fact, an outdated detail, a misread calculation, a fabricated citation, or an action that the model was not authorized to take. In production, the useful question is not only “Why did the model hallucinate?” but also “Which layer introduced or failed to prevent the error?”
An LLM application typically has several layers: user input, system instructions, retrieved context, conversation memory, tool calls, model generation, output parsing, and application-side validation. An error at any layer can appear to be a prompting issue. For example, a retrieval system may return irrelevant documents, while the final answer looks like a confident but poorly grounded completion. A tool may return an empty result, while the model fills the gap with an assumption.
Start with evidence. Save the exact user message, system prompt version, model and configuration identifiers, retrieved passages, tool inputs and outputs, final response, and any reviewer feedback that is safe to retain. Without this record, debugging becomes speculation and successful fixes are difficult to reproduce.
For broader reliability work, pair this process with a prompt regression testing workflow. The goal is not to make every response sound cautious. The goal is to make the application’s behavior accurate, traceable, and appropriately limited.
Template structure
Use the following symptom-to-cause table as a first-pass decision aid. Test the simplest likely cause before changing the model or adding another layer of complexity.
| Observed symptom | Likely cause | First diagnostic test | Common remediation |
|---|---|---|---|
| The answer addresses the wrong interpretation of the request. | Prompt ambiguity or missing task constraints | Rewrite the request as a precise task with an explicit output schema and one representative example. | Clarify scope, define terms, state what to do when information is missing, and separate instructions from user-provided data. |
| The answer includes details absent from the supplied context. | Missing grounding or weak refusal conditions | Ask the model to identify which claims are supported by the context, then test with a deliberately incomplete document. | Require citations or source spans where appropriate, and instruct the model to say that the information is unavailable. |
| The answer changes when the same question is repeated. | Sampling variation, unstable context, or hidden state | Replay the exact request with identical inputs and compare retrieved documents, tool results, and configuration. | Control relevant generation settings, stabilize inputs, and test a range of valid outputs rather than expecting identical wording. |
| The response cites irrelevant or nonexistent material. | Retrieval errors or citation generation without verification | Inspect the retrieved chunks and check every cited source against the original data. | Improve chunking and filtering, preserve source identifiers, and validate citations in application code. |
| The model claims an action was completed, but no action occurred. | Tool failure, unclear tool state, or unsupported confirmation | Compare the assistant’s claim with tool-call logs and the tool’s returned status. | Expose explicit success and failure states, require confirmation from the tool, and prohibit completion claims without a successful result. |
| The answer fails on long, complex, or multi-step inputs. | Model or context-window limits, instruction dilution, or reasoning complexity | Reduce the input and test each step independently. | Split the workflow, summarize selected context, use structured intermediate results, or route difficult cases for review. |
| Quality appears acceptable in demos but fails in production. | Evaluation gaps or unrepresentative test data | Compare production failures with the evaluation set by task type, language, input length, and risk. | Add failure cases to a versioned test set and measure groundedness, completeness, format validity, and escalation behavior. |
Once you have a likely cause, isolate one variable at a time. Keep the original prompt and application behavior as a control. Then test a single change—such as improved retrieval filtering, a clearer system instruction, or a stricter tool response—and compare it against the same cases.
Incident worksheet
Copy this worksheet into an issue tracker or evaluation record:
- Incident ID and date: Identify the failure without placing sensitive user data in the title.
- Expected behavior: State what a correct answer or action would contain.
- Observed behavior: Quote the relevant response and label each unsupported claim.
- Input snapshot: Record the user request, conversation state, and relevant metadata.
- Prompt version: Record the system, developer, and task prompt versions.
- Grounding inputs: List retrieved documents, source identifiers, and ranking or filtering decisions.
- Tool trace: Record tool arguments, returned data, errors, timeouts, and authorization outcomes.
- Hypothesis: Choose one primary suspected cause and explain why.
- Test: Describe the smallest reproducible experiment.
- Fix and regression case: Record the change and add the failure to a repeatable test set.
How to customize
Customize the workflow according to the type of LLM application rather than applying one universal prompt. A question-answering assistant needs reliable grounding and source handling. An agent needs additional checks around planning, permissions, tool arguments, and human approval. A structured extraction service needs schema validation and clear handling for missing fields.
For retrieval-augmented applications
Inspect the entire retrieval path, not just the final answer. Confirm that documents are current for the use case, chunks preserve enough context, metadata filters are correct, and the selected passages actually answer the question. A model cannot reliably ground an answer in content it never received. If the retrieved material conflicts, define how the application should represent that conflict instead of asking the model to silently choose.
Useful tests include questions whose answers are present, absent, ambiguous, and contradicted by two sources. Measure whether the assistant identifies the “not found” case rather than rewarding an answer for sounding complete. A vector database comparison can help with infrastructure selection, but changing databases will not correct poor source preparation or inadequate evaluation by itself.
For tool-using agents
Separate model intent from tool truth. The assistant can propose a tool call, but the application should validate arguments, enforce permissions, execute the operation, and return a typed result. Make failure states visible: an empty result, timeout, authorization failure, and successful operation should not all look like ordinary text. After the tool responds, instruct the model to describe only the returned status and data.
Review agent traces for loops, premature completion, unsupported assumptions, and missing handoffs. The guide to AI agent workflows, tools, memory, routing, and human approval provides a useful design context for these checks.
For prompt and output changes
Use system instructions to establish durable behavior: define the task, evidence boundaries, refusal or escalation conditions, and output format. Keep variable data clearly separated from instructions. Avoid adding broad statements such as “be accurate” without specifying what accuracy means for the task. For reusable patterns, see system prompt design templates.
Validate structured output outside the model. A valid JSON document can still contain a false claim, so combine schema checks with semantic tests. Keep prompt edits versioned, and evaluate them against both ordinary examples and known failure cases. A prompt testing framework makes this comparison easier to repeat.
Examples
Example 1: Unsupported policy answer
An internal assistant is given a short policy excerpt and asked whether a request is allowed. It answers “yes” and adds an exception that is not in the excerpt. The first test is not a new model. Compare the answer with a prompt that explicitly requires the assistant to quote the supporting passage or state that the excerpt does not decide the question. Then test a question whose answer is absent. If the assistant still invents an exception, strengthen the evidence boundary and add a required escalation outcome. If the retrieved excerpt was incomplete, fix retrieval and document assembly instead.
Example 2: False completion in an automation workflow
A support agent tells a user that a ticket was closed, but the ticketing API returned a timeout. The incident worksheet should show whether the model saw the timeout, whether the application translated it into a clear failure result, and whether the final response was generated before the tool completed. The remediation is an explicit tool status such as success, failed, or unknown, plus a rule that only success permits a completion statement. Add timeout and partial-failure cases to the evaluation set.
Example 3: Correct facts, wrong decision
A document assistant extracts the right fields but recommends the wrong next step. This may not be a retrieval problem. Inspect the decision rule, the expected output schema, and the evaluation rubric. If reviewers disagree about the correct action, the specification is incomplete. Define the decision criteria, include boundary cases, and score the recommendation separately from extraction accuracy.
When to update
Revisit this troubleshooting workflow whenever the application’s inputs or behavior change. The most important triggers include a model replacement, a new model configuration, prompt or tool-schema edits, changes to chunking or retrieval filters, a new data source, altered memory behavior, a new user population, or a production incident that reveals an untested case.
Review the evaluation set on a regular engineering cycle as well. Retire duplicate cases, preserve representative failures, add newly observed edge cases, and separate factual correctness from helpfulness, formatting, latency, and safety-related review criteria. A passing score on one dimension should not conceal a failure on another.
After each fix, run the original reproduction, nearby variants, and a regression suite. Confirm not only that the hallucination is reduced, but also that the change has not caused excessive refusals, omitted supported answers, broken tool calls, or invalid output. For production readiness checks, use the LLM app deployment checklist alongside application-specific tests.
Practical next step: choose one recent failure, complete the incident worksheet, and classify it using the decision table. Run the smallest test that can distinguish between prompt, context, retrieval, tool, model, and evaluation causes. Apply one targeted fix, add the failure to a versioned regression set, and schedule a review whenever the model, data, prompt, or workflow changes.
