A reliable LLM application needs more than a good prompt: it needs a repeatable evaluation process. This prompt testing and evaluation framework shows you how to create canonical test cases, compare prompt and model variants, combine automated metrics with human review, and add regression checks that catch quality or safety changes before release.
Overview
Prompt evaluation is the practice of testing an LLM against representative inputs and judging whether its outputs meet defined requirements. The goal is not to find one permanent “best” prompt. It is to understand how a prompt, model, retrieval layer, tool configuration, or system prompt performs under the conditions your application actually faces.
A useful evaluation suite has four layers:
- Canonical cases: A stable collection of normal, difficult, ambiguous, and adversarial inputs.
- Expected behavior: Ground-truth answers where exactness is possible, plus rubrics for qualities such as relevance, completeness, tone, and citation behavior.
- Comparisons: Batched tests across prompt versions, models, retrieval settings, or tool strategies.
- Release controls: Regression checks and guardrails that block or flag changes that fall below an agreed standard.
Evaluation should reflect the real failure modes of the product. A customer-support assistant may need accurate policy references and safe escalation. A classification workflow may prioritize label consistency and confidence handling. An AI agent may need correct tool selection, valid arguments, and appropriate human approval. For broader system-prompt guidance, see System Prompt Design: Reusable Templates for Reliable AI Assistants.
Checklist by scenario
1. Create the test set
- Collect real, consented, and appropriately sanitized examples from the application’s main use cases.
- Add representative edge cases: incomplete requests, conflicting instructions, spelling errors, long inputs, unsupported questions, and unusual formats.
- Include known historical failures. These are often more valuable than additional easy examples.
- Separate development, validation, and holdout cases so prompt changes are not tuned only to the examples being reviewed.
- Assign each case a stable ID, category, risk level, and expected behavior.
A compact CSV template can serve as a shared starting point:
case_id,category,risk,input,reference,required_facts,forbidden_behavior,pass_rule
support-001,account,medium,"How do I change my email?","Approved account-help article","identity check; confirmation","Inventing an alternate process","Explains the approved steps and does not request secrets"
rag-002,knowledge,high,"What is the refund window?","Current refund policy","specific window; source reference","Making up a policy","Answer matches the retrieved policy or clearly states uncertainty"
agent-003,tool-use,high,"Cancel my order","Order system record","confirmation; correct tool","Calling a tool without confirmation","Requests required confirmation before the action"Keep sensitive production data out of shared test files unless your handling process explicitly permits it. The LLM Data Privacy Checklist can help structure a review of prompts, logs, vendors, and retention.
2. Define scoring and rubrics
Use exact checks when the answer has a verifiable target. Examples include valid JSON, an allowed classification label, required fields, correct arithmetic, or the presence of a known fact. Use a rubric for open-ended responses. A practical rubric can score each dimension from 0 to 2:
- Accuracy: Are the claims supported by the reference or available context?
- Relevance: Does the response answer the user’s request without distracting material?
- Completeness: Are required steps, fields, or caveats present?
- Hallucination risk: Does the response invent facts, sources, actions, or certainty?
- Safety and bias flags: Does it expose sensitive information, apply an unsupported assumption, or treat comparable cases inconsistently?
Write the pass rule before looking at variant results. For example: “Pass if accuracy and relevance are at least 1, no critical hallucination flag is present, and the required output schema validates.” Thresholds should be treated as product decisions, not universal industry standards. High-risk workflows generally need stricter blocking rules and more human review.
3. Run controlled A/B comparisons
Change one major variable at a time where possible. Compare prompt A with prompt B using the same cases, model settings, retrieval context, and tool definitions. Then repeat the comparison if outputs are variable. Record the model or provider identifier, prompt version, system instructions, temperature or equivalent settings, retrieval configuration, tool schema version, timestamp, latency, token usage, and error details.
For an AI workflow, evaluate more than the final text. Check whether the system selected the right route, retrieved useful context, called the correct function, produced valid arguments, and stopped for approval when required. The guide to AI agent workflows provides useful categories for these checks.
4. Combine automated checks with human review
Automated evaluation is fast and repeatable, but it can miss subtle omissions, misleading confidence, or unfair treatment. Review a deliberately selected sample: failures, borderline passes, high-risk cases, new categories, and a random selection of ordinary cases. Have reviewers use the same rubric and record a reason for each failure. If reviewers disagree, clarify the rubric rather than silently averaging away the disagreement.
5. Add regression checks and guardrails
Promote a case to the regression suite when it represents a serious failure, a common user path, or a requirement that must not change. A release can then check hard gates such as schema validity, prohibited content, tool authorization, and critical factual errors, alongside softer metrics such as mean relevance or reviewer preference. Keep failures visible with the case ID, output, expected behavior, and likely cause. For a dedicated implementation pattern, see Prompt Regression Testing: How to Build an Evaluation Workflow.
What to double-check
- Test-set coverage: Does the suite represent actual traffic, or only the examples that are easy to label?
- Reference quality: Are ground-truth answers current, authorized, and precise enough to evaluate against?
- Retrieval effects: Is a failure caused by the prompt, missing context, stale documents, or poor chunk selection? A RAG knowledge-base audit can isolate retrieval problems.
- Output parsing: Does the evaluator validate the exact schema your application consumes rather than merely checking that the response looks plausible?
- Judge limitations: If another model scores outputs, are its criteria explicit, and are a sample of judgments reviewed by people?
- Cost and latency: Does the improved quality justify additional tokens, retrieval calls, tool calls, or review time? Track these alongside quality; the LLM cost optimization workflow offers a practical companion process.
- Privacy: Are logs, prompts, retrieved documents, and evaluator inputs handled consistently with your internal requirements?
Common mistakes
- Testing only happy paths. Add ambiguity, unsupported requests, malformed input, and known failures.
- Optimizing for a single score. A higher average can hide a severe failure in a high-risk category. Track category-level and critical-failure results.
- Changing several variables together. If the prompt, model, retriever, and tool schema all change, the result is difficult to explain.
- Using vague pass criteria. Replace “sounds good” with observable requirements and examples of unacceptable behavior.
- Leaking test answers into the prompt. Keep evaluation references separate from the production input unless the application is intentionally designed to receive them.
- Ignoring non-text behavior. Validate structured output, tool calls, routing, refusals, citations, latency, and failure recovery where relevant.
- Deleting failures after a fix. Preserve them as regression cases so a later change does not recreate the problem.
When to revisit
Re-run the suite before releasing a prompt change, switching models or providers, changing retrieval settings, revising tool schemas, adding a new user segment, or modifying safety and escalation rules. Schedule a recurring reliability review as well; monthly checks are a reasonable operational starting point for active systems, while lower-change workflows may use a longer interval.
Review the test set before seasonal planning cycles and whenever workflows or tools change. Refresh examples when product rules, source documents, output schemas, or common user language change. Keep a short evaluation report for each run with the date, versions, sample size, category results, critical failures, reviewer notes, cost, and follow-up owner.
Action checklist: choose 20–50 representative cases to start, label the expected behavior, define hard and soft checks, run two variants under identical conditions, manually inspect failures and a sample of passes, then save the results as a baseline. Expand the suite from real incidents and rerun it after every material change. This recurring loop turns prompt engineering from a one-time drafting exercise into a measurable reliability practice.
