A RAG application is only as trustworthy as the documents behind its retrieval layer. This maintenance checklist gives you a repeatable workflow for keeping a RAG knowledge base accurate, current, searchable, and aligned with the answers your users need.
Overview
Retrieval-augmented generation (RAG) connects an LLM to an external knowledge base. When a user asks a question, the application retrieves relevant passages and supplies them as context for the model's response. This approach can improve answers that depend on company documentation, product information, procedures, or other changing content, but retrieval quality depends on more than the vector database.
A reliable RAG knowledge base maintenance process covers the entire content lifecycle:
- Inventory: Know which sources are included, who owns them, and when they were last reviewed.
- Ingestion: Convert approved documents into clean, searchable records without losing important structure.
- Indexing: Apply consistent chunking, embeddings, metadata, and access controls.
- Validation: Test whether the system retrieves the right evidence and uses it correctly.
- Retirement: Remove, archive, or replace content that is obsolete, duplicated, or no longer authoritative.
Think of the knowledge base as a maintained application dependency rather than a one-time data import. Your embedding model, vector store, parsing tools, and retrieval settings may change over time, so record their versions and test changes before applying them to production. For related release practices, see this guide to LLM application CI/CD.
Document inventory template
Maintain an inventory in a spreadsheet, database, or repository with fields such as:
- Document ID and source URL or storage location
- Title, content type, department, and owner
- Effective date, last modified date, and next review date
- Version, status, and replacement document if applicable
- Audience, access scope, and sensitivity classification
- Ingestion status, index version, and deletion status
- Known limitations, dependencies, or required interpretation
The inventory should be authoritative enough for an operator to answer three questions quickly: what is in the index, why is it there, and what should happen when the source changes?
Checklist by scenario
When adding a new document source
- Confirm ownership. Identify a person or team responsible for accuracy and future review.
- Define scope. Decide which files, pages, records, or folders belong in the knowledge base. Avoid ingesting an entire repository when only a controlled subset is relevant.
- Check permissions. Ensure retrieval cannot expose content to users who could not access the original source.
- Inspect the format. Look for scanned PDFs, tables, repeated navigation text, hidden comments, broken encoding, and images containing essential information.
- Set metadata. Add fields such as source, department, document type, product, audience, effective date, expiration date, language, and access group.
- Choose chunk boundaries. Keep related headings, definitions, procedures, examples, and cautions together where possible. Avoid splitting a step sequence or a table across unrelated chunks.
- Run a small ingestion test. Review extracted text and retrieved results manually before indexing the full source.
- Record the index version. Save the parser, chunking configuration, embedding model identifier, metadata schema, and timestamp.
When an existing document changes
- Compare the new version with the indexed version.
- Determine whether the change is substantive, such as a new policy, product behavior, eligibility rule, or procedure.
- Reprocess all affected chunks rather than assuming an update will overwrite every prior representation.
- Remove obsolete chunks by stable document ID and version, not only by matching text.
- Check cross-references to related documents.
- Test questions that depended on the old wording.
- Keep an audit record showing who approved the update and when it became active.
For prompt, model, and retrieval changes that ship together, use a versioned release process. The guide to LLM prompt versioning provides a useful companion workflow.
When content becomes stale or invalid
Define explicit stale-content rules before you need them. A practical policy can distinguish between:
- Active: Approved for retrieval and within its review period.
- Needs review: Past its review date or affected by an upstream change.
- Superseded: Replaced by a newer approved version and excluded from normal retrieval.
- Archived: Retained for records but unavailable to the assistant unless a controlled use case requires it.
- Deleted: Removed from source storage and the retrieval index where appropriate.
Do not rely on an LLM to determine whether outdated content should be ignored. Mark status in metadata and enforce it with filters or ingestion rules. If a document is withdrawn, delete its chunks, cached retrieval results, summaries, and other derived records according to your application's retention design.
When retrieval quality drops
- Collect failed questions and classify the failure: missing source, poor parsing, weak query matching, incorrect filters, bad chunking, or answer-generation error.
- Inspect the top retrieved passages, not only the final answer.
- Check whether the correct source exists but is hidden by stale duplicates or overly narrow metadata filters.
- Compare different chunk sizes, overlap settings, query transformations, or reranking options in a controlled evaluation set.
- Re-index only after identifying the likely cause. Rebuilding the entire index can conceal the problem without fixing it.
A prompt testing and evaluation workflow can help separate retrieval failures from generation failures. Use representative questions, expected evidence, citation or source requirements, and an outcome label such as correct, incomplete, unsupported, or irrelevant.
What to double-check
Chunking and metadata review table
| Area | Review question | Warning sign |
|---|---|---|
| Chunk boundaries | Does each chunk preserve enough context to stand alone? | Answers contain fragments, missing conditions, or detached headings. |
| Chunk size | Are chunks small enough to retrieve precisely but large enough to explain the point? | Results are either vague snippets or oversized passages with unrelated material. |
| Metadata | Can the application filter by source, audience, date, product, and status? | Users receive content from the wrong team, version, or permission group. |
| Dates | Are effective, modified, and expiration dates distinct? | A recent edit is mistaken for current approval. |
| Duplicates | Can identical or near-identical documents be identified? | Retrieval returns conflicting copies of the same rule. |
| Tables and lists | Does extraction preserve row labels, units, ordering, and relationships? | The model combines values from different rows or loses required steps. |
| Access control | Are permissions enforced before context reaches the model? | A response reveals restricted text from a valid retrieval match. |
Source priority
Define which source wins when documents conflict. A useful hierarchy may prioritize an approved policy or product specification over an informal note, but the exact order depends on your organization. Store the priority as explicit metadata or configuration rather than leaving it to the model. Include a conflict-handling instruction in the application: when sources disagree, the assistant should identify the conflict, prefer the approved current source, and avoid presenting an uncertain conclusion as fact.
Embedding and index changes
Embedding models represent text for similarity search, and changing the model can alter which items are considered close to a query. Treat an embedding change as an index migration: preserve the old configuration, build a separate test index, compare retrieval results, and promote the new index only after evaluation. The same caution applies when changing distance metrics, normalization, chunking, language handling, or vector-database settings.
Maintain a small regression set containing common questions, difficult questions, recently changed topics, permission-sensitive requests, and questions that should return no answer. This makes it easier to verify whether a refresh improved the system or merely changed its failure modes.
Common mistakes
- Ingesting everything: More documents do not automatically produce better answers. Uncurated material increases duplicate and conflicting evidence.
- Updating without deleting: Adding a new version while leaving the old one active allows obsolete instructions to remain retrievable.
- Using modification dates as truth: A file can be edited without being reviewed or approved. Track lifecycle status separately.
- Ignoring extraction quality: A clean-looking source file may produce unusable text when parsed, especially when it contains tables, columns, footnotes, or images.
- Testing only final answers: Inspect retrieved passages to determine whether the issue is retrieval, metadata filtering, or generation.
- Changing several variables at once: If chunking, embeddings, prompts, and model settings change together, it becomes difficult to identify what helped or caused regression.
- Skipping access tests: Test retrieval as different user roles, including users with limited access and users with no matching authorization.
- Letting stale content fail silently: An expired document should trigger review, exclusion, replacement, or an explicit fallback—not remain active indefinitely.
For broader reliability practices, pair this workflow with a production checklist for reducing AI hallucinations and a documented data privacy review.
When to revisit
Use a recurring schedule, but also trigger an immediate review when the underlying knowledge changes. A practical cadence is:
- Per change: Re-ingest and validate documents after policy, product, process, or ownership changes.
- Weekly or on a suitable operational cycle: Review ingestion failures, stale-content alerts, unanswered questions, and access-control errors.
- Before seasonal planning cycles: Check dates, deadlines, operating procedures, staffing guidance, and other time-sensitive material.
- Monthly: Sample retrieved passages, inspect duplicates, review low-confidence or escalated questions, and confirm source owners.
- Quarterly: Audit the full inventory, lifecycle rules, metadata schema, evaluation set, embedding configuration, vector index, and deletion process.
- Before expanding scope: Test new departments, languages, repositories, user groups, or agent workflows in a separate evaluation environment.
- After tooling changes: Re-run regression tests when parsers, vector databases, embedding models, retrieval libraries, or LLM providers change.
End each review with a short decision record: what changed, what was tested, which documents were added or removed, what failed, and who owns the next action. A maintained RAG knowledge base is not finished when indexing succeeds. It is ready for production when its sources are governed, its changes are traceable, its retrieval behavior is tested, and its stale-content path is clear.
Start with one representative source collection and a small evaluation set. Establish the inventory and deletion rules first, then improve chunking, metadata, and retrieval configuration based on observed failures. This keeps maintenance practical while giving your AI application a dependable foundation for future growth.