Checklist
RAG maintenance checklist for production AI apps
A practical checklist for keeping RAG sources, chunks, retrieval tests, repairs, and agent handoffs healthy after launch.
RAG maintenance is the recurring work of checking whether sources are fresh, chunks are usable, retrieval tests still pass, and repairs are approved before users receive stale or wrong answers.
What should a RAG maintenance loop check?
A production RAG system should check source freshness, crawl health, chunk quality, duplicate context, retrieval coverage, repair status, and usage cost. These checks matter because RAG systems depend on external knowledge that changes after launch.
The maintenance loop should run on a schedule and after important source changes. A failing check should create a repair item, not just a log line.
- Sources: last synced time, owner, crawl status, and source availability.
- Chunks: thin text, oversized passages, duplicate content, risky or unsupported claims.
- Retrieval tests: important questions that should keep returning the right context.
- Repairs: clear owner, recommended action, approval state, and delivery history.
- Usage: care cycles, crawled pages, outbound events, and agent actions.
When does a RAG system start to rot?
RAG quality usually decays when source pages change, documents move, API docs are rewritten, product policies update, or support content gets duplicated. The system may still answer confidently, but it can be pulling old or weak context.
The risk is highest for support copilots, docs assistants, internal knowledge bots, sales enablement assistants, codebase agents, and product support bots because those sources change often.
What is the minimum viable maintenance setup?
Start with one source, five to ten retrieval tests, one action channel, and a repair queue. The goal is not to monitor everything on day one. The goal is to catch the first obvious drift before users do.
- Connect the highest-risk source first.
- Add questions users actually ask.
- Run a care cycle after every major source update.
- Send repair events to email, webhook, or an agent endpoint.
- Approve repairs before production-impacting writes.
Common questions
What is RAG maintenance?
RAG maintenance is the ongoing care work that keeps retrieval sources, chunks, tests, and repair workflows accurate after the first launch.
How often should a RAG system be checked?
A small RAG system should run a care cycle weekly and after important docs, policy, product, or code changes. High-change support and product systems may need daily checks.
Should repairs happen automatically?
Repairs should be recommended automatically, but production-impacting writes should require approval until preview, rollback, and ownership controls are strong.