RAG in Production Playbook
Ship a RAG system that survives week three — a decision playbook for chunking, retrieval, reranking, evaluation and the seven failure modes that kill most demos.
Chunking strategies, reranking setups, evaluation harnesses and the seven failure modes that kill every RAG demo in week three.
Indicative price only — not a final sale price. Checkout is not open yet; the final price requires owner confirmation before anything can be bought.
- Chunking decision tree
- Reranker selection guide + a run-your-own benchmark script
- Ragas eval harness
- 7 failure modes + fixes
The order, payment and file-delivery pipeline is built and verified end-to-end in test mode. Real purchases stay off until the owner verifies the payment account and approves final prices. The price above is indicative only, not an approved sale price.
Delivery system status & test record →Who it's for — and who it's not for
For
- Engineers building or fixing a RAG / retrieval system in production
- Teams whose RAG 'sounds smart but is wrong' and need a diagnosis path
- Anyone who wants to build the evaluation harness before scaling, not after
Not for
- Not a framework or library — it is the decisions and the reasoning around the build
- Not a substitute for benchmarking on your own data: it tells you how and gives you a harness, but your numbers come from your data
What you actually get
playbook.md≈ 23 KBBody: chunking decision tree (§3), retrieval and reranking (§4–5, including the 'run your own benchmarks' stance and a reproducible script), the Ragas evaluation harness and CI gate (§6), the seven failure modes (§7), the go-live checklist (§8), the cost model (§9), and three reference configurations (§10).playbook.html≈ 35 KBA4 print edition; export to PDF from the browser.
Listed straight from the shipped playbook.md + manifest.json — nothing below is a mock.
Table of contents
- Who this is for
- 1. Before you build: when RAG is the wrong answer
- 2. The pipeline, and where it actually breaks
- 3. Chunking decision tree
- Defaults that survive contact with production
- Three rules that matter more than size
- Late chunking
- 4. Retrieval: go hybrid on day one
- Metadata filters beat clever retrieval
- 5. Reranking: the cheapest quality win available
- Selecting a reranker
- Where reranking sits in the latency budget
- Implementation notes
- 6. Evaluation: build the harness before you need it
- Build the set first — 30 to 50 questions
- Metrics that actually move decisions
- Ragas harness
- Wire it into CI
- 7. The seven failure modes
- #1 Parsing rot
- #2 Chunk boundary amputation
- #3 Retrieval recall cliff
- #4 Permission leakage
- #5 Stale index
- #6 Evaluating on vibes
- #7 Cost blow-up
- 8. Go-live checklist
- 9. Cost model
- 10. Three reference configurations
- Appendix A: Benchmark harness
- Appendix B: Answer prompt that holds up
- Appendix C: What to log, minimum viable
- Changelog
A real preview
Who this is for
You already have a demo that works on ten clean PDFs. You now have to point it at 40,000 files, a permission model, and a stakeholder who will ask "why did it say that?" in front of other people. Every recommendation here is something you can implement in a day or less. Where a recommendation depends on your corpus, this playbook tells you how to measure it yourself instead of quoting a number — see Appendix A.
1. Before you build: when RAG is the wrong answer
The most expensive RAG mistake is building one when you needed something else. Work this table first. · Situation · Better answer than RAG · Why · · Corpus < ~200k tokens, stable, fits in context · Put it in the prompt · Retrieval infrastructure is pure overhead at this size · · Questions are lookups against structured data · Text-to-SQL over the database · A join beats a vector search every time · · Corpus changes hourly and must be current · Long-context + cache, or a…
Excerpted verbatim from the actual playbook.md — not sample/mock content.
How to use it
- Open playbook.md in any Markdown viewer, or print playbook.html to PDF (browser Ctrl/Cmd+P, enable background graphics).
- Section 6 includes a reproducible Ragas eval harness, and Appendix A has a benchmark script — run them on your own data.
- No cloud or vendor required; the three reference configurations in Section 10 are provider-neutral.
Version & updates
Version 1.0 · released 2026-09-27 — Version 1.0, released 2026-09-27. Corrections and additions are versioned in the changelog — never a silent rewrite.
License
Personal and internal team use. You may use and adapt these materials for your own deployments. You may not resell, redistribute or republish the playbook, in whole or in part, as a standalone product.
Frequently asked questions
Do I need a specific vector DB or model?
No. It covers the decisions that hold across providers and gives you three reference configurations to adapt.
Does it contain code?
Yes — a reproducible Ragas eval harness and a benchmark script in the appendices. The rest is the reasoning and the decision frameworks.
Is it suitable for beginners?
It assumes you have a RAG system, or are about to build one, and know what retrieval is. It is a field playbook, not an introduction.
What are the seven failure modes?
Parsing rot, chunk boundary amputation, retrieval recall cliff, permission leakage, stale index, evaluating on vibes, and cost blow-up — each with the fix that addresses it.
The FDE Deployment Toolkit 2026
Reference architectures, a 60-point go-live checklist, GPU cost models and a vendor shortlist — the four artefacts you rebuild on every single deployment.
30 AI Deployment Patterns from Live FDE Roles
Thirty deployment patterns extracted from real, live FDE job specs — each with the original employer ATS link, an editorial pattern label and an explicit list of what this data cannot tell you.
Enterprise Prompt Pack — 208 Task-Specific Templates
208 original, task-specific prompt templates across eight enterprise workflows, each with variables, an explicit output contract and an honest statement of what has not been benchmarked.