Choosing a vector store: four questions that settle it
Published benchmark tables compare recall on public datasets, which is not your dataset, your filters, or your query distribution. Four operational questions decide it instead.
Why the benchmark tables mislead
Nearly every comparison you will find measures recall and queries per second on a public dataset with no metadata filters. Your application has a specific corpus, selective filters on most queries, and a latency budget that includes network and reranking. Under those conditions the ranking on a benchmark table frequently inverts.
This is not an argument against measuring. It is an argument for measuring the thing you will actually run.
Question one: how big is the corpus, measured?
Count chunks, not documents, and multiply by dimensions and precision to get a rough memory footprint. If the result fits comfortably in the memory of the machines you already run, a single-node option will be simpler and cheaper than anything distributed. Teams adopt distributed stores years before they need them, and pay for it in operational surface rather than money.
Question two: do most queries carry a filter?
This is the question that separates the stores. If every query is constrained by tenant, permission, or recency, you need filtering in the index rather than after the search, and you should test with your real filter selectivity. A store that looks fast on unfiltered search can fall apart when ninety-nine per cent of candidates are excluded.
Question three: who operates it?
Be honest about on-call. If nobody wants to run a distributed system, a managed service or an extension to a database you already operate is the correct answer regardless of what wins a throughput comparison. The cheapest store is the one your team does not wake up for.
Question four: what is the exit cost?
Before you commit, know how you would leave. Can you export vectors and rebuild elsewhere? Is the query language portable, or does the SDK own your code? A store that is cheap to enter and expensive to leave is a liability that shows up at exactly the wrong moment — when you have traffic and no leverage.
How to run the test
Take two hundred real queries with their real permission filters. Load your actual corpus into two candidate stores. Measure end-to-end latency from your application, recall at the k you will use, and the operational steps to add one million vectors. One afternoon of this beats any published table.
What to do about it
- Public benchmarks omit your filters and your latency budget, and often invert under real conditions
- Measure corpus size in chunks and memory, not documents
- If most queries are filtered, test filtering inside the index with your real selectivity
- Know the exit cost before you commit — export, portability, and who owns the query language
RAG in Production Playbook
Chunking strategies, reranking setups, evaluation harnesses and the seven failure modes that kill every RAG demo in week three.