RAG in Production: What Breaks After the Demo

A retrieval-augmented generation prototype takes an afternoon. Embed some documents, drop them in a vector store, retrieve the top few by similarity, hand them to a model. It works immediately, which is exactly the problem — the demo gives no signal about the parts that get hard later.

Almost every difficulty in production RAG is a retrieval problem wearing a generation costume. If the right passage never reaches the model, no amount of prompt engineering recovers it.

Chunking decides your ceiling

Chunking is treated as preprocessing and is closer to schema design. Split too small and passages lose the context that made them meaningful. Split too large and the embedding blurs several topics into an average that matches nothing well.

A few things that help:

• Split on structure, not character counts. Headings, sections, and list boundaries carry meaning that a fixed 512-character window destroys.
• Overlap deliberately. Enough that an answer spanning a boundary survives; not so much that near-duplicates crowd out the results.
• Keep the parent reference. Retrieve the precise chunk, then expand to its surrounding section before handing anything to the model.
• Carry metadata through. Source, section, and date make filtering and citation possible later.

That last point matters more than it looks. A chunk that cannot say where it came from cannot be cited, and an answer that cannot be traced cannot be trusted.

Pure vector search is not enough

Embeddings capture meaning and are correspondingly weak on things that are not about meaning: product codes, error numbers, surnames, version strings. A user searching for ERR_4021 wants that exact token, and cosine similarity is relaxed about exactness.

Hybrid retrieval — dense vectors alongside keyword search — is the usual fix. Run both, then merge. Reciprocal rank fusion works well and needs no tuning, which is more than can be said for hand-picked score weights.

// Merge two ranked lists by position rather than by score.
// Scores from different retrievers are not on a comparable scale.
function reciprocalRankFusion(
  rankings: string[][],
  k = 60
): Map<string, number> {
  const scores = new Map<string, number>();

  for (const ranking of rankings) {
    ranking.forEach((id, index) => {
      scores.set(id, (scores.get(id) ?? 0) + 1 / (k + index + 1));
    });
  }

  return scores;
}

The instinct to normalise and blend raw scores is worth resisting. A dense score of 0.82 and a BM25 score of 12.4 have no shared meaning, and any weighting you pick will be tuned to whatever queries you happened to try.

Retrieve widely, then narrow

Retrieval optimises for recall; the model needs precision. These pull in opposite directions, and the standard resolution is two stages: fetch generously, then re-rank with something more expensive and more accurate before passing a short list on.

A cross-encoder re-ranker reads the query and passage together rather than comparing two independently computed vectors, so it catches relevance that embedding similarity misses. It is too slow to run over an entire corpus and quite fast enough over fifty candidates.

Measure retrieval separately

The most common evaluation mistake is judging only the final answer. When it is wrong, you cannot tell whether retrieval missed the passage or the model ignored it — and those have completely different fixes.

Evaluate the stages independently:

A modest set of question-and-source pairs, curated by hand, is worth more than a large synthetic one. Fifty real questions with known answers will find more problems than a thousand generated from the documents you are testing against — those tend to test whether the corpus contains itself.

Failure modes to plan for

• Confident answers from nothing. When retrieval returns weak matches, models fill the gap anyway. Threshold on score and let the system say it does not know.
• Stale content. Embeddings persist after their source changes. Re-embedding needs to be part of the ingest pipeline, not an occasional cleanup.
• Near-duplicates. Five chunks saying the same thing crowd out the one that adds something. Deduplicate after retrieval.
• Context dilution. More passages is not more accuracy. Beyond a point, relevant material competes with padding.

Wrap-up

The gap between a RAG demo and a RAG product is almost entirely in retrieval quality and honest evaluation. Chunk along the structure the documents already have, combine dense and keyword search, re-rank before you generate, and measure the stages separately so a failure tells you where to look.

None of it is glamorous, and all of it decides whether the answers hold up.