RAG, deeply — retrieval quality is the ceiling

1First principles — why RAG exists

An LLM's knowledge is (a) frozen at training time, (b) lossily compressed into weights, and (c) missing everything private to you. Three fixes exist: retrain/fine-tune (slow, expensive, still lossy), stuff everything into context (costly, capped, degrades — §10), or fetch just the relevant knowledge at query time and put it in context. RAG is option three.

The reframe that shows understanding: RAG converts a recall problem into a reading-comprehension problem. Models are unreliable at recalling tail facts from their weights, but very good at answering over text sitting right in front of them.

2The pipeline — every stage has a failure mode

# INGESTION (offline)
documents → parse/clean → chunk → embed → index (vector DB + keyword)
# QUERY (online)
query → [rewrite] → embed → retrieve (dense + BM25) → rerank
      → assemble context → generate → [cite/verify]

Ingestion — offline

Query — online

3Ingestion — the unglamorous 80%

Real-world RAG quality is mostly determined here, before any AI happens:

• Parsing: PDFs with multi-column layouts, tables, headers/footers; HTML boilerplate; scanned docs needing OCR. Garbage parsing → garbage everything. Tables are the classic killer — flattened row-by-row into text, they lose the row↔column association the answer depends on.
• Cleaning & metadata: strip nav/boilerplate, keep structure (headings, section paths); attach metadata per chunk — source, title, section, date, access permissions. Metadata powers filtering ("only docs from this repo", "only this user's tenant") and citations later.
• Access control is metadata too — retrieval must respect document permissions, or RAG becomes a data-leak machine. A genuinely senior point to raise unprompted.
• Freshness: documents change. You need incremental re-indexing (hash content, re-embed changed chunks) and deletion propagation — a stale index confidently serves last quarter's policy.

4Chunking strategies — and their failure modes

Why chunk at all? (1) Embeddings compress a passage into one vector — embed a whole document and the vector becomes a blurry average, matching nothing well. (2) Context budgets force selectivity. (3) Precise chunks make citations meaningful. Chunking trades retrieval precision (small chunks) against answer completeness (large chunks).

Cross-cutting failure modes to name

• Context-stripping: a chunk saying "it must be rotated every 90 days" retrieves poorly and reads ambiguously — what must? Fixes: prepend title/section-path to every chunk; or contextual enrichment — an LLM writes 1–2 situating sentences per chunk at ingest (Anthropic's "contextual retrieval" idea).
• Chunk-size mismatch with question type: factoid questions want small precise chunks; summary/"how does X work" questions want large ones. Fix: small-to-big retrieval — index small chunks for matching precision, but hand the LLM the parent section for completeness. This one technique resolves the central tension — a strong thing to volunteer.
• Tables/lists split across chunks → silently wrong numeric answers.

5Embeddings — what they are, precisely

An embedding model maps text → a fixed-length vector (typically 384–3072 dims) such that semantic similarity ≈ geometric proximity. Trained contrastively: pull related pairs (question↔answer, paraphrases) together, push unrelated apart. The result: "How do I revoke a token?" lands near "Invalidating API credentials" despite zero shared words — the thing keyword search can't do.

Operational facts an engineer is expected to know

• Similarity metrics: cosine similarity (angle between vectors — the default; magnitude-invariant), dot product (unnormalized; magnitude matters), Euclidean distance. Key detail: on normalized vectors all three give identical rankings — so the practical rule is "normalize embeddings, use cosine/dot, move on." Use the metric the embedding model was trained for.
• Asymmetric search: queries and documents are different kinds of text ("how do I reset my password" vs a policy paragraph). Good retrieval embedding models are trained for this; some require prefixes (query: … / passage: …) — ignoring that visibly degrades retrieval.
• The embedding model is a versioned dependency: vectors from different models/versions are incompatible — changing embedding models means re-embedding the entire corpus. Plan for it: store raw text + metadata so re-embedding is a batch job, not an archaeology project.
• Limitations that cause real failures: embeddings blur negation ("eligible" vs "not eligible" sit close); exact identifiers — error codes, SKUs, function names, version numbers — are where dense retrieval is weakest and keyword search is strongest. This is the setup for hybrid search.

6Vector DBs, hybrid search, and BM25

Vector DB = approximate nearest-neighbor (ANN) search + database features. Exact k-NN over millions of vectors is too slow; ANN indexes trade a little recall for orders-of-magnitude speed: HNSW — a navigable graph of neighbors, the dominant one — and IVF — cluster-then-search. The "database features" matter as much as ANN: metadata filtering combined with vector search (filter + search interact non-trivially), CRUD/upserts, namespaces/multi-tenancy.

BM25 — the classic keyword ranker (Elasticsearch/OpenSearch default): score by term overlap, weighting rare terms high (IDF), diminishing returns on repetition, normalized by doc length. Its strengths are exactly dense retrieval's weaknesses: exact matches on identifiers, error strings, names, jargon. Its weakness is exactly dense's strength: zero synonym/paraphrase understanding.

Hybrid search = run both, merge. Standard merge: Reciprocal Rank Fusion (RRF) — score each doc by Σ 1/(k + rank) across result lists; rank-based, so you dodge incomparable score scales. Hybrid is the production default worth asserting: "For a corpus with error codes and API names — like production logs or docs — I'd start hybrid; pure dense search whiffs on ERR_CONN_RESET-type queries, pure BM25 whiffs on paraphrases."

7Reranking and context assembly

Reranking — the precision stage

First-stage retrieval uses a bi-encoder: query and doc embedded separately, compared geometrically — fast but coarse. A cross-encoder reranker reads query+document together through one model and scores actual relevance — far more accurate, far slower. So: retrieve top-50 cheaply → rerank → keep top-5–10. This two-stage retrieve-then-rerank design is the standard IR architecture; typical gains are large, cost is +100–300ms and a per-query model call (Cohere Rerank, BGE-reranker, or an LLM-as-reranker).

Context assembly — the neglected stage

• Dedupe and diversify — near-identical chunks waste budget (MMR if needed); order deliberately — lost-in-the-middle: put the strongest evidence near the start or end, not buried mid-context.
• Attach citation handles — label chunks [1] source: runbook/payments.md#refunds and instruct answers to cite. Citations enable verification and discipline the model toward grounded claims.
• Prompt the abstention path: "Answer only from the provided context; if it's insufficient, say so." Without an explicit out, the model fills retrieval gaps from its weights — the primary hallucination vector in RAG.
• Query-side transforms (before retrieval, worth one breath): query rewriting for conversational follow-ups ("what about staging?" → self-contained query from chat history — table stakes for chat-RAG); HyDE (embed a hypothetical answer); decomposition for multi-hop questions.

8RAG evaluation — the triad

The key move: evaluate retrieval and generation separately — end-to-end scores can't tell you which stage to fix.

Retrieval metrics (needs a golden set of query → relevant-chunk labels): recall@k — is the answer's evidence in the top-k? — the single most important number; precision@k; MRR/nDCG — is it ranked high? Cheap to compute, no LLM judge needed, and directly actionable: low recall → fix chunking/embeddings/hybrid; low precision with good recall → add reranking.

Generation metrics — the RAG triad (RAGAS/TruLens vocabulary):

9Common failure modes → fixes (the cheat table)

10When NOT to use RAG

Long context instead

When the working set is bounded and fits comfortably — one contract, one codebase module, one incident's logs — just put it in context: zero infra, no retrieval-recall risk, the model sees everything (no "the relevant chunk wasn't retrieved" failure class). Costs: per-request token cost and latency, and long-context degradation on complex reasoning — but prefix caching makes repeated queries over the same corpus dramatically cheaper, eroding RAG's cost argument for small corpora. Rule of thumb: bounded working set that fits → long context; large/unbounded/multi-tenant corpus, strict latency/cost budgets, or citation requirements → RAG. (Hybrid: retrieval to select documents, long context to read them whole.)

Fine-tuning instead (or alongside)

The crisp separation: RAG injects knowledge; fine-tuning shapes behavior. Fine-tune for consistent style/format/domain dialect, reliable tool-use patterns, or distilling a big model's behavior into a small cheap one at volume. Fine-tuning is a bad knowledge store: facts don't reliably stick, updates require retraining, no citations, and you now own model versioning + eval + serving. They compose — a fine-tuned model inside a RAG pipeline: behavior from weights, facts from retrieval.

No augmentation at all

General-knowledge tasks the base model already does well; transformation tasks where all needed information is in the request itself (summarize this, refactor this). Retrieval bolted onto those adds latency, cost, and a new way to inject irrelevant context.

When agentic retrieval beats classic RAG

If queries need investigation — follow references, search iteratively, decide what to look for next — a tool-using agent with a search tool outperforms one-shot retrieve-then-generate. That's the next page — and closer to how Sentinel pulls logs via MCP: targeted, tool-driven fetching rather than a pre-embedded corpus. Being able to say "my production system deliberately uses tool-based retrieval rather than embedding a corpus, because incident context is fresh, structured, and queryable" turns this from theory into your story.

11Rapid-fire interview Q&A

12Self-test