llm-engineering/08

knowledge systems · core · ~4h practice

Retrieval-augmented generation

outcome

Build and evaluate a permission-aware retrieval pipeline from ingestion through grounded output.

Plan a practice session · ~4h including exercises

Read + trace the worked example: 30 min
Assignment + constraint variations: 140 min
Timed drill + repeated explanations: 50 min
Closed-book recall + self-review: 20 min

This is a suggested practice budget, including drawing, variations, and repeated drills. Reading is the first block. Split the work across sessions and use the rubric to judge when to move on.

why this matters

RAG injects external evidence into model context. Its quality ceiling is usually retrieval: if the right evidence is missing, buried, stale, or unauthorized, generation cannot reliably repair it.

coverage

Ingestion: parsing, normalization, chunking, metadata.
Retrieval: embeddings, vector search, lexical search, filters.
Ranking: hybrid fusion, reranking, diversity.
Assembly: context budget, ordering, citations.
Freshness: updates, deletion, index lag.
Security: document-level permissions and tenant isolation.
Failure: bad/missing/excess/conflicting context.
Evaluation: recall, ranking, answer grounding.

mental model

RAG is a search system feeding an evidence-constrained generation system.

1 →Ingest
Parse + chunk
2 →Retrieve
Hybrid + ACL
3 →Rerank
Relevance + diversity
4 →Generate
Cite + abstain

Read in numbered order: ingest → retrieve → rerank → generate.

lessons

01Chunk for the question shape

Chunks should be coherent enough to answer likely questions and small enough to rank precisely. Preserve headings, source location, timestamps, and permissions. Consider hierarchical parent/child retrieval for long documents.

Retrieval adds external evidence to the model's input. A lexical index matches terms such as exact error codes; an embedding maps text to a learned vector so geometrically nearby items may capture paraphrases. A vector is a retrieval feature, not a truth score. The next chapter develops that geometry; here, treat semantic retrieval as one candidate-generation method whose results must be evaluated.

Ingestion determines what can later be found. Preserve headings, source IDs, versions, permission metadata, and useful table or code structure. A chunk that cuts a condition away from its exception can look relevant while being insufficient. Compare chunking on known question/evidence pairs: does the required passage survive as a coherent retrievable unit, and can its source still be cited?

Overlap repairs boundary loss but creates duplicates
Tables and code need structure-aware parsing
Index updates and deletes need lifecycle tracking

02Hybrid retrieval covers different failure modes

Lexical search excels at exact identifiers and rare terms; semantic search covers paraphrase and concept similarity. Metadata filters enforce scope; rerankers spend more compute on a small candidate set.

For “What does E-1049 mean?”, lexical matching may retrieve the decisive identifier while semantic similarity returns general access problems. For “How can I get back into my workspace?”, a paraphrase match may help. Combine candidate sets with a documented fusion strategy, then rerank a smaller pool. Tune retrieval candidate count separately from final context size; the model need not see every candidate.

Enforce authorized scope in the retrieval system and recheck before publication where permission changes require it. Some engines apply filters during search; others filter candidates afterward. Measure the chosen strategy's recall and performance, because filtering away most candidates can leave too little usable evidence. Never let restricted text reach generation merely because you plan to remove citations later.

Vector similarity is not authority
Diversify near-duplicate chunks
Tune candidate count separately from final context count

03Evaluate retrieval before prose

Build queries with known relevant evidence. Measure recall@k and ranking, then evaluate citation support, answer completeness, and abstention. Inspect missing-context and conflicting-source slices.

Build questions with known authorized gold evidence. Ask whether the source was ingested, whether retrieval found it, whether reranking retained it, and whether context assembly included it. If evidence was lost before generation, changing answer wording does not repair the root cause. Keep source/version IDs in traces so failures can be localized to a stage.

Then judge the answer separately: are its claims supported, are important facts missing, do citations point to supporting passages, and does it abstain when needed? More top-k can increase recall but also add distracting or contradictory material. Evaluate stale, deleted, restricted, and missing-source cases. A grounded answer based on an obsolete policy may still be wrong for the current user task.

Too much context can distract and raise cost
Prefer recent/authoritative sources via metadata
Make evidence visible to users when useful

worked example

Separate retrieval failure from unsupported generation

An employee asks for the current travel policy. Search returns an old policy plus a current document the employee is not allowed to read.

Enforce caller and tenant permissions in retrieval and again before evidence publication where required. A relevant restricted document is still unusable.
Choose evidence freshness and version rules. If only the old authorized policy remains, the system may need to abstain or state its date.
Evaluate whether the authorized gold evidence appears in the candidates, survives reranking, and reaches the final context.
Then separately check whether the answer's claims are supported by that context and whether its citations identify the supporting passages.

Design decision: Instrument each stage so a wrong answer can be assigned to ingestion, retrieval, filtering, evidence construction, or generation.

Limit: A larger top-k can improve candidate recall while adding irrelevant or contradictory evidence. It is not a universal quality improvement.

Teaching scenario · traffic, prices, and capacity values are illustrative.

lab

interactive lab

RAG Pipeline — recall, groundedness, and cost per knob

1 · predict first

Raising top-k from 3 to 12: does groundedness go up (more evidence) or down (more noise)? Commit before dragging.

pedagogical model — values are illustrative, not measured.

assignment

Produce: A small fictional corpus, six questions with gold evidence, and a RAG failure table.

Paper, a text editor, or a spreadsheet is enough. Model access and paid services are optional.

Write current, outdated, ambiguous, and restricted source snippets; assign tenant, version, and access metadata.
Trace ingestion, query, hybrid retrieval, filtering, reranking, context, and cited answer; use the lab to test top-k and reranking tradeoffs.
Score retrieval recall and answer support separately, then revise the weakest stage without changing all stages at once.

Change a constraint

The relevant answer is split across two documents.
A source includes text instructing the assistant to reveal another workspace's files.

Self-review rubric

Gold evidence is permission-scoped.
Freshness and missing-evidence behavior are explicit.
Retrieval quality and answer faithfulness have separate measurements.

If a criterion is missing, revise that part of the artifact before rating your practice below.

Compare a sample artifact after your attempt

One defensible answer under these assumptions. Compare the reasoning and checks with your work; other designs can satisfy the same contract.

This fictional corpus belongs to workspaces T and U. A T member cannot use U's policy. Dates are exercise versions.

Corpus: T/v1 old “14 days”; T/v2 current “30 days with receipt”;
T/v2 FAQ “receipt may be electronic”; U/v2 restricted “60 days”.
Gold questions/evidence:
Q1 current T deadline→T/v2; Q2 receipt format→FAQ;
Q3 deadline+format→T/v2+FAQ; Q4 U deadline as T user→abstain;
Q5 unknown exception→abstain; Q6 old deadline vs new→T/v2 + version rule.
Failure table: absent source→ingestion; missing candidate→retrieval;
lost after ranking→rerank/assembly; unsupported claim→generation;
restricted evidence→authorization defect; obsolete answer→freshness defect.

Gold evidence is caller-scoped. Measure candidate recall, final-context evidence coverage, and answer support separately. Split evidence can require more than one chunk. Injected instructions in a source remain untrusted content and cannot grant another workspace's access.

Check your artifact

Restricted U text never enters T's generation context.
Current source/version rules choose T/v2 over T/v1.
An abstention case is scored as intended behavior rather than forcing every query to have an answer.

A tempting mistake: Increasing top-k cannot repair a missing authorized source and may add more stale or distracting evidence.

Put this chapter into practice: Repair an unsupported policy answer · guided worksheet →

cumulative practice

Harbor · build and evaluate a support assistant

Project brief

Keep one project document and a folder of fictional input/output fixtures for Harbor, a workspace-scoped support assistant. Start with ticket triage and policy-backed drafts at 20 requests/minute, 1,500 input and 200 output tokens/request. Human users remain responsible for consequential actions until you design explicit permissions and approval. Later chapters add retrieval, controlled tools, durable runs, evaluation, security, and scale. Use paper, a text editor, and mocked results; paid APIs are optional. Preserve first attempts, evidence, and revisions within the existing practice budget.

Keep one project document across this track. Reconstruct earlier reasoning before opening notes, then revise the same artifact. Use part of the assignment and review budget for this work. If you enter mid-course, sketch the earlier decisions first using the linked chapters.

Add to your project: Harbor's authorized evidence pipeline using the existing support fixtures.

Recover the missing-policy fixture from chapter 1. Add current, old, and restricted policy snippets and gold evidence labels. Trace what happens when the right source is retrieved but dropped during context assembly. Which earlier prompts and workflow states need revision?

Compare cumulative reasoning after your attempt

The initial retriever may have succeeded; inspect permission filtering, reranking, deduplication, and token-budget assembly before changing the model. Preserve source/version IDs and require cited support or abstention in the prompt. Add an evidence-assembly state and quality checks to the workflow. Compare the original unsupported fixture with the supported version while retaining stale and denied cases as regressions.

Keep your first attempt and revision together. If your answer differs, identify the assumption or mechanism that explains the difference before rating your recall.

senior signal

In an interview, separate retrieval recall, context precision, and generation faithfulness. Each needs different fixes.

failure patterns

Using only vector similarity for product codes
Adding more chunks whenever answers fail
Filtering permissions after generation
Measuring only final answer vibes

knowledge checks

Start with these quick checks, then explain the closed-book questions in your own words.

01Why combine lexical and semantic retrieval?
select an option to check your answer
02What should be measured before answer style?
select an option to check your answer

closed-book recall

Close the worked example and explain these from memory. Answers are self-reviewed. Drafts stay on this page; save anything you want to keep in chapter notes.

1. The correct passage is in top-20 candidates but absent from the final answer context. Where would you investigate?
2. Why should ‘no authorized evidence’ be an evaluated outcome?

interview drill

Design enterprise RAG over 100M documents with per-document ACLs and 5-minute freshness.

100M docs
ACLs
5m freshness
citations

compare a strong approach after your attempt

Stream document changes into parse/chunk/embed/index stages with versioned tombstones. Query with tenant and ACL filters before scoring, fuse lexical/vector candidates, rerank, and assemble diverse evidence. Store source/version IDs for citations and trace retrieval. Evaluate recall/freshness/security slices independently.

Compare decisions and failure behavior, then repeat with a changed constraint. Keep your own wording.

Use the timed mock workspace →

references

Lewis et al.: Retrieval-Augmented Generation ↗reviewed 2026-10
Sentence Transformers: semantic search ↗reviewed 2026-10
OWASP: prompt injection ↗reviewed 2026-10

Sources offer deeper context. Provider capabilities and API contracts can change; check current official docs before implementing.

notes

saved locally