skip to content

learn.smmeyer.devllm-engineering09

menu

llm-engineering/09

knowledge systems · core · ~2h practice

Embeddings

outcome

Reason about embedding applications, distance metrics, indexing, and approximation tradeoffs.

Plan a practice session · ~2h including exercises
Read + trace the worked example
20 min
Assignment + constraint variations
60 min
Timed drill + repeated explanations
25 min
Closed-book recall + self-review
15 min

This is a suggested practice budget, including drawing, variations, and repeated drills. Reading is the first block. Split the work across sessions and use the rubric to judge when to move on.

why this matters

Embeddings map inputs into vectors where geometric proximity can reflect learned similarity. They enable retrieval and grouping, but the geometry is task- and model-dependent—not universal meaning.

coverage

  • Representation: semantic vector spaces and model dependence.
  • Metrics: cosine similarity, dot product, Euclidean distance.
  • Applications: search, clustering, classification, recommendations, deduplication.
  • Scale: dimensionality, exact search, approximate nearest neighbors.
  • Indexes: graph, inverted/product-quantized intuition.
  • Quality: thresholds, hard negatives, drift, evaluation.

mental model

Embed query and candidates into one space, retrieve neighbors, then apply task constraints.

  1. 1 →Encode

    Input → vector

  2. 2 →Index

    Organize space

  3. 3 →Compare

    Distance/similarity

  4. 4 →Filter

    Task + metadata

Read in numbered order: encode → index → compare → filter.

lessons

01Similarity depends on geometry

Cosine compares direction, dot product includes magnitude, and Euclidean measures straight-line distance. If vectors are normalized, cosine and dot-product ranking align. Use the metric expected by the embedding/index setup.

An embedding model places inputs in a learned vector space. Similarity depends on what the model was trained to make nearby, so two equally sized vectors from different models do not automatically share meaningful coordinates. Use compatible encoders and the intended metric. Semantic proximity can help retrieve candidates, but a nearby document may contradict the query or lack the decisive detail.

Cosine similarity divides the dot product by both vector norms. For unit vectors [1,0] and [0.8,0.6], the dot product and cosine are 0.8. Without normalization, vector magnitude changes dot-product ranking. A similarity of 0.8 is not an 80% probability of relevance. Set thresholds using labeled task data and examine hard negatives such as similar wording with the wrong account or policy version.

  • Thresholds are dataset-specific
  • Nearest does not mean correct
  • Domain language can shift neighborhoods

02ANN trades exactness for speed

Scanning every vector is simple but expensive at large N. Approximate nearest-neighbor indexes prune the search space, trading memory, build/update cost, and some recall for latency.

Exact search compares the query with every candidate and gives a useful baseline on a small corpus. At large scale, approximate nearest-neighbor indexes avoid many comparisons using structures such as navigable graphs or compressed partitions. Increasing search effort often improves recall at a latency cost; memory and index build/update cost also matter. Measure the frontier on your data.

Distinguish ANN recall from task retrieval quality. An approximate index can accurately reproduce the nearest vectors while the embedding model ranks irrelevant documents highly. Conversely, a good representation can be harmed by aggressive index approximation. Compare exact vector search first, then ANN, then downstream answer quality so you know which layer caused a regression.

  • Tune recall vs latency empirically
  • Filter strategy affects index performance
  • Deletes and model upgrades require lifecycle plans

03Embeddings are reusable features

Beyond search, vectors support clustering, nearest-neighbor classification, recommendations, anomaly hints, and duplicate detection. For final decisions, combine them with labels, rules, and evaluation.

The same vectors can support nearest-neighbor classification, clustering, recommendations, and duplicate detection, but each task needs its own evaluation. Similarity useful for recommendations may be too loose for identifying duplicate legal clauses. Include identifiers, negation, domain jargon, and multilingual examples when those occur in production. Preserve metadata constraints alongside vector features.

An embedding upgrade is a data migration. Re-encode documents with the compatible new model, build a new index, compare query results and operational cost, then switch traffic gradually. Keep the old index for rollback until the new one is validated. Coordinate streaming updates and deletions during the overlap so a newly built index does not silently resurrect removed documents.

  • Use hard negatives that look deceptively similar
  • Version embedding model with the index
  • Re-embed or dual-index during migrations

worked example

Compare a semantic match with an exact identifier

A search query is ‘reset workspace access’, but another query is the exact error code E-1049. The index stores normalized embedding vectors.

  1. For normalized vectors [1,0] and [0.8,0.6], the dot product is 0.8, equal to their cosine similarity.
  2. Semantic similarity can retrieve paraphrases for the first query. For E-1049, a lexical match may preserve the decisive identifier better.
  3. Use a measured hybrid retrieval approach and rerank on representative queries; do not assume a cosine score has the same meaning across models.
  4. When changing the embedding model, rebuild compatible document vectors and compare quality, latency, and memory before switching traffic.

Design decision: Choose retrieval behavior from the query distribution and task-level recall, including exact-match cases.

Limit: A similarity score is not a calibrated probability of relevance. Thresholds and ANN recall depend on the dataset and implementation.

Teaching scenario · traffic, prices, and capacity values are illustrative.

assignment

Produce: A ten-query retrieval set, an exact/semantic query comparison, and an embedding migration plan.

Paper, a text editor, or a spreadsheet is enough. Model access and paid services are optional.

  1. Calculate two small cosine examples and explain why normalization matters for a dot-product comparison.
  2. Write paraphrase, identifier, negation, and multilingual query cases with expected relevant documents.
  3. Compare exact search, vector search, and hybrid search on paper; define the measurements and rollout checks needed for a model change.

Change a constraint

  • The embedding dimension doubles.
  • A new tenant's jargon differs sharply from the initial corpus.

Self-review rubric

  • Similarity is not described as truth or probability.
  • Exact identifiers have a deliberate retrieval path.
  • Migration preserves model/index compatibility and a rollback.

If a criterion is missing, revise that part of the artifact before rating your practice below.

Compare a sample artifact after your attempt

One defensible answer under these assumptions. Compare the reasoning and checks with your work; other designs can satisfy the same contract.

Vectors are normalized and compatible with their index. This miniature query set illustrates coverage rather than a measured retrieval benchmark.

Dot/cosine: [1,0]·[0.8,0.6]=0.8; [1,0]·[0,1]=0
Queries → expected document:
01 reset workspace access→access guide; 02 E-1049→exact error entry;
03 cannot sign in→access guide; 04 invoice INV-7→exact invoice;
05 no refund without receipt→conditional policy; 06 refund→current policy;
07 Spanish login paraphrase→access guide; 08 tenant jargon→labeled glossary;
09 obsolete deadline→current policy; 10 near-identical wrong product→exclude.
Compare: lexical / exact vector / ANN / hybrid, same gold set.
Migration: compatible re-embed → new index+updates/deletes → evaluate
           → canary → retain old index for rollback.

Exact identifiers favor a deliberate lexical path. A vector score is not relevance probability. Doubling dimension can increase raw vector bytes, but index overhead and representation quality need measurement. Compare exact vector search before ANN to distinguish representation failure from approximation loss.

Check your artifact

  • Both cosine calculations use unit vectors.
  • Query and document vectors come from a compatible embedding setup, not merely matching dimensions.
  • The migration handles updates, deletion, tenant filters, and rollback.

A tempting mistake: A new model with the same dimension does not make its vectors compatible with the old document index.

cumulative practice

Harbor · build and evaluate a support assistant

Project brief

Keep one project document and a folder of fictional input/output fixtures for Harbor, a workspace-scoped support assistant. Start with ticket triage and policy-backed drafts at 20 requests/minute, 1,500 input and 200 output tokens/request. Human users remain responsible for consequential actions until you design explicit permissions and approval. Later chapters add retrieval, controlled tools, durable runs, evaluation, security, and scale. Use paper, a text editor, and mocked results; paid APIs are optional. Preserve first attempts, evidence, and revisions within the existing practice budget.

Keep one project document across this track. Reconstruct earlier reasoning before opening notes, then revise the same artifact. Use part of the assignment and review budget for this work. If you enter mid-course, sketch the earlier decisions first using the linked chapters.

Add to your project: Harbor's exact/semantic retrieval comparison and migration record.

Reuse the corpus and gold questions from chapter 8. Add an exact error code and a paraphrase. Recall the difference between a data-split failure and retrieval failure. Plan an embedding upgrade while keeping authorized updates and tombstones correct.

Compare cumulative reasoning after your attempt

Measure lexical, exact vector, and ANN retrieval against the same caller-scoped evidence labels. Keep held-out queries separate from tuning. Build a compatible new document index, replay changes and deletions, evaluate slices, canary, and retain rollback. Strong exact-neighbor recall cannot compensate for an embedding space that misses the task's relevant documents.

Keep your first attempt and revision together. If your answer differs, identify the assumption or mechanism that explains the difference before rating your recall.

senior signal

State the metric and normalization assumption; then name the target recall/latency curve you would measure.

failure patterns

  • Treating similarity score as calibrated probability
  • Changing embedding model without rebuilding/comparing the index
  • Using one global threshold for every domain
  • Assuming approximate search returns the true nearest neighbors

knowledge checks

Start with these quick checks, then explain the closed-book questions in your own words.

  1. 01With normalized vectors, cosine ranking matches…

    select an option to check your answer

  2. 02ANN primarily trades…

    select an option to check your answer

closed-book recall

Close the worked example and explain these from memory. Answers are self-reviewed. Drafts stay on this page; save anything you want to keep in chapter notes.

Rate the assignment and recall together

Use the assignment rubric as evidence. Completion is your own judgment; this score schedules practice.

interview drill

Choose an embedding and indexing plan for 200M code symbols used by a coding assistant.

  • 200M vectors
  • code + prose queries
  • frequent updates
  • 50ms retrieval
compare a strong approach after your attempt

Evaluate candidate embeddings on real code queries and hard negatives. Use an ANN index with language/repo filters, measure recall@k under 50ms, version vectors by model, and stream updates/tombstones. Hybridize with lexical symbol matching for exact names.

Compare decisions and failure behavior, then repeat with a changed constraint. Keep your own wording.

Use the timed mock workspace →

references

Sources offer deeper context. Provider capabilities and API contracts can change; check current official docs before implementing.

notes

saved locally