skip to content

learn.smmeyer.devllm-engineering01

menu

llm-engineering/01

foundations · core · ~2h practice

Mental model of modern LLMs

outcome

Explain what an LLM does, why generation varies, and why hallucination is a natural failure mode.

Plan a practice session · ~2h including exercises
Read + trace the worked example
20 min
Assignment + constraint variations
60 min
Timed drill + repeated explanations
25 min
Closed-book recall + self-review
15 min

This is a suggested practice budget, including drawing, variations, and repeated drills. Reading is the first block. Split the work across sessions and use the rubric to judge when to move on.

why this matters

You need a crisp first-principles model that supports engineering decisions. The model predicts a distribution over the next token; useful behavior emerges from training, context, post-training, and sampling—not a database lookup of answers.

coverage

  • Representation: tokens, tokenization, context windows.
  • Objective: next-token probability distributions.
  • Sampling: temperature, top-p, determinism and seeds.
  • Failure: hallucination, ambiguity, missing knowledge.
  • Scale: why larger/capable models behave differently.
  • Lifecycle: training vs inference.

mental model

Generation repeatedly turns context into a distribution, samples, appends, and repeats.

  1. 1 →Tokenize

    Text → IDs

  2. 2 →Predict

    P(next token)

  3. 3 →Sample

    Policy chooses

  4. 4 →Append

    Context grows

Read in numbered order: tokenize → predict → sample → append.

lessons

01A model predicts continuations

Tokenization maps text into discrete IDs. Given prior tokens, the network produces logits that become a probability distribution. A decoding policy chooses the next token, appends it, and repeats.

A tokenizer might split an unfamiliar word into several pieces and a common word into one. The network consumes IDs and produces a score, called a logit, for each possible next token. Normalizing these scores gives a distribution conditioned on the current input. Choosing one token changes the input for the following step. A complete answer is built through many such steps, so output length adds sequential work.

Keep three forms of information separate. Parameters are learned during training; context is input supplied for this run; application storage persists information outside the model. Adding a policy document to context does not update parameters. Saving a preference in a database does not make it available to the model until the application retrieves and includes it. This distinction explains why a chat system can “remember” between sessions without retraining.

  • Tokens are not exactly words
  • Context is the model's active input—not permanent memory
  • Output length adds sequential latency

02Sampling shapes behavior

Lower temperature sharpens relative probabilities; top-p restricts candidates to a high-probability nucleus. Greedy decoding can be repeatable, but infrastructure and model changes may still affect outputs. Determinism is a spectrum.

Temperature rescales logits before normalization. Lower positive values amplify differences; higher values flatten them. Top-p keeps a small set whose cumulative probability reaches the chosen threshold, then samples within that set. For probabilities 0.6, 0.3, 0.1 and top-p 0.8, the first two remain and normalize to 2/3 and 1/3. The threshold is cumulative mass, not a requirement that each retained token exceed 0.8.

These controls change selection among candidate continuations. They cannot supply a missing refund policy or validate an invoice total. A concentrated distribution can confidently favor a wrong answer. For repeatable application behavior, also record model version, prompt, tools, retrieved evidence, and decoding settings. Identical seeds alone may not establish reproducibility across changed implementations or providers.

  • Temperature does not add knowledge
  • Sampling diversity can improve brainstorming and hurt exact extraction
  • Schema constraints bound form, not factual truth

03Hallucination follows the objective

The training objective rewards plausible next tokens, not an internal truth database. When evidence is absent, conflicting, or weak, fluent completion can outrun grounding. Retrieval, tools, abstention, and evals reduce risk at the system level.

Suppose the prompt asks for a company's refund deadline but supplies no policy. The model may continue with a familiar-looking deadline because that pattern fits the text. The probability of the next token is not a calibrated probability that the entire policy claim is true. Fluent wording and internal consistency are weak evidence of factual support.

Repair the information path: retrieve an authorized current policy, attach its source/version, and require support for the claim or an explicit abstention. Then evaluate missing, stale, contradictory, and malicious evidence separately. Lowering temperature may reduce variation but leaves those information failures intact. Grounding also needs checking: supplying a document does not guarantee every generated claim follows from it.

  • Confidence language is not calibrated probability
  • Capability scales unevenly across tasks
  • Training changes parameters; inference applies them

worked example

A confident completion with no supporting evidence

A support assistant sees ‘What is our refund deadline?’ but receives no company policy. Suppose the next-token candidates have illustrative probabilities 0.6, 0.3, and 0.1.

  1. The distribution describes possible continuation tokens conditioned on this context. It is not a probability that a complete policy claim is true.
  2. Top-p = 0.8 keeps the first two candidates because their cumulative mass reaches 0.9. Renormalizing gives probabilities 2/3 and 1/3 within that set.
  3. Reducing temperature can concentrate sampling on a plausible deadline while leaving the missing company policy unresolved.
  4. Supply authorized policy evidence through retrieval or a tool, then require an evidence-backed answer or an explicit abstention.

Design decision: Treat fluency, sampling variability, and factual support as separate properties of the application.

Limit: These three candidates are a teaching distribution. Actual vocabularies are larger, decoding settings interact, and model APIs may expose different controls.

Teaching scenario · traffic, prices, and capacity values are illustrative.

lab

interactive lab

Next-Token Playground — distribution in, sample out

1 · predict first

At temperature 2.0, does Paris stay above 40%? Commit, then drag the slider.

A guess makes the comparison useful. This lab draft lasts until you leave the page.

pedagogical model — values are illustrative, not measured.

assignment

Produce: A two-minute explanation and three annotated generation traces drawn on paper.

Paper, a text editor, or a spreadsheet is enough. Model access and paid services are optional.

  1. Trace tokenize → predict → sample → append for a tiny invented vocabulary; calculate one top-p renormalization.
  2. Use the lab to compare a concentrated and a diverse distribution, predicting each change first.
  3. Record your explanation for a backend engineer, then revise any claim that confuses token probability with factual confidence.

Change a constraint

  • The prompt contains an incorrect policy supplied by a user.
  • The task changes from brainstorming to extracting an exact invoice field.

Self-review rubric

  • Tokens, context, and parameters have distinct roles.
  • Sampling settings are not described as adding knowledge.
  • The product mitigation addresses missing evidence.

If a criterion is missing, revise that part of the artifact before rating your practice below.

Compare a sample artifact after your attempt

One defensible answer under these assumptions. Compare the reasoning and checks with your work; other designs can satisfy the same contract.

Use an invented three-token vocabulary with probabilities 0.6, 0.3, 0.1. The refund question has no policy evidence.

Trace A: context → [0.6,0.3,0.1] → choose token 1 → append
         next distribution must be computed for the changed context
Trace B: top-p 0.8 → retain tokens 1,2 → normalize [2/3,1/3]
Trace C: supplied authorized policy → new context/distribution
         require a supported deadline or abstain
Two-minute outline: IDs → distribution → selection → repeat;
context differs from parameters; fluency differs from factual support.

The traces show selection and information supply as separate interventions. A lower temperature can concentrate a wrong continuation. A user-provided policy is evidence with a source and authority to evaluate, not automatically a trusted instruction. Exact invoice extraction should be checked against the source and output contract rather than graded for creative variety.

Check your artifact

  • Top-p uses cumulative mass; the normalized retained probabilities sum to one.
  • Appending a token changes the next input; generation does not draw an entire answer from one fixed distribution.
  • Your explanation identifies missing evidence and does not call token probability factual confidence.

A tempting mistake: Saying temperature zero makes the policy true confuses selection with knowledge and verification.

cumulative practice

Harbor · build and evaluate a support assistant

Project brief

Keep one project document and a folder of fictional input/output fixtures for Harbor, a workspace-scoped support assistant. Start with ticket triage and policy-backed drafts at 20 requests/minute, 1,500 input and 200 output tokens/request. Human users remain responsible for consequential actions until you design explicit permissions and approval. Later chapters add retrieval, controlled tools, durable runs, evaluation, security, and scale. Use paper, a text editor, and mocked results; paid APIs are optional. Preserve first attempts, evidence, and revisions within the existing practice budget.

Keep one project document across this track. Reconstruct earlier reasoning before opening notes, then revise the same artifact. Use part of the assignment and review budget for this work. If you enter mid-course, sketch the earlier decisions first using the linked chapters.

Add to your project: Harbor v1: task contract and three annotated output fixtures.

Start a project document from the brief. Make one supported response, one plausible but unsupported response, and one valid abstention for a missing refund policy. Explain what changes in application input can repair the unsupported case.

Compare cumulative reasoning after your attempt

A supported fixture cites an authorized policy span; the unsupported one invents a deadline; abstention states that no current permitted policy was supplied. Retrieval or an authorized policy tool changes the evidence available to generation. Sampling changes alone do not repair missing support. Preserve fixtures and expected properties for later evaluation rather than treating confident prose as success.

Keep your first attempt and revision together. If your answer differs, identify the assumption or mechanism that explains the difference before rating your recall.

senior signal

Explain this without anthropomorphism: “It computes a context-conditioned distribution and generates under a decoding policy; truthfulness is a learned behavior we still verify.”

failure patterns

  • Saying the model searches its training data
  • Equating context length with reliable use of every token
  • Assuming temperature zero proves identical output
  • Treating hallucination as a rare bug removable by one prompt

knowledge checks

Start with these quick checks, then explain the closed-book questions in your own words.

  1. 01Temperature primarily changes…

    select an option to check your answer

  2. 02Why can an LLM hallucinate?

    select an option to check your answer

closed-book recall

Close the worked example and explain these from memory. Answers are self-reviewed. Drafts stay on this page; save anything you want to keep in chapter notes.

Rate the assignment and recall together

Use the assignment rubric as evidence. Completion is your own judgment; this score schedules practice.

interview drill

Give a two-minute whiteboard explanation of an LLM to a backend engineer, then explain one product consequence.

  • 2 min
  • no equations required
  • one failure mode
  • one mitigation
compare a strong approach after your attempt

Define tokens and next-token distributions, show the autoregressive loop, separate training from inference, then connect hallucination to grounding and verification. Avoid claims about consciousness, intent, or literal lookup.

Compare decisions and failure behavior, then repeat with a changed constraint. Keep your own wording.

Use the timed mock workspace →

references

Sources offer deeper context. Provider capabilities and API contracts can change; check current official docs before implementing.

notes

saved locally