skip to content

learn.smmeyer.devllm-engineering19

menu

llm-engineering/19

interview mode · core · ~3h practice

AI-native engineering judgment

outcome

Place probabilistic behavior only where it creates value, then measure and bound it.

Plan a practice session · ~3h including exercises
Read + trace the worked example
25 min
Assignment + constraint variations
100 min
Timed drill + repeated explanations
35 min
Closed-book recall + self-review
20 min

This is a suggested practice budget, including drawing, variations, and repeated drills. Reading is the first block. Split the work across sessions and use the rubric to judge when to move on.

why this matters

The senior question is rarely ‘can an LLM do this?’ It is ‘where does semantic flexibility outweigh variability, cost, latency, and security risk—and how will the system fail safely?’

coverage

  • Boundary: deterministic code vs model judgment.
  • Levers: prompting, RAG, tools, fine-tuning.
  • Complexity: workflow vs agent.
  • Autonomy: human approval and bounded effects.
  • Correctness: probabilistic evidence, abstention, verification.
  • Economics: quality/latency/cost measurement.
  • Safety: safe failure and graceful degradation.

mental model

Use the least powerful probabilistic mechanism that clears the measured task bar.

  1. 1 →Prompt

    Known context

  2. 2 →RAG

    External knowledge

  3. 3 →Tune

    Repeated behavior gap

  4. 4 →Agent

    Adaptive actions

Read in numbered order: prompt → rag → tune → agent.

lessons

01Choose the lever that matches the gap

Prompt when the model has context and needs clearer specification. Retrieve when evidence is missing or private. Add tools for exact data/actions. Fine-tune for repeated behavioral/style/task gaps supported by enough data. Add an agent only for adaptive paths.

Localize a failure with counterfactuals. If supplying the correct evidence fixes the answer, retrieval or context is implicated. If the same evidence works with another model, capability may matter. If output is correct but the downstream action fails, inspect orchestration and validation. Change one boundary at a time so the comparison can distinguish hypotheses.

Choose the remedy that addresses the diagnosed cause. Prompting can clarify a task; retrieval can provide current private evidence; a tool can calculate or act; tuning can improve repeated behavior with representative data. A deterministic rule should stay deterministic when its inputs and desired outcome are exact. Combining every technique at once makes cost and failure harder to understand without necessarily improving the result.

  • Do not fine-tune facts that change frequently
  • Do not RAG arithmetic
  • Do not agentize a fixed workflow

02Bound autonomy by consequence

Let the model classify, draft, prioritize, and propose. Deterministic systems validate, authorize, commit, and audit. Increase approval and isolation as reversibility decreases and impact rises.

Assess consequence and reversibility for each action. Drafting a reply, publishing it, and transferring funds require different authority. A useful model may propose the action in all three cases, while the executor imposes different checks and approval. Approval should occur when the effect and parameters are known and should be invalidated by meaningful changes.

Keep autonomy within a testable envelope: allowed resources, maximum spend, steps, deadline, and completion evidence. A broad goal such as “fix production” is not an operational policy. Start with read-only diagnosis or a narrow reversible action, measure outcomes, and expand only when verification and recovery are credible.

  • Read-only differs from money or external communication
  • Make budgets and stop conditions explicit
  • Prefer preview → approve → execute

03Make probabilistic correctness measurable

Define success cases, tolerated errors, abstention policy, and human escalation. Measure end-to-end task completion by slice plus latency, cost, and safety.

Translate “good enough” into a contract with slices and tolerated errors. A helpful drafting assistant may accept some wording variation; an account-access tool must reject unauthorized actions. Include abstention and escalation as legitimate outcomes. Measure successful task completion, harmful effects, latency, and total cost rather than relying on the model's confidence or an aggregate preference score.

Write a decision record for the intervention: evidence, hypothesis, proposed change, strongest alternative, success gate, and rollback. Include a counterexample where another mechanism would win. If the evidence later changes, revise the record. This habit joins technical understanding to product judgment and prevents an attractive architecture from becoming its own justification.

  • Model confidence is not enough
  • Verification can use tools, citations, tests, or humans
  • Design partial success intentionally

worked example

Choose the intervention that addresses the observed failure

An assistant follows the requested output format but answers using a policy superseded yesterday. Its evaluation set has no freshness cases.

  1. Add dated gold-evidence cases and reproduce the stale answer before changing the system.
  2. Inspect ingestion lag, version selection, retrieval, and evidence assembly. A new prompt cannot supply a document that was never indexed.
  3. Consider retrieval or a source-of-truth tool for current facts. Tuning may change behavior but is not an automatic freshness mechanism.
  4. Choose the smallest intervention, compare it on the new cases and existing regressions, and record the limits and switch trigger.

Design decision: Diagnose the layer before selecting prompt, retrieval, tool, tuning, or adaptive planning changes.

Limit: Failures can span layers. A targeted change still needs end-to-end evaluation and monitoring after rollout.

Teaching scenario · traffic, prices, and capacity values are illustrative.

assignment

Produce: Six short diagnosis records, each with evidence, intervention, alternative, and success gate.

Paper, a text editor, or a spreadsheet is enough. Model access and paid services are optional.

  1. Cover stale facts, malformed output, ambiguous intent, unsafe action, excessive latency, and repeated agent loops.
  2. For each, state what evidence would distinguish a context, capability, control, or infrastructure problem.
  3. Choose the least complex useful intervention and argue a counterfactual where another choice would win; rehearse the decisions aloud.

Change a constraint

  • An evaluator change caused the apparent regression.
  • A product rule can be implemented exactly in ordinary code.

Self-review rubric

  • The diagnosis precedes the intervention.
  • The alternative receives a fair comparison.
  • Success is observable in a case set and product metric.

If a criterion is missing, revise that part of the artifact before rating your practice below.

Compare a sample artifact after your attempt

One defensible answer under these assumptions. Compare the reasoning and checks with your work; other designs can satisfy the same contract.

Each diagnosis begins with an observed case. The proposed intervention is conditional on that evidence and must pass a defined comparison.

Failure       distinguishing evidence → intervention / alternative / gate
Stale fact    correct fresh context repairs it → retrieval / tuning / freshness cases
Bad format    correct semantics, parse fails → schema/parser / larger model / valid+correct
Ambiguity     two labels fit policy → clarify / forced tie-break / labeled boundary set
Unsafe action valid proposal, no permission → executor deny / prompt warning / zero effect
Slow request  queue dominates trace → admission/capacity / model swap / useful p95
Loop          repeated calls, no new state → stop/progress gate / larger budget / bounded finish
Record model/prompt/evidence/grader versions before attributing a regression.

A changed evaluator may explain an apparent score drop. If an exact business rule can resolve a case, ordinary code may be the best intervention. Fine-tuning is a candidate for persistent measured behavior gaps with representative data, not an automatic repair for yesterday's policy update. Each record should argue when the alternative would win.

Check your artifact

  • A counterfactual test distinguishes the proposed cause.
  • The success gate measures the user outcome and critical side effects.
  • The intervention's cost and rollback are recorded, including a case favoring the alternative.

A tempting mistake: Adding RAG, tuning, and an agent simultaneously can conceal which defect was repaired and create new ones.

cumulative practice

Harbor · build and evaluate a support assistant

Project brief

Keep one project document and a folder of fictional input/output fixtures for Harbor, a workspace-scoped support assistant. Start with ticket triage and policy-backed drafts at 20 requests/minute, 1,500 input and 200 output tokens/request. Human users remain responsible for consequential actions until you design explicit permissions and approval. Later chapters add retrieval, controlled tools, durable runs, evaluation, security, and scale. Use paper, a text editor, and mocked results; paid APIs are optional. Preserve first attempts, evidence, and revisions within the existing practice budget.

Keep one project document across this track. Reconstruct earlier reasoning before opening notes, then revise the same artifact. Use part of the assignment and review budget for this work. If you enter mid-course, sketch the earlier decisions first using the linked chapters.

Add to your project: Harbor's six diagnosis records and final intervention decision.

Select one failing case from each earlier slice: evidence, format, ambiguity, unsafe action, latency, and looping. Without notes, name a distinguishing experiment and the smallest useful repair. Include a failure caused by a changed grader rather than the application.

Compare cumulative reasoning after your attempt

Gold evidence tests retrieval/context; independent parsing tests structure; labeling review tests ambiguity; executor receipts test effects; phase traces localize latency; state differences expose non-progress. Regrade stored outputs with old and new graders to detect evaluator drift. Choose interventions only after those comparisons, then rerun cumulative and holdout cases with cost/latency gates. Preserve a counterexample and rollback for each decision.

Keep your first attempt and revision together. If your answer differs, identify the assumption or mechanism that explains the difference before rating your recall.

senior signal

For every model decision, ask: What evidence can verify it? What happens if it is wrong? Can the effect be reversed?

failure patterns

  • Using RAG, fine-tuning, and agents simultaneously before measuring baseline
  • Letting the model enforce business invariants
  • No abstention or escalation path
  • Optimizing raw model score instead of successful outcome

knowledge checks

Start with these quick checks, then explain the closed-book questions in your own words.

  1. 01When should you fine-tune?

    select an option to check your answer

  2. 02Where should human approval sit?

    select an option to check your answer

closed-book recall

Close the worked example and explain these from memory. Answers are self-reviewed. Drafts stay on this page; save anything you want to keep in chapter notes.

Rate the assignment and recall together

Use the assignment rubric as evidence. Completion is your own judgment; this score schedules practice.

interview drill

Choose an architecture for automatically triaging, diagnosing, and optionally fixing production incidents.

  • high consequence
  • live systems
  • partial automation
  • audit
compare a strong approach after your attempt

Use deterministic alerts/runbooks for known conditions, a model for synthesis and hypothesis generation, retrieval for current service docs, read-only diagnostic tools by default, and approval-gated bounded remediation. Evaluate diagnosis quality and harmful actions; require rollback, idempotency, budget, and full audit.

Compare decisions and failure behavior, then repeat with a changed constraint. Keep your own wording.

Use the timed mock workspace →

references

Sources offer deeper context. Provider capabilities and API contracts can change; check current official docs before implementing.

notes

saved locally