Repair an unsupported policy answer

A North-workspace employee asks: “How long do I have to submit an expense, and do I need a receipt?” Your assistant returns a confident answer with a citation. Inspect the source trace to decide whether the answer deserves trust.

Treat the supplied excerpts and metadata as the complete fictional corpus. You can inspect all three as the reviewer of this diagnostic fixture; the employee and generation path must not receive unauthorized C. The caller may read North documents only. The product requires the current policy; the known gold evidence is A. You are preparing a supported draft, not executing a reimbursement.

Allow two minutes for the brief, eight for your attempt, five for comparison, and five for revision. No model account is required. Use the supplied trace rather than your own knowledge of expense policies.

A · current North policy

North · version 3 · CURRENT · authorized. “Submit expenses within 30 calendar days of the expense. A receipt is required.”

B · old North policy

North · version 1 · SUPERSEDED · authorized. “Submit expenses within 14 calendar days of the expense. A receipt is required.”

C · South policy

South · version 4 · CURRENT · NOT authorized. “Submit expenses within 60 calendar days. A receipt is optional.”

Observed trace

Ingestion has A/B/C → candidates A/B/C → authorization removes C → reranking retains A/B → context assembly passes only B to generation.

Displayed answer

“You have 60 calendar days and do not need a receipt. [B]” Count its deadline and receipt statements as two policy claims.

Expected source

Gold evidence set = {A}. Final supplied context = {B}. For this case, gold-evidence coverage is |gold ∩ context| / |gold| × 100%.

Produce: An evidence decision, two measurements, a stage-by-stage diagnosis, a corrected cited answer, and a missing-evidence response.

Useful preparation

• Prompting as software engineering →
• Retrieval-augmented generation →