LLM evaluation and held-out cases
Development and evaluation
A case used to change a prompt is no longer independent evidence of how it generalizes. Keep development and held-out case identities disjoint. For a fictional policy assistant, reserve a case with a source revision absent from the development examples; check the split by identity before scoring any result.
Freeze the cases and versions used in a comparison. Report missing outputs and newly failed cases rather than quietly dropping them from the denominator. A candidate with a better average still fails release if it breaks a mandatory tenant boundary.
Local evaluation harness
The course grades prepared outputs with explicit Python rules. It teaches split integrity, denominators and release gates, not the measured quality of a live LLM. Continue with the grading rubric and regression comparison.
Product setup changes. The OpenAI Evals documentation, checked November 14, 2026, marks its hosted Evals surface deprecated and read-only, with closure scheduled for November 30. This lesson does not ask you to create a new hosted Evals job. Use current vendor documentation when moving the method into a live environment.