The Evaluation Lab
You are here · Protocol 37
Shift briefing · Evaluation Laboratory
A new model version improves the headline score while failing on a high-impact subgroup. Build a repeatable evaluation gate before promoting it.
Read before the shift. It tells you what is at stake, never how to do it.
What you practise
Create a representative test set and measure quality, regressions, and subgroup failures.
- Golden set
- metric selection
- regression testing
- coverage
- slices
Experiment Change one thing, run it again, compare on evidence.
The facility for this shift
3 rooms, 4 stations, two hands.
- 01Case ArchiveCase archive
- 02Metric RoomMetric console
- 03Comparison LabComparison wall · Release gate
Compromised Signal · sealed
Something in this shift will change, and the briefing will not tell you what. You find it from the evidence, or your after-action review shows you where it was.
How the shift is assessed
3 criteria · 0 to 6
Scored from what you did: the evidence you checked, the stations you operated, the artifacts you produced and the gates you passed. The criteria are revealed in your after-action review, not before.
Capabilities evidenced
- ReasonModels dependencies, constraints, causal paths, and trade-offs.In 22 of 48
- ReliabilityTests, reproduces, validates, recovers, and produces consistent outcomes.In 39 of 48
- ResponsibilityProtects privacy, limits permissions, documents decisions, and blocks unsafe releases.In 23 of 48
Chamber III · Action Chamber
Build only what you can stand behind, and stop a release that is not safe.
Construct, integrate, evaluate, debug, secure, deploy, observe and responsibly operate AI-enabled systems.
Play Protocol 37 in first person, with the clock running.