worker/qa/fixtures/library-quality.json contains six original synthetic sources: spaced/retrieval practice, reflective journaling versus a TV watchlist, and clinical prediction versus an AI assistant. A rubric requires key concept coverage, literal evidence, a useful retrieval link, absence of incidental links, consolidation of equivalent learning subjects, and separation of different subjects. This is a small regression set, not a general semantic accuracy claim. Expand it with reviewed examples as failure modes are discovered; change its revision when changing a rubric or source.
The harness uses production processing functions. The default extraction path reviews organization. --extraction=breakdown exercises full Balanced breakdowns instead. Grouping always uses the production Balanced configuration, so choose a model that supports that configuration. The explicit model override affects benchmark Quick/Balanced only, never production settings. Comparing a context change means running the same fixtures on the baseline and candidate source revisions. The report records fixture revision, context sizes, output, latency, observed token usage and the existing Claude estimate; unknown models stay unpriced.
Live runs require ANTHROPIC_API_KEY in the local environment and explicit opt-in. They make at most ten serial paid requests (six extractions, three connection checks, one grouping batch), without hidden SDK retries, and do not use a production user’s allowance ledger. No fallback key is supplied. Never run live in CI. Keep real keys and private-source examples out of the repository.
node worker/qa/quality-benchmark.mjs --live --model=MODEL --label=baseline --output=/tmp/baseline.json
node worker/qa/quality-benchmark.mjs --live --model=MODEL --label=candidate --compare=/tmp/baseline.json --output=/tmp/candidate.json
node worker/qa/quality-benchmark.mjs --responses=/tmp/candidate.json --compare=/tmp/baseline.json --output=/tmp/rescored.json
Use the same extraction mode and fixture revision when comparing. Replay incurs no provider calls. A failed rubric returns exit status 1 while preserving the report. Tests deliberately remove useful concepts, add incidental/fabricated links and overmerge groups to verify detection; they do not simulate model quality or prove prompts work against a live provider.
Before changing defaults: review every regression and proposed relationship, compare key-concept coverage and complete breakdowns, inspect actual input/output tokens and the provider bill, then repeat with representative sources (including long sources, publisher captures and transcripts). Lower estimated cost alone does not justify losing grounded concepts or creating weak map edges. No production switch is automatic.