bench: the curated graph as the answer key; transcript corpus; Q03 one-query arm; link audit

[aaltshuler]

Summary

Reframes the benchmark around the curated graph as the answer key, and tightens the curation behind it. Stacked on bench-omnigraph-vs-markdown; decisions #41–#44 in bench/SPEC.md.

• Corpus from the source transcripts (#41). bench corpus builds the 337 talk files from transcripts/<label>.txt instead of re-joining the seed's chunks: one paragraph per speaker turn, one sentence per line, same words. The quote check's judge context uses its own ~220-word passages. Corpus and calibration tests need the local transcripts/ and skip without it.
• One-query Q03 (#42). New stored queries contested_claims (every directly contradicted claim with its support count, each contradicting signal's talk and verbatim passage) and pattern_support_talks, with aliases. bench run --og-alias holds the Omnigraph arm to one alias; --runs-dir keeps such runs apart; graph_head() reads the served graph when there is no local one.
• The graph as the answer key (#43, #44). bench key reads keys from the served graph through the reader login (bench/keys/Q0{2,3,4,5}.gq → .json); bench key-score measures how much of a key an arm's answers recover. A cached Opus call only maps answer entries to key entries and sides; the numbers are computed. Q04 needs no judge.
• Curation audit (graph data). Opus-judged audit of the links behind the keys, applied to the served graph on a review branch and merged: 136 of 295 FormsPattern links into pat-harness-over-model dropped; of 40 counters on the most-contradicted patterns, 4 dropped, 10 relabelled, 1 added. seed/10-edges.jsonl re-exported (17,942 → 17,803); README counts updated. Chunk parts unchanged.

Results (markdown arm, 3 clean runs per question, vs the graph's key)

Question	Markdown recovered	Time / cost per run	Graph
Q05 top companies	93% of key companies	223 s / $0.48	0.29 s
Q02 unsolved problems	33% of key problems	122 s / $0.27	0.27 s
Q03 contested claims	22% of claims, 0% of pushback talks	232 s / $0.65	0.66 s
Q04 harness > model talks	7% recall (11% core), 58–92% precision	167 s / $0.54	0.18 s

Already live

The two new queries (with the branch's query descriptions and talk_semantic fix) are deployed (cluster revision 3), and the curated links are on the served graph's main (head 01M3A3P1QWSDXJFFQDTP7026M2). This PR brings the repo in line with what's served.

For review

• The curation changed a lot of harness support; the audit's verdicts and reasons are in the maintainer's local seed-work/curation-q3/audit.jsonl. Two spots to check: ia-aie-chandegra-beyond-harness is a contradiction in the graph but every markdown run called it a supporter; three talks (Osmani, Kalandadze, Rallabandi) lost their only harness link.
• Q02's key has 4 entries (the graph's 4 challenge patterns); Verification Gap's 184 supporting talks were not audited.

Test plan

• uv run pytest: 376 passed
• ruff check clean; every queries/*.gq lints against schema.pg; cluster validate passes
• Served graph verified after the merge: talk and evidence links unchanged (1,512 / 2,619), harness support 159

🤖 Generated with Claude Code