P0: OTel 事件信封 + 裁判双向盲评与锚定 rubric - #17
Merged
Merged
Conversation
Every event now carries, in addition to the legacy fields (matchId, seq,
ts, type — unchanged), an additive OTel-style envelope:
- event_id (UUID via crypto.randomUUID)
- schema_version ("2.0")
- trace_id (= match id, spans the whole match)
- span_id / parent_span_id (match_start is the trace root; other events
default to its children unless an explicit parent is passed)
- actor (regime id / judge / skill-learner / system)
- kind (fine-grained: llm_call|tool_call|judge_score|skill_propose|
skill_commit|turn|match_start|match_end; coexists with legacy type)
- payload_hash (sha256[:16] of the type-specific payload)
- optional pass-through: model, model_version, prompt_hash, tokens, cost
schemas/match-event.schema.json documents all new fields as optional and
updates examples; frontend MatchEvent type gains the same optional fields
(no consumer changes needed — parsing is additive-safe).
- Anchored rubric: the judge prompt now asks for scores on the three
existing dimensions (legality, feasibility, resilience) using an
anchored 4-point scale (1-4 with per-level anchor descriptions) and
requires a structured JSON answer. A markdown-table fallback parser
keeps legacy judge outputs working.
- Order swap: each matchup is judged twice (default on,
CIVAGENT_JUDGE_SWAP=0 disables) — original order and reversed — and
per-regime scores are averaged across passes to cancel presentation
bias. For 2-civ duels this is exactly A/B → B/A.
- Audit trail: a tournament-level event stream (trace_id = tournament
id) records one judge_score event per pass with provider/model,
rubric prompt_hash (sha256[:16]), swapped flag, and presentation
order, as child spans of a judge span under the trace root.
- Manifest gains judge.swap / judge.passes / judge.rubric / judge.events
and per-regime rubric dims alongside the existing {regime, score}
entries (scores normalized to /10 for leaderboard continuity).
- judge_chain fallback (codex → opencode-reviewer → cn-glm) unchanged.
Tests: +test/events-envelope.test.mjs (envelope presence, UUID/span
formats, trace topology), +test/judge-swap.test.mjs (swap calls judge
twice with reversed order, audit flags, mean aggregation, markdown
fallback, failure path), integration test extended with envelope and
judge-swap audit assertions.
LeoLin990405
added a commit
that referenced
this pull request
Jul 22, 2026
The swap-judge rubric (PR #17) normalizes 1-4 dimension means to /10, so the smallest score grain is ~0.83; with a 0.5 threshold almost nothing would ever count as a tie. 0.8 aligns the tie band with the actual measurement resolution. --tie remains available to override. Tests assert the new default and the decisive side of the boundary.
Closed
3 tasks
LeoLin990405
added a commit
that referenced
this pull request
Jul 22, 2026
LeoLin990405
added a commit
that referenced
this pull request
Jul 22, 2026
* feat(stats): cross-tournament Bradley-Terry ranking + bootstrap CI (P2)
Academic-grade evaluation pipeline over tournament manifests
(judge.scores = [{regime, score}]; extra per-score fields tolerated):
- engine/v5/stats.mjs (zero-dep):
- collectManifests: scan a tournaments dir for */manifest.json
- extractComparisons: per-tournament pairwise win records; |Δ| below
the tie threshold (default 0.5, strict <, configurable) counts as a
half-win tie
- fitBT: Bradley-Terry abilities via MM iteration, geometric-mean
normalized; a phantom-opponent prior (gamma=1, one fictitious game,
half won) keeps abilities finite under perfect win/loss separation
- analyze: bootstrap over tournaments (resample with replacement,
mulberry32 seeded for reproducibility) → 95% ability CIs, rank
intervals, median rank; pairwise significance from bootstrap log-
ability differences (CI containing 0 = not separable) plus
head-to-head records; explicit 样本不足 warning below 5 tournaments
- formatTable human-readable ranking table
- CLI: 'civagent stats [--dir D] [--boot N] [--tie X] [--json] [dirs...]'
(default dir ~/.civagent/tournaments; positional tournament dirs or
manifest paths also accepted)
- test/stats.test.mjs (14 cases): BT ordering recovery on synthetic
strong/mid/weak data, tie thresholds and boundary, equal abilities for
tied regimes, significant vs alternating-rival pairs, small-sample
warning, empty-input handling, PRNG determinism, CLI smoke incl.
bin/civagent wiring.
* feat(stats): default tie threshold 0.5 → 0.8
The swap-judge rubric (PR #17) normalizes 1-4 dimension means to /10, so
the smallest score grain is ~0.83; with a 0.5 threshold almost nothing
would ever count as a tie. 0.8 aligns the tie band with the actual
measurement resolution. --tie remains available to override. Tests
assert the new default and the decisive side of the boundary.
LeoLin990405
added a commit
that referenced
this pull request
Jul 23, 2026
试批数据暴露的三个数据卫生问题： 1. DEFAULT_TIE_THRESHOLD 0.8 → 0.4：双评取均值后实际分数粒度 ~0.4， 0.8 阈值下 tieRate 70.6%，BT 区分度崩塌。注释更新理由。 2. swap 时代过滤：#17 后 manifest 含 judge.swap 字段，量表与旧数据 不同。analyze() 默认只统计 judge.swap === true 的锦标赛，跳过数 通过 skippedLegacy + 警告 'skipped N legacy tournaments' 报告； CLI 新增 --include-legacy 恢复旧行为。splitSwapEra() 导出。 3. regime id 别名：内置 ALIASES = {global/athenian: global/athens}， extractComparisons 收集时归一；~/.civagent/aliases.json 用户扩展 合并（损坏文件容错）。hillclimb.mjs 的 scores/civs/skill 事件 三处 regime 名同样归一（复用 loadAliases/normalizeRegime）。 测试：阈值边界（0.3 tie / 0.5 decisive @ 默认 0.4）、splitSwapEra 三类 manifest、analyze 跳过数与 includeLegacy、内置别名合并与 未知 id 不受影响、用户 aliases.json 扩展/覆盖/损坏容错、别名合并 后 BT 单一条目 games 累计。hillclimb 测试不受影响（夹具已带 swap）。 真实数据验证：13 场 swap 时代、5 场 legacy 正确跳过并警告， athenian 不再单独出现。
LeoLin990405
pushed a commit
that referenced
this pull request
Aug 6, 2026
- wanikua#37: Replace static AUTH_TOKEN with dynamic getAuthToken() to fix 401 after login - #29: Change silijian agent id from 'main' to 'silijian' to avoid conflicts - #19: Remove deprecated fields (runTimeoutSeconds, maxConcurrent), fix api field, add troubleshooting - #20: Add --with-gui / --no-gui flag for optional Dashboard installation - #31: Add comments recommending openclaw CLI over direct config editing - #26: Add docs/windows-wsl.md for WSL2 setup guide - #25/#17: Add hanlinyuan (翰林院) agent as optional configuration
LeoLin990405
added a commit
that referenced
this pull request
Aug 6, 2026
* feat(events): add OTel-style envelope to match event stream (schema v2)
Every event now carries, in addition to the legacy fields (matchId, seq,
ts, type — unchanged), an additive OTel-style envelope:
- event_id (UUID via crypto.randomUUID)
- schema_version ("2.0")
- trace_id (= match id, spans the whole match)
- span_id / parent_span_id (match_start is the trace root; other events
default to its children unless an explicit parent is passed)
- actor (regime id / judge / skill-learner / system)
- kind (fine-grained: llm_call|tool_call|judge_score|skill_propose|
skill_commit|turn|match_start|match_end; coexists with legacy type)
- payload_hash (sha256[:16] of the type-specific payload)
- optional pass-through: model, model_version, prompt_hash, tokens, cost
schemas/match-event.schema.json documents all new fields as optional and
updates examples; frontend MatchEvent type gains the same optional fields
(no consumer changes needed — parsing is additive-safe).
* feat(judge): blind double evaluation with order swap + anchored rubric
- Anchored rubric: the judge prompt now asks for scores on the three
existing dimensions (legality, feasibility, resilience) using an
anchored 4-point scale (1-4 with per-level anchor descriptions) and
requires a structured JSON answer. A markdown-table fallback parser
keeps legacy judge outputs working.
- Order swap: each matchup is judged twice (default on,
CIVAGENT_JUDGE_SWAP=0 disables) — original order and reversed — and
per-regime scores are averaged across passes to cancel presentation
bias. For 2-civ duels this is exactly A/B → B/A.
- Audit trail: a tournament-level event stream (trace_id = tournament
id) records one judge_score event per pass with provider/model,
rubric prompt_hash (sha256[:16]), swapped flag, and presentation
order, as child spans of a judge span under the trace root.
- Manifest gains judge.swap / judge.passes / judge.rubric / judge.events
and per-regime rubric dims alongside the existing {regime, score}
entries (scores normalized to /10 for leaderboard continuity).
- judge_chain fallback (codex → opencode-reviewer → cn-glm) unchanged.
Tests: +test/events-envelope.test.mjs (envelope presence, UUID/span
formats, trace topology), +test/judge-swap.test.mjs (swap calls judge
twice with reversed order, audit flags, mean aggregation, markdown
fallback, failure path), integration test extended with envelope and
judge-swap audit assertions.
LeoLin990405
added a commit
that referenced
this pull request
Aug 6, 2026
* feat(stats): cross-tournament Bradley-Terry ranking + bootstrap CI (P2)
Academic-grade evaluation pipeline over tournament manifests
(judge.scores = [{regime, score}]; extra per-score fields tolerated):
- engine/v5/stats.mjs (zero-dep):
- collectManifests: scan a tournaments dir for */manifest.json
- extractComparisons: per-tournament pairwise win records; |Δ| below
the tie threshold (default 0.5, strict <, configurable) counts as a
half-win tie
- fitBT: Bradley-Terry abilities via MM iteration, geometric-mean
normalized; a phantom-opponent prior (gamma=1, one fictitious game,
half won) keeps abilities finite under perfect win/loss separation
- analyze: bootstrap over tournaments (resample with replacement,
mulberry32 seeded for reproducibility) → 95% ability CIs, rank
intervals, median rank; pairwise significance from bootstrap log-
ability differences (CI containing 0 = not separable) plus
head-to-head records; explicit 样本不足 warning below 5 tournaments
- formatTable human-readable ranking table
- CLI: 'civagent stats [--dir D] [--boot N] [--tie X] [--json] [dirs...]'
(default dir ~/.civagent/tournaments; positional tournament dirs or
manifest paths also accepted)
- test/stats.test.mjs (14 cases): BT ordering recovery on synthetic
strong/mid/weak data, tie thresholds and boundary, equal abilities for
tied regimes, significant vs alternating-rival pairs, small-sample
warning, empty-input handling, PRNG determinism, CLI smoke incl.
bin/civagent wiring.
* feat(stats): default tie threshold 0.5 → 0.8
The swap-judge rubric (PR #17) normalizes 1-4 dimension means to /10, so
the smallest score grain is ~0.83; with a 0.5 threshold almost nothing
would ever count as a tie. 0.8 aligns the tie band with the actual
measurement resolution. --tie remains available to override. Tests
assert the new default and the decisive side of the boundary.
LeoLin990405
added a commit
that referenced
this pull request
Aug 6, 2026
试批数据暴露的三个数据卫生问题： 1. DEFAULT_TIE_THRESHOLD 0.8 → 0.4：双评取均值后实际分数粒度 ~0.4， 0.8 阈值下 tieRate 70.6%，BT 区分度崩塌。注释更新理由。 2. swap 时代过滤：#17 后 manifest 含 judge.swap 字段，量表与旧数据 不同。analyze() 默认只统计 judge.swap === true 的锦标赛，跳过数 通过 skippedLegacy + 警告 'skipped N legacy tournaments' 报告； CLI 新增 --include-legacy 恢复旧行为。splitSwapEra() 导出。 3. regime id 别名：内置 ALIASES = {global/athenian: global/athens}， extractComparisons 收集时归一；~/.civagent/aliases.json 用户扩展 合并（损坏文件容错）。hillclimb.mjs 的 scores/civs/skill 事件 三处 regime 名同样归一（复用 loadAliases/normalizeRegime）。 测试：阈值边界（0.3 tie / 0.5 decisive @ 默认 0.4）、splitSwapEra 三类 manifest、analyze 跳过数与 includeLegacy、内置别名合并与 未知 id 不受影响、用户 aliases.json 扩展/覆盖/损坏容错、别名合并 后 BT 单一条目 games 累计。hillclimb 测试不受影响（夹具已带 swap）。 真实数据验证：13 场 swap 时代、5 场 legacy 正确跳过并警告， athenian 不再单独出现。
LeoLin990405
added a commit
that referenced
this pull request
Aug 7, 2026
试批数据暴露的三个数据卫生问题： 1. DEFAULT_TIE_THRESHOLD 0.8 → 0.4：双评取均值后实际分数粒度 ~0.4， 0.8 阈值下 tieRate 70.6%，BT 区分度崩塌。注释更新理由。 2. swap 时代过滤：#17 后 manifest 含 judge.swap 字段，量表与旧数据 不同。analyze() 默认只统计 judge.swap === true 的锦标赛，跳过数 通过 skippedLegacy + 警告 'skipped N legacy tournaments' 报告； CLI 新增 --include-legacy 恢复旧行为。splitSwapEra() 导出。 3. regime id 别名：内置 ALIASES = {global/athenian: global/athens}， extractComparisons 收集时归一；~/.civagent/aliases.json 用户扩展 合并（损坏文件容错）。hillclimb.mjs 的 scores/civs/skill 事件 三处 regime 名同样归一（复用 loadAliases/normalizeRegime）。 测试：阈值边界（0.3 tie / 0.5 decisive @ 默认 0.4）、splitSwapEra 三类 manifest、analyze 跳过数与 includeLegacy、内置别名合并与 未知 id 不受影响、用户 aliases.json 扩展/覆盖/损坏容错、别名合并 后 BT 单一条目 games 累计。hillclimb 测试不受影响（夹具已带 swap）。 真实数据验证：13 场 swap 时代、5 场 legacy 正确跳过并警告， athenian 不再单独出现。
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
P0: OTel 事件信封 + 裁判双向盲评与锚定 rubric
落实迭代路线图的 P0 两项（调研结论：事件流可观测化 + 裁判双向盲评）。
改动
1. 事件流 OTel 风格信封（schema v2）
每个事件在保留 legacy 字段（
matchId/seq/ts/type）的基础上，纯增量追加：event_id（UUID）、schema_version="2.0"、trace_id（=matchId）span_id/parent_span_id：match_start为根 span，其余事件默认挂根，可显式覆盖actor（judge / skill-learner / system / regime）、kind（tool_call|judge_score|skill_commit|...）、payload_hash（sha256[:16]）model/model_version/prompt_hash/tokens/cost可选透传同步更新
schemas/match-event.schema.json（新字段全部 optional，向后兼容）与前端类型。2. 裁判双向盲评 + 锚定 rubric
CIVAGENT_JUDGE_SWAP=0可关闭judge_score事件，带provider/model、prompt_hash、swapped、order、parent_span_idjudge段新增swap/passes/rubric/events；## Verdict段保留（前端依赖）judge.mjs零改动测试
npm run lint:syntax通过待决策（不阻塞合并）
schema_version取值："2.0"vs"2.0.0"JUDGE_RUBRIC_PROMPT）matches/<tournamentId>/events.jsonl，可能被前端当作对局列出（有tournament:true标记可过滤）