Repository navigation
feat(advisor-board): --reply-format json, a verified machine-consumable reply per seat - #1443
ViperJuice wants to merge 3 commits into
Conversation
…le reply per seat advisor-board --reply-format json asks every seat for ONE JSON object (verdict, summary, findings[]) then the usual last line, and verifies it with the strict panel_reply verifier that shipped in 0.7.25 with no caller. A verified reply is attached to the seat (PanelLegResult.panel_reply) and appears under reply in --json and in the streamed per-seat verdict file; one that does not verify is DEGRADED with a typed panel_reply_<kind> detail and its text kept, so the governed classifier holds it like any non-conforming review. The JSON is followed by the legacy last line on purpose: every seat transport decides a reply is complete from its last line, so no transport changes, and the verifier requires the line to agree with the JSON. The format reaches the seats through the brief, not the prompt: the brief is bound by the HARDEN instruction digest and the sealed prompt envelope and is resolved at about ten call sites, so the CLI extends the review brief itself, stages it as one file and passes it as brief_ref, with the digest minted from exactly those bytes (the advisory contract's mechanism). --advisory is refused with it (an advisory review is recognised by the exact digest of its contract) and so is agy canary capture. invoke_board(reply_format=...) verifies each seat before any on_leg_complete callback sees it and again after native fills are bound (both binding sites). The default is byte for byte unchanged and _default_spawn's signature is untouched. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c7990383b4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| artifact_text = artifact_path.read_text(encoding="utf-8") | ||
| payload = native_fill_request_payload( | ||
| board, artifact_text, **({"brief_ref": advisory_brief_ref} if advisory else {}), | ||
| board, artifact_text, **({"brief_ref": advisory_brief_ref} if advisory_brief_ref is not None else {}), |
There was a problem hiding this comment.
Preserve reply format in native-fill resume output
When --emit-native-request --reply-format json is used, this binds the request to the extended structured-reply brief, but the early-return JSON record does not include reply_format and the printed re-run command omits --reply-format json. Following that generated command reconstructs the ordinary brief, so preflight_native_leg_fills rejects the completed fill with a brief-digest mismatch before any seat launches. Include the reply format in both the record and suggested resume command.
Useful? React with 👍 / 👎.
Full suite vs
|
| passed | failed | errors | distinct failing/erroring tests | |
|---|---|---|---|---|
| this branch | 12,145 | 72 | 320 | 387 |
main |
12,103 | 75 | 320 | 390 |
- Failing on the branch but not on
main: none. - Failing on
mainbut not on the branch: 3 (test_contract_floor_guard_378::test_real_dist_owns_imported_module,test_sandbox_placement::test_local_equivalence_matches_the_base_golden,test_seat_executor_review::test_convergence_review_waits_for_completion_without_a_model_deadline). This change touches none of them and I do not know why they differ; I read it as run-to-run variance in the scratch environment, not as an effect of this PR. - The 320 errors and ~70 failures are the same on both sides. They are environmental to the scratch venv (agy canary and similar suites that need the CI image), not caused by this change; CI's
pytestjobs are the authority. - Passed counts differ by 42: the 38 tests this PR adds, plus the 3 above that pass on the branch, plus one test that was skipped on
mainand ran on the branch (skipped 667 vs 668; I did not identify it).
Correction to the earlier comparison: my first run of this comparison silently ran nothing (pytest-xdist missing from the scratch venv, pytest exited on a usage error, both result files empty, and the diff printed "no differences"). The numbers above are from the rerun, where both runs completed and produced summaries.
🤖 Generated with Claude Code
CHANGELOG.md only: keep both the --reply-format json entry and the grok seat tree entry (agent-harness#1438). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
CHANGELOG.md only: keep the --reply-format json entry alongside the entries main gained since the last merge. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Wires the strict
panel_replyverifier (agent-harness#1286, shipped in 0.7.25 with no caller) intoadvisor-board, so a board can return one machine-consumable, verified reply per seat. Opt-in; the default path is unchanged.What it does
phase-loop advisor-board <artifact> --reply-format jsonasks every seat for one JSON object (verdict,summary,findings[]withseverity,title,body, optionallocation) followed by the usual last line (the verdict word), and verifies the reply.PanelLegResult.panel_reply, a non-field attachment soasdictand the goldens never see it), underreplyin--json, and in the seat's streamedleg-*.verdict.json.DEGRADEDwith a typedpanel_reply_<kind>detail and its text kept, so the governed classifier holds it as it holds any non-conforming review (panel_nonconforming). Kinds:no_json,ambiguous_reply,schema_mismatch,verdict_missing,verdict_forbidden,verdict_inconsistent,terminal_mismatch,recommendation_missing,too_large; all exact literals in the closed detail vocabulary.--jsongains a top-levelreply_formatand a per-seatreplyonly when the flag is used.Design decisions (and the two I changed my mind on)
_completion_okis applied at nine places inside the seat transports (TUI, broker, gemini) to decide a reply is complete, so a JSON-only reply would never read as complete. Keeping the last line means no transport changes; the verifier requires it to agree with the JSON (terminal_mismatch), so a seat cannot satisfy the transport with one verdict and the consumer with another.verify_harden_evidence.py::sealed_prompt_partsvalidates end to end (preamble, digest-bound instruction frame, digest-bound bundle frame), the stagedreview-instructions.mdmust hash to the digest minted from_resolve_brief(mode, brief_ref), and about ten call sites (governed review, runner, train packet, the digest minting) each resolve the brief independently. So the CLI writes review brief + format instructions as one file, passes it asbrief_refeverywhere, and mints the digest from exactly those bytes: the mechanism--advisoryalready uses for its contract._default_spawn's signature and the sealed envelope are untouched.--advisoryis refused with the flag. An advisory review is recognised by the exact sha256 of its contract (is_advisory_brief, "never landing evidence"). Appending the format would change the digest and a non-landing review would stop looking like one. A test pins that the extended contract is no longer recognised, which is why the CLI refuses to build it. Agy canary capture is also refused (it records the exact default brief).invoke_boardit runs inside the per-seat worker, before_run_legs_orderedhands the result toon_leg_completeand the verdict file, so a streaming consumer never sees an unverified reply as OK. Native fills are bound at two sites (the early all-deferred path and the per-seat path); both verify. My first version covered only one of them: the end-to-end native-fill test I wrote caught it (a legacy-format fill stayedOK), and a mutation on each site is now killed separately.object.__setattr__(the_finalize_research_resultprecedent), so attachments the seat already carries survive.Verification
tests/test_panel_reply_board.py, 35; three added totests/test_panel_reply.py), Python 3.10 and 3.14. They cover the flag and its refusals, the derived brief and the digest, every failure kind, a realinvoke_boardrun over the four-seat default board with mixed replies, streaming ordering, the default path unchanged, an advisory-mode board, and native fills on both binding paths._run_seatnot verifying; the early path not verifying fills; the per-seat path not verifying fills; the last-line check dropped; the recommendation check dropped; a failure no longer degrading; the streamed file and the CLI payload droppingreply; the CLI not passing the flag;--advisoryallowed; the digest minted from the base brief; no idempotence marker; always-verify when the flag is off. (My first mutation run silently ran nothing under zsh; I reran it with real counts. Two initially survived: the CLI payloadreplykey and one fill site. Both now have tests.)test_agent_runtime_provider_cs08.py(the_default_spawnsignature guard), the detail-vocabulary tests, the goldens, the advisor-board CLI/advisory/integration suites, both native-fill suites, session labels, placement and login-wait. Python 3.10: 574 passed on the changed-code suites.ruffanddocs-auditclean. Layout identity still805042ce…; jail profile digest still5ccd7e86….main: running now against a cleanorigin/maincheckout; I will post the failing-set comparison here.Not verified
aiis the check, and I would rather run it with you than spend seat quota unasked.--landing-tier, the president sees seat text that now contains JSON; its prompt is unchanged. The extraction of findings from seat prose (agent-harness#1110) is not touched.main's own CI for the agent-harness#1404 merge (d6d02f61) was still running when I opened this, and the run forcd16aa48had one failed job (test_mutation_definitions_are_frozen_but_not_executed_preimplementation, the known local-only flake of agent-harness#1127, here in CI for the first time).Overlap
git merge-treeagainst the open PRs that touch the same files: agent-harness#1432 (cli.py) no conflict; agent-harness#1438 (panel_invoker.py) conflicts only inCHANGELOG.md, which whichever merges second resolves.🤖 Generated with Claude Code
Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.