Repository navigation
fix(shepherd): do not charge an attempt when the implementer refuses to act - #369
Conversation
b9ed38d to
ca7ee7a
Compare
|
Claude finished @mashkovd's task in 5m 52s —— View job Code review —
|
| if not path.is_file(): | ||
| return None | ||
| tracked = _run( | ||
| ["git", "ls-files", "--error-unmatch", REFUSAL_MARKER_FILENAME], |
There was a problem hiding this comment.
P2 — the marker has no .git/info/exclude protection, so a single git add -A can permanently disable the fix for that PR.
This tracked-check is the right guard, but it is the only defence, and it fails closed in a way that is silent and permanent. _stage_implementer_agent (L581-585) documents the exact threat model for the other piece of runtime scaffolding it stages:
We register it in
.git/info/exclude(a per-clone, untracked-by-design ignore list) so even a broadgit add -Afrom the sub-agent leaves it untouched.
.implementer-refusal.json is the same category of file — written into the worktree root by the agent, never meant to reach the PR — but it gets no exclude entry. Concrete sequence:
- Tick N: the agent writes the marker and also commits a real fix with
git add -A(the prompt forbids it, but the repo already treats broad staging as realistic). The commit beats the marker,_push_followupruns, and the marker lands onfeat/agents-<slug>. - Tick N+1: a genuine refusal. The fresh clone checks out the branch, the marker is now tracked, this check trips, and the run falls back to
implementer produced no follow-up commits-> 42 ->review_attemptscharged.
From that point on the marker can never be honoured on that branch again, so #360 silently reverts to today's behaviour for exactly the PR that already demonstrated the failure — plus a junk file in a target repo's PR diff.
The fix is one line in _stage_implementer_agent: register REFUSAL_MARKER_FILENAME in .git/info/exclude alongside .claude/agents/implementer.md (loop over a tuple of entries rather than the single entry at L609). That makes the tracked-check pure belt-and-braces instead of the only line of defence, and it keeps the marker out of the PR in the commit-beats-marker case too.
There was a problem hiding this comment.
Fixed in 8097ea4 — agreed, and the framing ("the untracked check is the backstop, not the defence") is the right one.
_stage_implementer_agent now loops over a tuple of entries and registers REFUSAL_MARKER_FILENAME alongside .claude/agents/implementer.md. Because the failure you describe is silent and permanent, the reason is in the code rather than only in the commit message: the docstring now says the exclude entry is the defence and the tracked-check the backstop, and names the consequence (every later refusal on that branch falls back to 42).
Two tests, in tests/test_run_implementer_refusal.py:
test_staging_excludes_the_refusal_marker— both entries present, and re-staging is idempotent.test_excluded_marker_survives_git_add_dash_a— asserts the property against real git rather than the file's text: stage a marker plus a real change, rungit add -A, assert the change is staged, the marker is not, and_read_refusal_markerstill honours it.
Note the backstop still does its job for a marker that became tracked before this change: .git/info/exclude does not affect already-tracked files, so git ls-files --error-unmatch still trips on one.
| `{REFUSAL_MARKER_FILENAME}` in the root of the current working | ||
| directory, with exactly this shape: | ||
|
|
||
| {{"refused": true, "reason": "<what you declined and the evidence for it: |
There was a problem hiding this comment.
P3 — the example the agent is told to copy "exactly" is not valid JSON.
The placeholder reason string wraps across three prompt lines, so the shape shown is a JSON string containing raw newlines — json.loads rejects that. The instruction says "with exactly this shape", which is the one place an agent is most likely to mirror the layout literally, and the failure is the exact one #360 is fixing: _read_refusal_marker warns, returns None, the run falls back to 42, and the attempt is charged.
Putting the example on a single line (and shortening the placeholder, e.g. {"refused": true, "reason": "<what you declined, and the evidence: quote the operator note or the code that already satisfies the finding>"}) removes the ambiguity at no cost.
Related, same section: the marker contract lives only in this outer prompt, while agents/_generic/.claude/agents/implementer.md L69-71 still tells the sub-agent that actually makes the decision to "STOP without committing and explain what's missing in your final message" — with no mention of the marker. That relies on the outer agent noticing the sub-agent declined and writing the file on its behalf. Worth stating the contract in implementer.md too so the decider can record it directly.
There was a problem hiding this comment.
Both fixed in 8097ea4.
The example. You are right that this is the worst place for it — "exactly this shape" invites a literal mirror, and the failure lands on the charged path. It is one line now:
{"refused": true, "reason": "<what you declined, and why>"}
with the guidance about what to put in reason (quote the operator note, or the code that already satisfies the finding) moved to prose beside it, so the sample stays inside the 120-column limit while the instruction keeps its substance. test_prompt_example_is_itself_valid_json pulls the {"refused"...} line out of the rendered prompt and json.loads it, so the example cannot regress into something unparseable.
The sub-agent. Also right, and it was the weaker half of the design: relying on the parent to notice the child declined and write the file on its behalf. All nine agents/*/.claude/agents/implementer.md definitions now carry the contract as the first bullet under "Rules of engagement", stating the marker, the one-line JSON shape, and the consequence of omitting it (the orchestrator cannot tell a considered decision from a crashed run, and charges an attempt). test_subagent_definitions_carry_the_marker_contract asserts every definition mentions the marker filename, so a newly added per-service implementer cannot ship without it.
| reason = data.get("reason") | ||
| if not isinstance(reason, str) or not reason.strip(): | ||
| return None | ||
| return " ".join(reason.split()) |
There was a problem hiding this comment.
P3 — the shepherd-side cap is applied at the use sites, not at this boundary.
MAX_NOTES_CHARS is described as the backstop against agent-authored prose, but this reader returns the string unbounded. Two consumers then diverge: the wait note and the review-stuck note both slice to MAX_NOTES_CHARS, while the review-stuck ShepherdResult.notes at L2037 (f"repeated refusal; proposal not at fault ({reason})") and the info: log line at L2014-2020 do not. In practice the implementer caps at MAX_REFUSAL_REASON_CHARS, so this only bites if the child writes the file some other way — but capping here ([:MAX_NOTES_CHARS]) makes the backstop actually terminal and removes the per-call-site discipline.
There was a problem hiding this comment.
Fixed in 8097ea4 — capped at the boundary, return " ".join(reason.split())[:MAX_NOTES_CHARS].
You named the real problem: a cap applied per call site is one new log line away from being incomplete, and it already was (the review-stuck ShepherdResult.notes and the info: line were unbounded). Capping where agent-authored text enters the shepherd makes the backstop terminal for every consumer instead of a discipline each one has to remember. The docstring now says that is why it lives there. test_read_refusal_reason_caps_at_the_trust_boundary pins it directly.
| status=status, | ||
| review_attempts=int(data.get("review_attempts", 0) or 0), | ||
| harness_failures=int(data.get("harness_failures", 0) or 0), | ||
| refusals=int(data.get("refusals", 0) or 0), |
There was a problem hiding this comment.
P3 — two notes on the new counter's lifecycle; neither blocks merge.
-
No test covers this read.
MAX_REFUSALSonly bounds anything ifrefusalssurvives a tick, and this line is the only thing that carries it back from.status.yamlintoProposalRef.test_refusals_are_bounded_and_flip_to_review_stucksetsref.refusalsin memory, so if this read regressed the counter would reset every tick and the cap would never trip — an unbounded series of paid SDK runs with no terminal state, which is precisely what the counter exists to prevent.harness_failures(L833) has the same gap, so this is parity rather than a new omission; one_discover_refsround-trip test would pin both. -
refusalsis cleared only on a successful follow-up or a reconcile un-stick, so it is effectively lifetime-per-PR rather than consecutive-per-finding-set. If the reviewer re-runs and the findings change (which shepherd never merges an approved PR: re-anchored inline comments are counted as current findings #359 makes likely), refusals recorded against the old bundle still count toward the cap, and a third refusal on genuinely new findings flips a healthy PR toreview-stuck— a milder version of the shepherd charges an attempt when the implementer correctly refuses to act (exit 42 conflates refusal with failure) #360 failure mode. Thereview-stucknote does say the proposal is not at fault, so a human sees the right framing; if the shepherd already tracks a reviewed head SHA, resettingrefusalswhen it moves would make the counter mean what theMAX_REFUSALScomment says it means.
There was a problem hiding this comment.
Both taken, in 8097ea4.
1. The read is now tested. test_discover_refs_round_trips_every_counter writes review_attempts, harness_failures, refusals and refusals_head into a real .status.yaml and asserts _discover_refs carries all four onto the ProposalRef. You are right that this was the load-bearing gap: seeding the ref in memory hides it, and a regression there would silently make every cap non-binding — an unbounded series of paid SDK runs, the exact failure the counters exist to prevent. One test pins both counters, so the harness_failures parity gap closes with it.
2. refusals is now per head SHA, not per PR lifetime. New refusals_head field: a refusal increments only when the recorded head matches the PR's current head, and otherwise starts the count at 1. That makes the counter mean what the MAX_REFUSALS comment claims — "the same bundle produces the same answer" only holds while the branch has not moved — and removes the milder #360 replay you describe, where refusals against a superseded bundle push a healthy PR to review-stuck. Cleared alongside refusals everywhere, including reconcile_one's un-stick path (test_unstick_clears_the_refusal_head_too — a cleared counter with a stale head would resume counting at the cap).
Tests: test_refusals_accumulate_while_the_head_is_unchanged, test_refusals_reset_when_the_branch_moves.
One case deliberately still counts: the reviewer re-running on the same head and producing a different bundle (which #359 makes likely). The head is the only evidence the shepherd has that the code changed, and I would rather the cap be slightly conservative there than key it on a findings hash that #359's re-anchoring churn would make meaningless. At the cap the review-stuck note says explicitly that review_attempts was never charged and the proposal is not at fault, so the human sees the right framing.
There was a problem hiding this comment.
Has P1/P2 findings, changes requested: 0 P1, 1 P2, 3 P3.
P2: .implementer-refusal.json gets no .git/info/exclude entry, unlike the other runtime file staged into the same worktree. If a broad git add -A ever commits it, the marker becomes tracked on that branch and _read_refusal_marker ignores every subsequent refusal — #360 silently reverts for that PR, permanently. One line in _stage_implementer_agent.
Otherwise the design holds up: the marker contract errs toward charging in every ambiguous case, 47 is classified on its own rather than folded into either existing set, the exit code (not the advisory reason file) carries the decision, review_attempts is untouched on the refusal path, and the bound lives on a separate counter that reconcile clears. test_42_43_44_still_charge_an_attempt_end_to_end is the right guard against weakening #12.
Note: I could not run pytest/ruff/mypy in this environment (command approval), so the author's reported 1066 passed is unverified here.
…racked Review round 1 on #369: 1 P2 and 3 P3. P2 — `.implementer-refusal.json` had no `.git/info/exclude` entry, unlike the other runtime file staged into the same worktree. `_read_refusal_marker` honours the marker only while it is untracked, so one broad `git add -A` that swept it into a follow-up commit (a commit beats the marker, so the push proceeds) would land it on `feat/agents-<slug>` — and from the next tick every genuine refusal on that branch falls back to exit 42 and charges an attempt. Silent, permanent, and aimed at exactly the PR that motivated #360. `_stage_implementer_agent` now registers both entries, and says in the docstring why this one is the defence and the untracked check the backstop. P3s: - The prompt's "exactly this shape" example wrapped across three lines, so the shape shown was not valid JSON. It is one line now, with the guidance about what to put in `reason` moved into prose beside it. - The marker contract lived only in the outer prompt, while the sub-agent that actually decides was still told only to "STOP without committing and explain". All nine `implementer.md` definitions now carry it. - `_read_refusal_reason` caps at `MAX_NOTES_CHARS` at the trust boundary instead of leaving each consumer to slice; two of four did not. - `refusals` is now counted per head SHA (`refusals_head`), not per PR lifetime. The bound's premise is "the same bundle produces the same answer", which stops holding once the branch moves; without this, refusals against a superseded bundle could flip a healthy PR to `review-stuck`. Cleared with `refusals` everywhere, including the un-stick path. - `_discover_refs` now has a round-trip test over every counter — that read is the only thing that makes any of the caps bind, and seeding refs in memory hid the gap for `harness_failures` too. Refs #360 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VmJS1T2Lw4SN85pwou8Yj1
|
Round 1 addressed in 8097ea4 — 1 P2 and 3 P3, each answered on its thread with the change.
Audit: can the refusal path reach the counter-less transient arm?Prompted by the
No arm falls through to the generic exit 1. The one path that reaches Still stacked on #367; I will rebase once that branch settles. |
Antigravity (agy) reviewModel: gemini-3.1-pro-high · commit No significant issues found. |
8097ea4 to
c85cf6a
Compare
|
Claude finished @mashkovd's task in 4m 43s —— View job Code review —
|
|
Rebased onto What #367 changed at this seam, and how it was reconciled:
Gates on the rebased tree: All four findings from round 1 (1 P2, 3 P3) were fixed before the rebase and answered on their threads; the P2 exclude-entry fix and the per-head |
Antigravity (agy) reviewModel: gemini-3.1-pro-high · commit No significant issues found. |
| ref.status, | ||
| refusals=new_refusals, | ||
| refusals_head=pr.head_sha, | ||
| notes=note, |
There was a problem hiding this comment.
P2 — the refusal arm does not clear harness_failures, so an alternating 46/47 series reports a platform incident that did not happen.
test_deterministic_failure_also_clears_the_harness_counter (tests/test_run_shepherd.py:3500) codifies the invariant this arm misses:
"Consecutive" has to be true of every proof that the handoff works. A deterministic 42/43/44 proves it as well as a success does: the child ran to a terminal state and the driver adjudicated its output. Without clearing here, the interleaving 46, 42, 46, 42, 46 reaches the cap and reports "the platform lost the work 3 time(s) in a row" — false, and that counter is what an operator reads to decide whether this is a platform incident.
Exit 47 is the strongest proof of a working handoff in the whole set: the child ran, reasoned, wrote a marker, exited on a sentinel, and apply_followup read the reason back out of the real --refusal-out file. Yet harness_failures is cleared on success (L2208), on a deterministic failure (L2185) — and not here.
Concrete sequence, all on one head SHA:
- tick 1 — 46 (orphaned sub-agent):
harness_failures0 -> 1. - tick 2 — 47:
refusals0 -> 1;harness_failuresstays 1. - tick 3 — 46: 1 -> 2.
- tick 4 — 47:
refusals1 -> 2. - tick 5 — 46: 2 -> 3 ==
MAX_HARNESS_FAILURES->review-stuck, notes: "The platform lost the implementer's work 3 time(s) in a row ... This is a harness defect ... see mctl-agents#366."
Three orphans spread across five ticks with two clean terminal runs in between is not a harness defect, and the note points the operator at #366 instead of at the standoff that MAX_REFUSALS was written to surface. The proposal also lands in review-stuck two ticks before MAX_REFUSALS would have got there with the correct note ("review_attempts was never charged — the proposal is not at fault").
Fix is one kwarg on both update_status calls in this arm (L2066-2091), matching L2185: harness_failures=None. Worth extending test_deterministic_failure_also_clears_the_harness_counter to the refusal kind rather than adding a near-duplicate — the invariant is "every terminal, adjudicated child run clears the harness counter".
(The transient arm not clearing it is not the same case: it writes nothing to .status.yaml at all by design, and a transient exit does not prove the driver adjudicated anything.)
There was a problem hiding this comment.
Fixed in aae6b76 — correct, and the sequence you traced is exactly right: three orphans across five ticks with two clean terminal runs in between is not a platform incident, and the note sent the operator to #366 instead of to the standoff.
harness_failures=None on both update_status calls in the arm, and the arm's opening comment now carries the rule rather than leaving it implicit.
The mirror, since you raised the invariant rather than the instance. Checking both directions gave two rules:
- any outcome proving the handoff worked clears
harness_failures— success, deterministic, refused; - any outcome proving the agent engaged with the findings clears
refusals/refusals_head— success, deterministic.
The deterministic arm was the asymmetric one, so it now clears refusals/refusals_head at the same call site where it already cleared harness_failures; the run of refusals before it is no longer consecutive, and that tick is charged to review_attempts, which bounds the sequence from there.
The mirror deliberately stops short in one place: a harness failure clears neither. The child never reached a terminal state, so it proves nothing about the findings — and, decisively, now that 47 clears the harness counter, a 46/47 alternation that also cleared refusals would trip neither cap and run as an unbounded paid loop. That is the one outcome both counters exist to prevent, so the asymmetry is load-bearing and is stated where the code makes the choice.
Pinned by one table, per your steer and the un-stick precedent:
test_counter_contract_across_outcomes— parametrised oversuccess | deterministic | refused | harness, seeding all three counters on disk as well as in memory (the harness arm writes only its own key, so "untouched" is only observable in the file). A new outcome kind is a missing row, not a test nobody writes.test_alternating_harness_and_refusal_reaches_the_refusal_cap— drives 46, 47, 46, 47, 46, 47 and asserts it converges onMAX_REFUSALSwithreview_attemptsstill 0, the note saying "not at fault", and no "harness defect" claim in it.
Both fail without the one kwarg (verified by removing it).
|
|
||
| ## Rules of engagement | ||
|
|
||
| - **Declining to act is a valid outcome — but it must be recorded.** When you |
There was a problem hiding this comment.
P3 — this bullet sits directly above the "blocked, so stop" bullet, and nothing tells the agent the marker does not apply there.
The opening clause is broad ("When you stop without committing on a review follow-up ... ALSO write") and only the parenthetical narrows it. The very next bullet is:
If the proposal is unclear, self-contradicting, or an open question blocks it, STOP without committing and explain what's missing in your final message.
That is the "spec unimplementable" row of the PR description's own table — the one that should charge an attempt. An agent that reads the two bullets together and writes the marker when blocked converts a genuine failure into a non-charged no-op, which is the exact asymmetry the design argues against ("matching a crash narration as a refusal would exempt a genuinely broken proposal from the cap"). The blast radius is bounded — MAX_REFUSALS (3) is tighter than MAX_REVIEW_ATTEMPTS (5), so it still converges — but it converges to a review-stuck note asserting "the proposal is not at fault" when the proposal is precisely what is at fault, which is what a human reads to decide where to look.
One clause closes it, e.g. "This is only for a finding you decided not to act on. If you are blocked — missing context, unclear spec, an unimplementable finding — stop and explain, but do NOT write the marker: that is a failed attempt, not a refusal." Same asymmetry the outer prompt already states; the sub-agent is the one actually making the call.
The outer prompt (_build_prompt step 5) is correctly scoped, so this is only about the nine sub-agent definitions. All nine carry identical text, so it is one edit replicated.
There was a problem hiding this comment.
Fixed in aae6b76 — and you picked the one direction this design must not err in: mistaking a blocked run for a refusal converts the case that must charge the cap into one that does not, and then tells a human "the proposal is not at fault" about the proposal that is precisely at fault. Bounded blast radius is not much comfort when the note points the triage in the wrong direction.
All nine definitions now end the bullet with:
And ONLY for a finding you decided not to act on — if you are BLOCKED (missing context, an unclear or self-contradicting spec, an unimplementable finding), stop and explain as the next bullet says, but do NOT write the marker. That is a failed attempt, not a refusal, and it must still charge the cap: recording it as a deliberate decision would tell a human "the proposal is not at fault" about the one case where the proposal is exactly what is at fault.
It names the next bullet explicitly, since adjacency was what made the two readable as one instruction. test_subagent_definitions_exclude_the_blocked_case asserts every definition carries the exclusion, alongside the existing test that every definition carries the contract at all — so a new per-service implementer cannot ship with half of it.
| cwd=repo_dir, | ||
| check=False, | ||
| ) | ||
| if tracked.returncode == 0: |
There was a problem hiding this comment.
P3 — this is the one check in the function that errs toward "it is a refusal" when it cannot tell.
Every other branch below treats "I could not establish the fact" as "not a refusal": unreadable file, unparseable JSON, refused not exactly True, reason missing or blank. This one inverts it — only returncode == 0 is treated as tracked, so any other failure of the git ls-files invocation (git missing from PATH, a corrupt or absent index, an unexpected 128) reads as "untracked" and the marker is honoured.
In practice the probability is low: _read_refusal_marker only runs inside a fresh _clone_target clone, and a timeout raises ImplementerOperationTimeout before reaching here rather than returning a code. But the docstring right above states the rule as absolute — "Every check below exists to make 'the agent refused' something that cannot be produced by accident" — and this line is the exception to it. Distinguishing 0 (tracked) / 1 (untracked) / anything else (unknown, so do not honour) costs one condition and makes the function match what it says about itself.
There was a problem hiding this comment.
Fixed in aae6b76 — agreed, and the argument that the docstring states the rule as absolute while this line was the exception to it is the one that settles it, independently of how likely the failure is.
Now: 0 is tracked (ignore), 1 is the real "no such path in the index" answer (honour), anything else — git missing from PATH, a corrupt or absent index, an unexpected 128 — is not an answer at all and returns None with a warning. The docstring's second bullet says "UNTRACKED, provably so" and spells out that an invocation which fails to answer the question counts as a no, not a yes.
Two tests: test_unknown_ls_files_result_is_not_treated_as_untracked (stubs exit 128 and asserts the marker is discarded) and test_exit_one_is_still_honoured_as_untracked, so the narrowing cannot quietly break the ordinary case it guards.
There was a problem hiding this comment.
Has P1/P2 findings, changes requested: 0 P1, 1 P2, 2 P3.
P2: the kind == "refused" arm of process_one does not clear harness_failures, unlike the success path (L2208) and the deterministic path (L2185). test_deterministic_failure_also_clears_the_harness_counter already states the invariant this arm misses — exit 47 proves the handoff works at least as well as a 42 does. An alternating 46/47 series on one head reaches MAX_HARNESS_FAILURES and flips to review-stuck reporting "the platform lost the work 3 time(s) in a row", which is false and points the operator at #366 instead of at the standoff MAX_REFUSALS exists to surface. One kwarg on both update_status calls.
P3: the sub-agent implementer.md bullet does not exclude the blocked/unimplementable case that sits in the very next bullet, so a genuine failure could be recorded as a refusal; and _read_refusal_marker treats any non-zero git ls-files result as untracked, the one check in that function that errs toward honouring the marker.
Round 1 was addressed well. The exclude entry plus the tracked-check backstop is the right pairing, the cap now lives at the trust boundary in _read_refusal_reason, the prompt example is one line of valid JSON asserted by json.loads, refusals is anchored to a head SHA, and _discover_refs round-trips every counter — the read that makes any cap bind. test_42_43_44_still_charge_an_attempt_end_to_end is the right guard that #360 does not weaken #12.
Note: pytest/ruff/mypy needed command approval in this environment, so the reported 1077 passed is the author's result, not one reproduced here.
Review round 2 on #369: 1 P2, 2 P3. P2 -- the `kind == "refused"` arm did not clear `harness_failures`, unlike the success and deterministic arms. Exit 47 is the strongest proof in the set that the handoff works: the child ran, reasoned, wrote a marker, exited on a sentinel, and the driver read the reason back out of the real file. Without the clear, an alternating 46, 47, 46, 47, 46 on one head reached MAX_HARNESS_FAILURES and told the operator "the platform lost the work 3 time(s) in a row" -- false, with two clean terminal runs in between -- pointing at #366 instead of at the standoff MAX_REFUSALS exists to surface, two ticks before the refusal cap would have got there with the correct note. The mirror, checked and made symmetric: a deterministic failure now clears `refusals`/`refusals_head` for the same reason it already cleared `harness_failures` -- the child engaged with the findings and the driver adjudicated the result, so any run of refusals before it is no longer consecutive; that attempt is charged to `review_attempts`, which bounds the sequence from there. The mirror stops short in one place, deliberately: a harness failure clears neither, because the child never reached a terminal state and so proved nothing about the findings -- and because, with 47 now clearing the harness counter, a 46/47 alternation that also cleared `refusals` would trip neither cap and run as an unbounded paid loop. Both rules are pinned by one parametrised table over every outcome kind plus the alternating-sequence regression, rather than four near-duplicate tests: a new kind is a missing row, not a test nobody thought to write. P3s: - The sub-agent bullet sat directly above "if the proposal is unclear, STOP and explain" without excluding it, so a blocked run could be recorded as a refusal -- converting the one case that must charge the cap into one that does not, and landing a human on a note asserting "the proposal is not at fault" about a faulty proposal. All nine definitions now exclude it explicitly. - `_read_refusal_marker` treated any non-zero `git ls-files` result as untracked, the one check in the function that erred toward honouring the marker. Exit 1 is the real "not in the index" answer; anything else means the fact could not be established, and now reads as "not a refusal" like every other check there. Refs #360 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VmJS1T2Lw4SN85pwou8Yj1
|
Claude finished @mashkovd's task in 5m 32s —— View job Code review —
|
| sequence | converges on |
|---|---|
47, 47, 47 |
MAX_REFUSALS → review-stuck, "not at fault" |
46, 47, 46, 47, 46, 47 |
MAX_REFUSALS (the regression test), not "harness defect" |
47, 42, 47, 42, … |
MAX_REVIEW_ATTEMPTS (each 42 clears refusals and charges) |
47, success, 47, success, … |
MAX_REVIEW_ATTEMPTS |
46, 46, 46 |
MAX_HARNESS_FAILURES (unchanged) |
COUNTER_CONTRACT as a parametrised table over all four kinds is the right test for this — a fifth kind is a missing row rather than a test nobody wrote — and the seeding-on-disk detail matters, since "untouched" is only observable in the file.
Both P3s are closed as well. _read_refusal_marker now distinguishes 1 (genuinely not in the index) from any other non-zero (fact not established → not a refusal), which makes the function match the absolute rule its docstring states. And all nine implementer.md definitions now exclude the blocked/unimplementable case explicitly — that mattered because the excluded case is the only one that must charge, and recording it as a refusal would land a human on a note asserting "the proposal is not at fault" about a faulty proposal. The exclusion is pinned by test_subagent_definitions_exclude_the_blocked_case, and a blocked run still exits 42 → deterministic → charged, so the design table holds end to end.
P3 (non-blocking)
orchestrator/run_shepherd.py:2098— thereview-stucknote interpolatesreasonlast and slices the whole f-string atMAX_NOTES_CHARS, so ~210 chars of fixed prose eat the budget and a reason the trust boundary admits at 700 is clipped to ~490. It is the terminal note on the one path whose argument is "a human must reconcile this", and the agent's evidence is the part that gets dropped. Budget the reason instead of the sum, or put it first. Thewaitnote is unaffected (28-char prefix).
What holds up
Nothing in the earlier rounds regressed under the rebase. The .git/info/exclude entry as the defence with the untracked check as the backstop, _read_refusal_reason capping at the trust boundary, the one-line JSON example pinned by json.loads over the rendered prompt, refusals anchored to refusals_head, _discover_refs round-tripping every counter (the read that makes any cap bind), 47 classified in its own set rather than widening #367's tuple, the deferred run_implementer import keeping the Temporal worker off the SDK, and test_42_43_44_still_charge_an_attempt_end_to_end guarding that #360 does not weaken #12. Dataclass field insertion is safe — every ProposalRef(...) construction in orchestrator/ and tests/ is keyword-only — and _update_status_if_changed already treats None as delete-if-present, so reconcile_one clearing refusals_head cannot cause per-tick GitOps churn.
Not verified: pytest, ruff, and mypy needed command approval in this environment, so the reported 1092 passed / 8 skipped is the author's result, not one reproduced here.
· branch fix/agents-360-refusal-not-an-attempt
Antigravity (agy) reviewModel: gemini-3.1-pro-high · commit Here is the review of the provided diff. Severity: P2 |
| f"time(s) and the findings still stand. " | ||
| f"review_attempts was never charged — the proposal " | ||
| f"is not at fault; a human must reconcile the " | ||
| f"review with the operator decision. Last reason: " |
There was a problem hiding this comment.
P3 — the reason is last in this note and shares the 700-char budget with ~210 chars of fixed prose, so the one field a human needs can be clipped mid-sentence.
MAX_NOTES_CHARS is 700 and _read_refusal_reason already caps the reason at exactly that, so the slice on this f-string is charged against the sum: the fixed prose ("The implementer declined to act N time(s) … Last reason: ") is ~210 characters, leaving ~490 for a reason that the trust boundary happily admits at 700 (or 600 when the implementer wrote it). Everything beyond that is dropped, and because the reason is interpolated last it is the only part that can be dropped.
This is the terminal note on the one path whose whole argument is "a human must reconcile the review with the operator decision" — the fixed prose is boilerplate they can reconstruct, the agent's evidence ("quote the operator note, or the code that already satisfies the finding") is not. The wait note above is unaffected: its prefix is 28 characters, so a 600-char reason survives whole.
Budgeting the reason rather than the whole string keeps the cap terminal and the evidence intact, e.g. build the prose first and interpolate reason[:MAX_NOTES_CHARS - len(prose)], or put Last reason: {reason} at the front. Not a blocker — the truncation is graceful and only bites long reasons.
There was a problem hiding this comment.
Fixed in 3b8bdd6 — budgeting the reason, not the whole string.
_stuck_note(refusals, reason) builds the fixed prose first and gives the reason MAX_NOTES_CHARS - len(prose), with the docstring stating why this note in particular gets that treatment: its entire argument is "a human must reconcile the review with the operator decision", the boilerplate is reconstructible, and the evidence the agent quoted is not.
Two tests: test_stuck_note_spends_its_budget_on_the_evidence (a 4000-char reason yields a note exactly at the cap that still ends in the reason, so the prose cannot silently eat the budget again) and test_refusal_cap_note_keeps_the_reason end to end through .status.yaml.
Left the wait note as a plain slice, as you noted — a 28-char prefix cannot crowd out a reason the trust boundary already caps.
There was a problem hiding this comment.
No P1/P2 findings (1 P3). Good to merge.
Round 2's P2 is fixed and generalised: the refusal arm clears harness_failures, the deterministic arm now clears refusals/refusals_head, and the one asymmetry that must hold — a harness failure clears neither — is argued in the code and pinned by COUNTER_CONTRACT plus the 46/47 alternation regression. I traced every interleaving of {success, deterministic, harness, refused} on one head: each converges on review_attempts (5), harness_failures (3) or refusals (3), so no sequence reaches the counter-less transient arm. Both P3s are closed too — git ls-files now distinguishes 1 (untracked) from anything else (unknown, not honoured), and all nine implementer.md definitions exclude the blocked/unimplementable case that must still charge.
P3 (non-blocking): the review-stuck refusal note interpolates the reason last and slices the whole string at MAX_NOTES_CHARS, so ~210 chars of fixed prose eat into the budget and a long reason — the one field the human needs to settle the standoff — is clipped. Inline comment has the suggestion.
Note: pytest/ruff/mypy needed command approval in this environment, so the reported 1092 passed, 8 skipped is the author's result, not one reproduced here.
…ng it Round 3 on #369: agy's P2 (which claude[bot] did not raise on the same commit), plus the P3 from claude[bot]'s approving review. P2 -- `_read_refusal_marker` called `path.read_text()` with no size bound. The threat model that matters is not a malicious repository: the file lives in a cloned target repo's worktree and is written by an LLM holding a Bash tool, so a redirect into the wrong path, a loop that appends, or a stray `tee` is enough to make it enormous with no malice at all. `read_text()` pulls all of it into the orchestrator, and an implementer OOM is the least diagnosable failure this platform has: memory pressure on that workload is a live issue and an OOMKill does not appear in the container's `last_terminated_reason`. Now `stat()` first and refuse above MAX_REFUSAL_MARKER_BYTES (64 KiB, generous beside a legitimate marker whose reason is itself capped at 600 chars). Deliberately not a truncating read: a truncated marker would fail the JSON check and reach the charged path for an accidental reason, hiding an operational problem behind a correct outcome. The test makes `read_text` fatal, so a future implementation that reads first and checks afterwards fails rather than quietly restoring the unbounded read. The same bound is applied to the shepherd's `_read_refusal_reason`, which has the same unbounded read on the same class of file. P3 -- the `review-stuck` refusal note sliced the finished string, charging ~210 chars of fixed prose against MAX_NOTES_CHARS and leaving the reason, which is interpolated last, as the only part that could be lost. Backwards for this note in particular: its whole argument is that a human must reconcile the review with the operator decision, and the boilerplate is reconstructible while the agent's evidence is not. `_stuck_note()` now budgets the remaining room for the reason. Refs #360 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VmJS1T2Lw4SN85pwou8Yj1
agy's P2 (unbounded marker read) — fixed in 3b8bdd6Taking it on the reasoning that makes it stand rather than the malicious-repo framing, which I agree is the weakest version of the argument:
Not a truncating read, deliberately: a truncated marker would fail the JSON check and land on the charged path, which is the right outcome for the wrong reason and would hide an operational problem behind a correct-looking one. Refusing outright and warning is honest about what happened. The same unbounded Tests:
Gates on Worth recording for #359's neighbourhood: claude[bot] approved |
…to act
`exit 42` conflated three outcomes: the implementer crashed or the spec is
unimplementable (charge a MAX_REVIEW_ATTEMPTS slot), the findings were already
addressed (nothing to charge), and a principled refusal honouring an explicit
operator decision recorded on the PR (a success). On portfolio#56 the refusal
was correct, burned 2 of 5 attempts, and forced a GitOps reset
(mctl-gitops#1205).
Adds a machine-readable refusal signal end to end:
- The follow-up prompt, and every `implementer.md` sub-agent definition, now
ask the agent to write `.implementer-refusal.json`
({"refused": true, "reason": "..."}) in the worktree root instead of only
explaining itself in prose. A named file with an exact shape is an act the
agent cannot perform by accident; a regex over a final message is one
wording away from reading a crash narration as a refusal.
- `_read_refusal_marker()` honours the marker only when it parses as an object
with `refused` exactly `true`, a non-empty string `reason`, and the file is
untracked. `_stage_implementer_agent` registers it in `.git/info/exclude`
so a broad `git add -A` cannot commit it in the first place: the exclude
entry is the defence, the untracked check the backstop. A committed marker
would otherwise make every later refusal on that branch fall back to 42 --
silently, and permanently.
- New sentinel `EXIT_DELIBERATE_NO_OP` = 47, mapped in
`_review_feedback_exit_code()` in the existing string-prefix style, and
`--refusal-out` carries the reason to the caller as JSON. The reason is
advisory: the exit code alone carries the decision.
- The shepherd classifies 47 via `_refusal_codes()` as its own kind,
"refused": no `review_attempts` increment, no status change, the reason
recorded in `.status.yaml` notes and in the run summary, and the proposal
left at `wait` so a later tick can act on new information.
- The retry is bounded on its own counter (`refusals` / `MAX_REFUSALS`), the
same shape as the harness bound, and counted per head SHA
(`refusals_head`): a refusal moves no head and re-triggers no review, so an
indefinite series on one head would pay for the identical answer every tick,
while a refusal on a moved branch deserves a fresh budget. At the bound it
flips to `review-stuck` with the proposal explicitly marked not at fault.
`reconcile_one`'s un-stick path clears both fields alongside the counters it
already cleared -- un-stuck on one axis only is un-stuck in name only.
42/43/44 still charge an attempt -- asserted end to end so this does not weaken
the #12 fix. `_discover_refs` gains a round-trip assertion over every counter:
that read is the only thing that makes any of the caps bind.
Closes #360
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VmJS1T2Lw4SN85pwou8Yj1
Review round 2 on #369: 1 P2, 2 P3. P2 -- the `kind == "refused"` arm did not clear `harness_failures`, unlike the success and deterministic arms. Exit 47 is the strongest proof in the set that the handoff works: the child ran, reasoned, wrote a marker, exited on a sentinel, and the driver read the reason back out of the real file. Without the clear, an alternating 46, 47, 46, 47, 46 on one head reached MAX_HARNESS_FAILURES and told the operator "the platform lost the work 3 time(s) in a row" -- false, with two clean terminal runs in between -- pointing at #366 instead of at the standoff MAX_REFUSALS exists to surface, two ticks before the refusal cap would have got there with the correct note. The mirror, checked and made symmetric: a deterministic failure now clears `refusals`/`refusals_head` for the same reason it already cleared `harness_failures` -- the child engaged with the findings and the driver adjudicated the result, so any run of refusals before it is no longer consecutive; that attempt is charged to `review_attempts`, which bounds the sequence from there. The mirror stops short in one place, deliberately: a harness failure clears neither, because the child never reached a terminal state and so proved nothing about the findings -- and because, with 47 now clearing the harness counter, a 46/47 alternation that also cleared `refusals` would trip neither cap and run as an unbounded paid loop. Both rules are pinned by one parametrised table over every outcome kind plus the alternating-sequence regression, rather than four near-duplicate tests: a new kind is a missing row, not a test nobody thought to write. P3s: - The sub-agent bullet sat directly above "if the proposal is unclear, STOP and explain" without excluding it, so a blocked run could be recorded as a refusal -- converting the one case that must charge the cap into one that does not, and landing a human on a note asserting "the proposal is not at fault" about a faulty proposal. All nine definitions now exclude it explicitly. - `_read_refusal_marker` treated any non-zero `git ls-files` result as untracked, the one check in the function that erred toward honouring the marker. Exit 1 is the real "not in the index" answer; anything else means the fact could not be established, and now reads as "not a refusal" like every other check there. Refs #360 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VmJS1T2Lw4SN85pwou8Yj1
…ng it Round 3 on #369: agy's P2 (which claude[bot] did not raise on the same commit), plus the P3 from claude[bot]'s approving review. P2 -- `_read_refusal_marker` called `path.read_text()` with no size bound. The threat model that matters is not a malicious repository: the file lives in a cloned target repo's worktree and is written by an LLM holding a Bash tool, so a redirect into the wrong path, a loop that appends, or a stray `tee` is enough to make it enormous with no malice at all. `read_text()` pulls all of it into the orchestrator, and an implementer OOM is the least diagnosable failure this platform has: memory pressure on that workload is a live issue and an OOMKill does not appear in the container's `last_terminated_reason`. Now `stat()` first and refuse above MAX_REFUSAL_MARKER_BYTES (64 KiB, generous beside a legitimate marker whose reason is itself capped at 600 chars). Deliberately not a truncating read: a truncated marker would fail the JSON check and reach the charged path for an accidental reason, hiding an operational problem behind a correct outcome. The test makes `read_text` fatal, so a future implementation that reads first and checks afterwards fails rather than quietly restoring the unbounded read. The same bound is applied to the shepherd's `_read_refusal_reason`, which has the same unbounded read on the same class of file. P3 -- the `review-stuck` refusal note sliced the finished string, charging ~210 chars of fixed prose against MAX_NOTES_CHARS and leaving the reason, which is interpolated last, as the only part that could be lost. Backwards for this note in particular: its whole argument is that a human must reconcile the review with the operator decision, and the boilerplate is reconstructible while the agent's evidence is not. `_stuck_note()` now budgets the remaining room for the reason. Refs #360 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VmJS1T2Lw4SN85pwou8Yj1
| f"at fault; a human must reconcile the review with the operator " | ||
| f"decision. Last reason: " | ||
| ) | ||
| return prose + reason[: max(0, MAX_NOTES_CHARS - len(prose))] |
There was a problem hiding this comment.
P3 — this is a no-op refactor: it produces byte-for-byte the same note the sliced f-string did, so the P3 it answers is still open.
For any prefix p with len(p) <= N, (p + r)[:N] == p + r[:N - len(p)]. That is exactly the transformation here: prose is unchanged (the line re-wrapping does not alter the concatenated text), reason was already interpolated last, and the old slice was at the same MAX_NOTES_CHARS. So both forms yield prose plus the first 700 - len(prose) (~487) characters of the reason.
Which means the docstring's claim is not true of the code: it says it "budgets MAX_NOTES_CHARS for the REASON rather than for the whole string", but the reason's budget is still MAX_NOTES_CHARS - len(prose) — the fixed prose still spends from the same 700. An implementer whose reason is at its own 600-char cap still loses ~113 characters off the end, on the one note whose argument is that the agent's evidence is the part a human cannot reconstruct.
test_stuck_note_spends_its_budget_on_the_evidence does not catch this, and cannot: len(note) == MAX_NOTES_CHARS and note.endswith("E" * 400) were both already true of the sliced version, so the test passes against the code it was written to rule out. Its comment ("not what is left of a budget the boilerplate already spent") describes a property neither form has.
Two ways to actually close it, either fine:
- give the reason the full
MAX_NOTES_CHARSand let the note exceed it (the cap is on agent-authored prose, and the prose here is ours) — then assertnote.count("E") == MAX_NOTES_CHARS; - or put
Last reason: {reason}first and the boilerplate after, so the truncation drops the reconstructible half — then assert the reason survives whole at the implementer's 600-char cap.
Either way the test wants an assertion that fails against a plain (prose + reason)[:MAX_NOTES_CHARS], since that is the implementation being ruled out.
There was a problem hiding this comment.
Fixed in d99d4d1 — budgeting is now real rather than algebraically identical.
You are right on the arithmetic, and the consequence you drew from it matters more than the bug: the test passed against the implementation it was written to forbid, so it was documenting the old behaviour while claiming to rule it out.
Took your first option. MAX_NOTES_CHARS now bounds the agent-authored portion of a note rather than the finished note — our framing is fixed-size and reconstructible, the evidence is not, so charging boilerplate against the same budget spends the wrong thing. The constant's comment says that explicitly, since "bound on a note" was the reading that made the slice look right.
_stuck_note is prose + reason[:MAX_NOTES_CHARS], and test_stuck_note_gives_the_whole_budget_to_the_evidence asserts note.count("E") == MAX_NOTES_CHARS — which is 700 here and 487 under the plain slice, so it fails against the implementation being ruled out. test_a_boundary_capped_reason_survives_whole_in_both_notes pins the realistic case: nothing the trust boundary admits is ever clipped by the framing around it.
Two existing tests moved with the contract rather than being left asserting the old one: test_refusal_note_is_bounded now asserts the agent's text is capped and the note is still bounded (a fixed prefix plus a bounded reason is bounded), and test_refusal_cap_note_keeps_the_reason asserts the reason survives whole instead of merely being present.
Antigravity (agy) reviewModel: gemini-3.1-pro-high · commit Severity: P2 The To fix this, do not rely on with path.open("r", encoding="utf-8") as f:
content = f.read(MAX_REFUSAL_MARKER_BYTES + 1)
if len(content) > MAX_REFUSAL_MARKER_BYTES:
print(f"warn: {REFUSAL_MARKER_FILENAME} is over the cap")
return None
data = json.loads(content)Severity: P3 Similar to Replace with marker.open("r", encoding="utf-8") as f:
content = f.read(MAX_REFUSAL_FILE_BYTES + 1)
if len(content) > MAX_REFUSAL_FILE_BYTES:
print(f"warn: refusal reason file {path} is over the cap")
return None
data = json.loads(content) |
| # `run_implementer` must stay a deferred import here (#149) and this is read on | ||
| # a path that runs before that import; a drifting copy of a generous bound is | ||
| # cheaper than loading the SDK into the Temporal worker. | ||
| MAX_REFUSAL_FILE_BYTES = 64 * 1024 |
There was a problem hiding this comment.
P3 — the stated reason for duplicating the bound is not the actual one, and the real one has a cheaper fix.
The comment says the constant is duplicated because "this is read on a path that runs before that import". It is not: _read_refusal_reason is called at L1788, seven lines after apply_followup already does from orchestrator import run_implementer (L1781) — and _refusal_codes() imports it again on the same call. The genuine reason is narrower and sufficient: this is a module-level constant, so it cannot reference a deferred module at all. Worth correcting, since the comment is the thing a future reader will trust when deciding whether the copy is still needed.
Given that, the duplication is avoidable: read it inside _read_refusal_reason from the module it belongs to (run_implementer.MAX_REFUSAL_MARKER_BYTES, deferred import inside the function, exactly like _refusal_codes), and the two bounds cannot drift. If you prefer the module-level constant for readability, a one-line equality assertion in the tests pins the copy instead — nothing else in the diff would notice a silent drift here, since both values are generous and either one "works".
There was a problem hiding this comment.
Fixed in d99d4d1 — kept the module-level constant, corrected the comment, added the drift test.
You are right that the stated reason was false: _read_refusal_reason is called after apply_followup's deferred import, not before it. The real constraint is the narrower one you name — a module-level constant cannot reference a deferred import at all — and that is what the comment says now, along with why the equality is load-bearing in one direction (this is the outer bound, so a raised implementer-side cap left un-mirrored would honour a marker whose reason the shepherd then refuses to read, losing the evidence for exactly the oversized case the bound exists to surface).
I kept the constant rather than importing inside the function: at the point of use the shepherd is validating a file it did not necessarily write, and a module-level bound reads as a property of this module's input handling rather than as a borrowed value. test_the_duplicated_byte_cap_matches_the_implementers makes the copy fail loudly instead of drifting — the same treatment as COUNTER_CONTRACT and the sub-agent contract tests, per your point that the rule should be in a test and not only in prose.
There was a problem hiding this comment.
No P1/P2 findings (2 P3). Good to merge.
Round 3 closed agy's P2 in the right shape: stat() before read_text() on both sides, refused outright rather than truncated (a truncated marker would fail the JSON check and reach the charged path for an accidental reason), the boundary inclusive so a legitimate marker cannot be clipped, and the stat/read race degrading to "not a refusal". test_oversized_marker_is_refused_without_being_read making read_text fatal pins the property rather than today's implementation.
I traced the refusal path end to end on this commit and found no way for it to reach the counter-less transient arm: every malformed/unreadable/oversized marker returns None -> exit 42 (charged), an unwritable --refusal-out still exits 47, a missing or oversized reason file leaves the kind as refused (the exit code decides), the git ls-files check is check=False so it cannot raise CalledProcessError, and a timeout maps to 44. Counter interleavings on one head each converge on exactly one cap (47xN -> MAX_REFUSALS; 46/47 alternation -> MAX_REFUSALS with no "harness defect" claim; 47/42 and 47/success -> MAX_REVIEW_ATTEMPTS), pinned by COUNTER_CONTRACT plus the alternation regression. _stage_implementer_agent registering the marker in .git/info/exclude (asserted against real git) with the untracked check as backstop, refusals anchored to refusals_head, _discover_refs round-tripping every counter, 47 in its own code set, the deferred run_implementer import, and test_42_43_44_still_charge_an_attempt_end_to_end all hold. No other module or schema enumerates the counter fields, and every ProposalRef(...) construction is keyword-only, so the field insertion is safe.
P3 (non-blocking): _stuck_note is a no-op refactor — prose + reason[:N - len(prose)] is byte-identical to the old (prose + reason)[:N], so the reason still gets ~487 of the 600 chars the implementer can emit, the docstring's claim about the budget is not true of the code, and test_stuck_note_spends_its_budget_on_the_evidence passes against the implementation it was written to rule out. Second P3: the comment justifying the duplicated MAX_REFUSAL_FILE_BYTES states a reason that is not the case (_read_refusal_reason runs after the deferred import, not before). Both inline with suggestions; neither blocks.
Not verified: pytest, ruff, and mypy needed command approval in this environment, so the reported 1107 passed / 8 skipped is the author's result, not one reproduced here.
… escape Round 4 on #369: agy's two P2s (the second raised on the same commit claude[bot] approved), plus four P3s from claude[bot]'s approving review. P2 (a) -- `_read_refusal_marker` caught `(OSError, ValueError)`. Deeply nested JSON raises `RecursionError`, a `RuntimeError`, from a payload that fits inside the 64 KiB cap. The escape does not fail safe: it surfaces as a bare `Exception` with no prefix `_review_feedback_exit_code` recognises, exits 1, and the shepherd classifies 1 as *transient* -- the one arm with no counter at all, so a paid implementer runs every tick forever. Both handlers there, the `--refusal-out` write, and the shepherd's reader are now broad, which is safe precisely because every failure funnels to "not a refusal" -> exit 42, the conservative charged path. Enumerating exception types is the same class of bug as a bound written `value <= 0` that misses `nan`; the test pins the class, not today's four members. P2 (b) -- `stat()` then `read_text()` is a TOCTOU: the size was a fact about a syscall that had already returned, not about the read that followed it, and the marker sits in the agent's own workspace with a Bash tool pointed at it, so a background appender passes the check and then makes the read grow without bound. Replaced by a bounded read of one byte past the cap in both functions -- no race, no `stat()`, and "exactly at the cap" stays distinguishable from "over it" without a second syscall. Read in binary: `read(n)` on a text handle counts characters, which would let a multi-byte payload pull 4n bytes past a cap named in bytes. The old stat/read race test is replaced rather than kept: a test cannot fix a racy pattern, so the new one asserts the function never asks the file for more than `cap + 1` bytes. P3s from claude[bot]: - `_stuck_note` was a no-op refactor -- `prose + reason[:N - len(prose)]` is byte-identical to `(prose + reason)[:N]`, so the reason still lost ~113 of the 600 characters the implementer can emit, and its test passed against the implementation it was written to rule out. `MAX_NOTES_CHARS` now bounds the AGENT-AUTHORED PORTION of a note rather than the note: our framing is fixed-size and reconstructible, the evidence is not. The test asserts `note.count(...) == MAX_NOTES_CHARS`, which fails against the plain slice. - The `wait` note had the same shape; `_declined_note()` puts both under one rule. - The duplicated byte cap's comment stated a reason that was not the case (`_read_refusal_reason` runs after the deferred import, not before). Corrected to the real one -- a module-level constant cannot reference a deferred import -- and the equality is now pinned by a test, since the shepherd's copy is the outer bound and silent drift would lose the evidence for exactly the oversized case the bound exists to surface. - `apply_followup`'s `try` started at the fork, so anything raising between `mkstemp` and the subprocess -- notably the deferred import this repo expects to be absent in some environments -- leaked two /tmp files per tick. The `try` now covers the paths' whole lifetime. Refs #360 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VmJS1T2Lw4SN85pwou8Yj1
|
Claude finished @mashkovd's task in 4m 57s —— View job Code review —
|
Round 4 — agy's two P2s, fixed in
|
| failure | lands on | bounded by |
|---|---|---|
| marker missing / malformed / wrong shape / blank reason | None → "no follow-up commits" → 42 |
MAX_REVIEW_ATTEMPTS |
| marker unreadable, undecodable, nested, or any other exception | broad handler → None → 42 |
MAX_REVIEW_ATTEMPTS |
| marker over the cap | refused before the read completes → 42 | MAX_REVIEW_ATTEMPTS |
git ls-files non-0/1, or unavailable |
None → 42 |
MAX_REVIEW_ATTEMPTS |
git ls-files exceeding its bound |
ImplementerOperationTimeout → 44 |
MAX_REVIEW_ATTEMPTS |
--refusal-out unwritable, or any exception in that write |
warned, still exits 47 | MAX_REFUSALS |
| reason file missing / oversized / unparseable | kind stays refused; the exit code decides |
MAX_REFUSALS |
git ls-files runs with check=False, so it cannot raise CalledProcessError into the generic arm. Two residuals, both pre-existing and neither introduced here: the transient arm itself still has no counter (reachable via review_feedback_one's clone/push failures, which is what it is for), and apply_followup's tempfile calls abort the tick if /tmp is unwritable — the bundle's NamedTemporaryFile has had that exposure since PR #12 and runs first, so the refusal temp file adds no new route.
Gates on d99d4d1: uv run pytest -q → 1151 passed, 8 skipped; ruff check and mypy clean.
Nothing to push back on this round: I checked both findings against the source before taking them, and both are real. The only correction is the crash-vs-catch-all detail above, which does not change either verdict.
Antigravity (agy) reviewModel: gemini-3.1-pro-high · commit Severity: P2 Severity: P3 |
| cmd.extend(["--state-dir", str(state_dir)]) | ||
| print(f"$ {' '.join(cmd)}") | ||
| proc = subprocess.run(cmd, check=False, text=True) # noqa: S603 — cmd is list[str], built above | ||
| refusal_reason = _read_refusal_reason(refusal_path) |
There was a problem hiding this comment.
P2 — this runs on every follow-up, and the empty file mkstemp just created makes it warn on every healthy run.
refusal_path is pre-created empty at L1806-1810 (deliberately: "a child that dies before writing leaves an empty file rather than an ambiguous absence"). This read is then unconditional — it happens before the if proc.returncode != 0 test, so it fires on a successful follow-up, on 42/43/44, on 46, and on every transient exit. In all of those the child never passed through _write_refusal_out, so the file is still zero bytes, and json.loads(b"".decode()) raises JSONDecodeError straight into the broad handler:
warn: could not read refusal reason from /tmp/shepherd-refusal-mctl-web-<slug>-xxxx.json (JSONDecodeError: Expecting value: line 1 column 1 (char 0))
So the dominant path of the shepherd — address-review, the one that worked — now emits a warn: line every tick. CONTRIBUTING.md keeps warn:/error: as the greppable signal for unattended cron/Argo logs precisely so an operator can filter on it; a warning that is true of 100% of healthy runs is the fastest way to make that filter worthless, and it will be the first thing someone sees when they go looking for the real refusal warnings this PR adds.
Two things also do not line up with the comment at L1804-1806. The stated benefit of pre-creating the file — distinguishing "empty" from "absent" — is not realised: both land on the same return None with the same warning, and test_apply_followup_refusal_survives_a_missing_reason_file shows the absent case is already handled. And the file being pre-created is what turns "absent, so nothing to say" into "present but unparseable, so warn".
Gating the read on the code that is the only thing it is for keeps the plumbing identical and the log quiet:
proc = subprocess.run(cmd, check=False, text=True)
refusal_reason = (
_read_refusal_reason(refusal_path)
if proc.returncode in _refusal_codes()
else None
)_refusal_codes() re-imports the already-imported run_implementer, so this costs nothing and stays inside the try/finally. Worth a test too — this block has no coverage of what a non-refusal run prints, which is why a warning on the happy path got through: capsys on an apply_followup that returns 0, asserting "refusal reason" not in out.
There was a problem hiding this comment.
Fixed in 5f9bbfb — the read is gated on proc.returncode in _refusal_codes().
Settled rather than argued: agy found this independently on the same commit, and the reasoning is right. A warn: on the dominant healthy path is worse than the missing prose it was reporting — it destroys the greppable signal this whole feature exists to produce, and a real refusal warning would arrive buried under one from every success.
The part worth keeping is your closing remark, so I turned it into the test rather than a fix for this instance: nothing asserted what a non-refusal run prints. test_a_non_refusal_run_prints_no_warning is parametrised over 0, 42, 43, 44, 46, 1 and 137 and asserts no warn: on any of them — six of the seven fail without the gate. test_the_empty_marker_file_is_only_read_on_a_refusal states the gate directly (the reader is called exactly once, on 47).
I also checked the adjacent question rather than assuming: refusal_path is still created unconditionally, and the reason is now written at the call site. mkstemp is not there for the child's benefit — _write_refusal_out creates the file if missing — but it is the only stdlib call that reserves a unique path for a child process atomically (mktemp is racy and deprecated), and with the read gated the empty file it leaves is inert. The old comment claiming it distinguishes "died before writing" from "absent" is gone, since the gate makes both cases identical.
| f"treating the run as a plain no-commit failure" | ||
| ) | ||
| return None | ||
| tracked = _run( |
There was a problem hiding this comment.
P3 — this _run sits outside every handler in a function whose docstring argues that enumerating exception types is the bug, and the comment below it names a case that is not a return code at all.
L302-307 lists "git missing from PATH" among the non-zero return codes this branch absorbs. It is not one: subprocess.run(["git", ...]) raises FileNotFoundError when the binary is absent, regardless of check=False, so nothing is returned to inspect. And unlike the file read above and the json.loads below, this call is not wrapped, so the exception leaves _read_refusal_marker entirely. It is an OSError, so review_feedback_one's except subprocess.CalledProcessError does not take it either — it lands on the trailing except Exception, becomes error="FileNotFoundError: ...", matches no prefix in _review_feedback_exit_code, and returns EXIT_GENERIC_FAILURE. Exit 1 is the shepherd's transient arm: the one with no counter, retried on every tick.
That is exactly the escape route the docstring above spends a paragraph ruling out for RecursionError ("an escape from here does not fail safe ... the one arm with no counter at all"), and the same route the PR description's audit table asserts is unreachable from this path. Practically it is unreachable — _clone_target has already run git several times in the same process — but the argument the rest of this function makes is that "practically unreachable" is not the standard it holds itself to, and a comment claiming coverage it does not have is what a future reader will trust.
Two ways to close it, either fine: wrap the _run call in the same broad except Exception → warn → return None as the two reads around it (which also makes the branch below true as written), or drop "git missing from PATH" from the comment and say plainly that the enumerated cases are return codes only. The first is better — it makes the whole function honour the rule its docstring states, at no cost, since "could not establish the fact" already means "not a refusal" here.
There was a problem hiding this comment.
Fixed across 5f9bbfb and 05f2747 — both halves, and you were right about both.
The unwrapped call. Now inside a broad handler like everything else in the function. It deliberately swallows ImplementerOperationTimeout too: a slow git ls-files would otherwise escape as a 44, asserting that the implementer timed out, when the honest statement is narrower — we could not establish whether the marker is tracked, so it is not a refusal (42). Both charge an attempt; only one of them is true.
The comment naming a case that is not a return code. Corrected. FileNotFoundError from a missing binary never reaches the returncode branches, so listing it there described a path that does not exist; the comment now says explicitly that the handler above absorbs it, not that branch.
Two tests for exactly the routes the comment was wrong about: test_a_missing_git_binary_is_not_a_refusal and test_a_slow_ls_files_is_not_a_refusal_either. Both would previously have escaped to the counter-less transient arm (the first) or mislabelled the run (the second).
| # Where the implementer writes its reason if this run ends in a deliberate | ||
| # no-op (mctl-agents#360). Created here, not by the child, so a child that | ||
| # dies before writing leaves an empty file rather than an ambiguous absence. | ||
| refusal_fd, refusal_path = tempfile.mkstemp( |
There was a problem hiding this comment.
P3 — the try moved down to cover the deferred import, but bundle_path is still created above it, and this mkstemp is now what sits in the gap.
Round 3's finding was that both temp paths were created before the try; the fix moved the try to L1816, which covers refusal_path but leaves bundle_path (written at L1797-1803) outside it — and inserts a new failure point, this mkstemp, between them. tempfile.mkstemp is not a pure-Python call: a missing or unwritable TMPDIR, ENOSPC, or EMFILE all raise here, and bundle_path is then orphaned. Same failure the round-3 comment was about, on a process that ticks on a schedule; only the leaking file changed.
Closing it is a one-line move — create bundle_path inside the same try as everything else, or start the try immediately after the NamedTemporaryFile block and register refusal_path in the finally list only once it exists:
bundle_path = ... # NamedTemporaryFile, as today
paths = [bundle_path]
try:
refusal_fd, refusal_path = tempfile.mkstemp(...)
os.close(refusal_fd)
paths.append(refusal_path)
...
finally:
for path in paths:
...Low stakes on its own — the failure is loud and it is one small JSON file — but the finally was widened specifically to own these paths' whole lifetime, and it does not yet.
There was a problem hiding this comment.
Fixed in 5f9bbfb — the try now starts above mkstemp.
You are right that round 3's fix only moved the gap rather than closing it: mkstemp is not a pure-Python call, and with the try starting at it, bundle_path was already on disk and the finally was never entered. refusal_path is initialised to None before the try and the cleanup loop skips it, so a failed mkstemp still cleans the bundle.
test_bundle_is_cleaned_up_when_mkstemp_itself_fails forces EMFILE out of mkstemp and asserts the bundle path is gone — the case that was leaking.
There was a problem hiding this comment.
Has P1/P2 findings, changes requested: 0 P1, 1 P2, 2 P3.
P2: apply_followup calls _read_refusal_reason(refusal_path) unconditionally, before the returncode != 0 test, on a file mkstemp pre-created empty. Every non-refusal run — success, 42/43/44, 46, transient — therefore parses zero bytes, raises JSONDecodeError into the broad handler, and prints warn: could not read refusal reason from /tmp/shepherd-refusal-.... A warn: line on 100% of healthy ticks devalues the exact greppable signal CONTRIBUTING.md reserves for unattended logs, and it will be the first thing an operator hits when looking for the real refusal warnings this PR adds. Gate the read on proc.returncode in _refusal_codes(); no test covers what a non-refusal run prints, which is how it got through.
P3: the git ls-files call in _read_refusal_marker is the one step in that function outside a handler — a missing git binary raises FileNotFoundError, escapes to review_feedback_one's generic handler, and exits 1 into the counter-less transient arm, which is the escape the docstring above it spends a paragraph ruling out (and the comment lists "git missing from PATH" as a return code, which it is not). And bundle_path is still created above the widened try, with the new mkstemp now in the gap, so round 3's leak finding is half-closed.
Round 3 is otherwise properly closed. The bounded binary read replaces stat()+read_text() and cannot be raced by an appender; the one extra byte makes the boundary inclusive without a second syscall; over-cap is refused rather than truncated. _stuck_note now genuinely gives the reason the whole budget and lets the fixed prose sit outside it — no longer the no-op slice — and test_refusal_cap_note_keeps_the_reason asserts notes.count("D") == 500, which fails against a plain slice. The duplicated byte cap's comment states the real reason (a module-level constant cannot use a deferred import) and is pinned by an equality test. _declined_note puts the non-terminal note under the same rule. Broad except Exception in both readers is argued from the consequence — an escape lands on exit 1, the one arm with no counter — and both are pinned by a 20k-deep nesting test.
The rest of the design holds end to end: the exclude entry as the defence with the untracked check as the backstop, git ls-files distinguishing 1 from any other non-zero, the exit code rather than the advisory file carrying the decision, refusals anchored to refusals_head, _discover_refs round-tripping all four counters, reconcile_one clearing every one on un-stick, the deferred import keeping the Temporal worker off the SDK, COUNTER_CONTRACT plus the 46/47 alternation pinning the four-kind contract, and test_42_43_44_still_charge_an_attempt_end_to_end guarding that #360 does not weaken #12.
Not verified: pytest, ruff, and mypy needed command approval in this environment, so the reported 1107 passed / 8 skipped is the author's result, not one reproduced here.
Round 5 on #369: one P2 that claude[bot] and agy found independently, plus a P3 from each. P2 -- `apply_followup` called `_read_refusal_reason(refusal_path)` before the returncode test, on a file `mkstemp` pre-creates EMPTY. Every non-refusal run therefore parsed zero bytes and printed `warn: could not read refusal reason from /tmp/shepherd-refusal-...`: a warning on 100% of healthy ticks, which destroys the greppable signal this feature exists to produce and buries a real refusal warning under one from every success. The read is now gated on `proc.returncode in _refusal_codes()` -- the reason is only meaningful when the child exited on a refusal sentinel. The gap that let it through was not the handler, it was coverage: nothing asserted what a healthy run PRINTS. `test_a_non_refusal_run_prints_no_warning` is parametrised over 0/42/43/44/46/1/137 and asserts no `warn:` on any of them, which pins the class rather than this instance, and it fails on six of the seven without the gate. P3 (agy) -- the `try` started AT `mkstemp`, so `mkstemp` raising (out of fds or inodes) left `bundle_path` on disk with the `finally` never entered. The `try` now starts above it and `refusal_path` is None-guarded in the cleanup loop. P3 (claude) -- the `git ls-files` call was the one step in `_read_refusal_marker` still outside a handler. Now broad like the rest, and deliberately swallowing ImplementerOperationTimeout too: a slow `git ls-files` would otherwise reclassify the whole run as a timeout (44) when the honest statement is narrower -- we could not establish whether the marker is tracked, so it is not a refusal (42). Both charge an attempt; only one of them is true. Checked rather than assumed, per the review question: `refusal_path` is still created unconditionally, and the reason is written down at the call. `mkstemp` is not there for the child's benefit -- the implementer's `_write_refusal_out` creates the file if it is missing -- but it is the only stdlib call that reserves a unique path for a child process atomically (`mktemp` is racy and deprecated), and with the read gated, the empty file it leaves is inert. Refs #360 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VmJS1T2Lw4SN85pwou8Yj1
|
Claude finished @mashkovd's task in 8m 25s —— View job Code review —
|
Follow-on to the round-5 push, from the same claude[bot] P3: the comment under the `git ls-files` returncode test listed "git missing from PATH" among the non-zero codes that branch absorbs. It is not one -- `subprocess.run(["git", ...])` raises FileNotFoundError rather than returning, `check=False` or not, so that branch never sees it. The broad handler added around the call in 5f9bbfb is what absorbs it; the comment now says so instead of describing a path that does not exist. Two tests for the routes the comment was wrong about: a missing `git` binary and a timed-out `ls-files` both mean "not a refusal" (42), not exit 1 and not 44. Refs #360 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VmJS1T2Lw4SN85pwou8Yj1
|
Claude finished @mashkovd's task in 8m 46s —— View job Code review —
|
| failure | lands on | bounded by |
|---|---|---|
| marker malformed / unreadable / oversized / wrong shape | None → "no follow-up commits" → 42 |
MAX_REVIEW_ATTEMPTS |
git ls-files non-0/1, missing binary, or timed out |
marker discarded → 42 | MAX_REVIEW_ATTEMPTS |
--refusal-out unwritable, or any exception in that write |
warned; still exits 47 | MAX_REFUSALS |
| reason file missing / empty / unparseable / oversized | kind stays refused — the exit code decides |
MAX_REFUSALS |
Every interleaving on one head converges on exactly one cap: 47×3 → MAX_REFUSALS; 46, 47, … → MAX_REFUSALS with no "harness defect" claim; 47, 42, … and 47, success, … → MAX_REVIEW_ATTEMPTS; 46×3 → MAX_HARNESS_FAILURES. COUNTER_CONTRACT plus the alternation regression pin that, and seeding the counters on disk as well as in memory matters — "untouched" is only observable in the file.
Checked beyond the diff: nothing outside run_shepherd.py and its tests enumerates the counter fields, update_status_file treats None as delete-the-key so every clear is real on disk and mirrored in memory, and the agent-authored reason is whitespace-normalised and capped at the trust boundary before it reaches a YAML note — and never reaches a command line (the only gh pr comment body is a fixed literal).
P3 (non-blocking)
run_implementer.py:2098— the review-feedback summary prints a refusal asfail mctl-web/<slug>: deliberate no-op: …. CONTRIBUTING.md reservesok/fail/skipas the greppable classification for unattended logs, and a refusal is the one outcome in this PR's own table that is explicitly not a failure. Same argument round 5 accepted for thewarn:on healthy ticks, one level up.run_implementer.py:363— the"reason"key is the whole wire format torun_shepherd._read_refusal_reason(L627), and each side is asserted only against its own hand-written literal; neither consumes the other's output. Rename it on either side and both suites stay green while production silently falls back to "no reason recorded by the implementer" — the evidence a human needs to settle the standoff. Same drift classtest_the_duplicated_byte_cap_matches_the_implementersalready pins, on a constant with a smaller consequence.README.md:279— "after five consecutiveaddress-reviewticks" is no longer how a proposal reachesreview-stuck: there are three counters and three doors, and a tick can charge none of them. Parity note, not a new omission — Implementer follow-up ends its session while its own async sub-agent is still running; the lost work is charged as a review attempt #366/fix(implementer): await the sub-agent the CLI launched in the background #367 did not documentharness_failuresthere either.
What holds up
Nothing from the earlier rounds regressed. The .git/info/exclude entry as the defence with the untracked check as the backstop (asserted against real git via git add -A); the bounded binary read that cannot be raced by a background appender, refusing over-cap rather than truncating; git ls-files distinguishing 1 from any other non-zero; _stuck_note/_declined_note giving the reason the whole MAX_NOTES_CHARS with the framing outside it, pinned by assertions that fail against a plain slice; 47 classified in its own set rather than widening #367's tuple; the deferred run_implementer import keeping the Temporal worker off the SDK; refusals anchored to refusals_head and reset when the branch moves; _discover_refs round-tripping all four fields (the read that makes any cap bind); reconcile_one clearing every counter on un-stick; all nine implementer.md definitions carrying the contract and excluding the blocked case that must still charge; and test_42_43_44_still_charge_an_attempt_end_to_end guarding that #360 does not weaken #12.
Not verified: pytest, ruff, and mypy needed command approval in this environment, so the reported 1107 passed / 8 skipped is the author's result, not one I reproduced. I did check line length (≤120) and the new noqa: BLE001 markers by hand.
· branch fix/agents-360-refusal-not-an-attempt
Antigravity (agy) reviewModel: gemini-3.1-pro-high · commit No significant issues found. |
Antigravity (agy) reviewModel: gemini-3.1-pro-high · commit No significant issues found. |
| cwd=repo_dir, | ||
| check=False, | ||
| ) | ||
| except Exception as e: # noqa: BLE001 — see the docstring: broad on purpose |
There was a problem hiding this comment.
P2 — the handler added for round 4's escape route is the one part of this function with no test, and it guards the counter-less arm.
This is the right fix, and the reasoning in the comment is right too. But 5f9bbfb's test changes are entirely in tests/test_run_shepherd.py — nothing exercises this except. The two tests that do touch git ls-files stub return codes through a fake _run that cannot raise:
def fake_run(cmd, *args, **kwargs):
if cmd[:2] == ["git", "ls-files"]:
return subprocess.CompletedProcess(cmd, 128, "", "fatal: not a git repository")so test_unknown_ls_files_result_is_not_treated_as_untracked and test_exit_one_is_still_honoured_as_untracked both pass with or without the try.
Every sibling check in this function is pinned, and the one this mirrors is pinned as a class for exactly this reason — test_any_read_failure_means_not_a_refusal is parametrised over RecursionError/UnicodeDecodeError/MemoryError/RuntimeError with the docstring "no exception type escapes this function", because the narrow-tuple pattern is the bug the docstring argues against. The _run call sat unwrapped through four commits on precisely that argument, and what would catch a future re-narrowing is a test, not the comment above it.
What a regression costs is the reason it is worth one: FileNotFoundError (git absent) is an OSError, so review_feedback_one's except subprocess.CalledProcessError misses it, the trailing except Exception turns it into error="FileNotFoundError: ...", _review_feedback_exit_code matches no prefix, exit 1 is classified transient — the one arm with no counter — and a paid implementer runs every tick. That is the route this PR's own audit table asserts is unreachable from the refusal path, and right now the assertion lives only in prose.
Mirroring the existing test is a few lines:
@pytest.mark.parametrize(
"exc",
[FileNotFoundError(2, "No such file or directory: 'git'"),
run_implementer.ImplementerOperationTimeout("command exceeded 600s: git ls-files"),
RuntimeError("something nobody enumerated")],
ids=["git-missing", "timeout", "unknown"],
)
def test_a_failing_tracked_check_is_not_a_refusal(repo, monkeypatch, exc) -> None:
"""Not establishing the fact must not escape into the counter-less arm."""
_write_marker(repo, {"refused": True, "reason": "r"})
def boom(*_a, **_kw):
raise exc
monkeypatch.setattr(run_implementer, "_run", boom)
assert run_implementer._read_refusal_marker(repo) is NoneThe timeout case is worth including on its own: swallowing ImplementerOperationTimeout here is a deliberate behaviour change (a slow git ls-files now reports 42 instead of 44), and the commit message is currently the only record of that decision.
| # errors continue to exit 1 so the shepherd's transient-failure | ||
| # path is unchanged. | ||
| sys.exit(_review_feedback_exit_code(result.error)) | ||
| code = _review_feedback_exit_code(result.error) |
There was a problem hiding this comment.
P3 — the implementer's own summary line calls a deliberate no-op a fail.
This is the review-feedback summary, so a correct refusal prints:
=== Review-feedback summary ===
fail mctl-web/my-slug: deliberate no-op: Findings 2 and 3 are out of scope by explicit operator decision recorded on the PR
The whole PR argues this outcome is not a failure — the exit code is kept out of deterministic_codes, the shepherd logs info: ... not charging a review attempt, and the terminal note says "the proposal is not at fault". The one place that states the outcome in the implementer's own log says the opposite. It matters more than cosmetics because of where it is read: the shepherd forks this child with inherited stdio, so this line is what lands in the cron/Argo log next to the shepherd's info:, and it is also the only record of the reason when --refusal-out is not passed (which the --refusal-out help text explicitly supports).
code == EXIT_DELIBERATE_NO_OP is already computed a few lines below; branching the label on it (no-op / declined, or reusing the skip shape) keeps the summary honest. The result.error field carrying a non-failure is the underlying awkwardness, but the label is the part an operator reads.
| except Exception as e: # noqa: BLE001 — see the docstring: broad on purpose | ||
| print(f"warn: could not read refusal reason from {path} ({type(e).__name__}: {e})") | ||
| return None | ||
| if not isinstance(data, dict): |
There was a problem hiding this comment.
P3 — these two rejections are the only silent ones in the reader, and they are the cases where the child did write a file.
Round 5 established that what the shepherd prints on each path is part of the contract. This reader now warns on over-cap and on any exception, but returns None with no output when the payload parses and is either not an object or carries a blank/non-string reason. That is the one shape where the child exited 47 and produced a file and the file is unusable — the case most likely to mean a real bug in _write_refusal_out or in something else writing this path, since _write_refusal_out cannot produce it.
What the operator sees instead is the refusal arm's fallback, no reason recorded by the implementer, which reads as "the child didn't bother" rather than "we read its file and rejected it" — and it is not greppable as warn:, so it does not show up in the filter round 5 was protecting. A one-line warning before each return None makes the reader's own log match the standard the rest of the path now holds to. (run_implementer._read_refusal_marker warns on both of the equivalent checks.)
There was a problem hiding this comment.
Has P1/P2 findings, changes requested: 0 P1, 1 P2, 2 P3.
P2 is a coverage gap on correct code, not a defect: the broad except Exception added around the git ls-files call in _read_refusal_marker (5f9bbfb's implementer-side fix) has no test. The commit's test changes are all in tests/test_run_shepherd.py, and the two tests that touch git ls-files stub return codes through a fake _run that cannot raise, so both pass with or without the try. The sibling handler it mirrors is pinned as a class (test_any_read_failure_means_not_a_refusal) precisely because the narrow-tuple pattern is what the docstring argues against — and this call sat unwrapped through four commits on that same argument. If it re-narrows, FileNotFoundError is an OSError, so review_feedback_one's except subprocess.CalledProcessError misses it, exit 1 is classified transient, and the counter-less arm retries a paid run every tick. Swallowing ImplementerOperationTimeout here is also a deliberate behaviour change (a slow git ls-files now reports 42, not 44) recorded only in the commit message. A parametrised test mirroring the existing one closes both.
P3: the review-feedback summary prints fail <service>/<slug>: deliberate no-op: ... for exit 47, in the one log line an operator reads, contradicting every other statement on this path; and _read_refusal_reason's non-dict / blank-reason rejections are the only silent ones in the function, which is the case where the child did write a file and it was unusable — the operator gets no reason recorded by the implementer and no greppable warn:.
Round 4 is properly closed. The read is gated on proc.returncode in _refusal_codes(), and test_a_non_refusal_run_prints_no_warning parametrised over 0/42/43/44/46/1/137 pins the class rather than the instance — the gap was never the handler, it was that nothing asserted what a healthy run prints. test_the_empty_marker_file_is_only_read_on_a_refusal states the gate directly. The try now starts above mkstemp with refusal_path None-guarded in the cleanup loop, and test_bundle_is_cleaned_up_when_mkstemp_itself_fails forces the exact failure.
I re-traced the safety property on this commit and every interleaving on one head SHA still converges on exactly one cap: 47×3 → MAX_REFUSALS; 46, 47, … → MAX_REFUSALS with no "harness defect" claim (47 clears the harness counter, a harness failure clears neither); 47, 42, … and 47, success, … → MAX_REVIEW_ATTEMPTS (both clear refusals/refusals_head and charge); 1, 47, … → MAX_REFUSALS, since the transient arm writes nothing. 45 remains unreachable from this path. Verified independently of the diff: update_status_file has no field allowlist, so the two new keys persist; None means delete-the-key, so every clear is real on disk and mirrored in memory; review_feedback_one is called from exactly one place; _run uses run_capturing, so git ls-files's stderr on the untracked (happy) path is captured rather than leaked as an error: line; and all nine implementer.md definitions carry both the contract and the blocked-case exclusion.
Everything from the earlier rounds holds: the .git/info/exclude entry as the defence with the untracked check as the backstop, the bounded binary read that cannot be raced by an appender, git ls-files distinguishing 1 from any other non-zero, the exit code rather than the advisory file carrying the decision, _stuck_note/_declined_note budgeting the evidence rather than the sum, the duplicated byte cap pinned by an equality test, refusals anchored to refusals_head, _discover_refs round-tripping all four counters, reconcile_one clearing every one on un-stick, the deferred import keeping the Temporal worker off the SDK, and test_42_43_44_still_charge_an_attempt_end_to_end guarding that #360 does not weaken #12.
Not verified: pytest, ruff, and mypy needed command approval in this environment, so the reported 1151 passed / 8 skipped is the author's result, not one I reproduced.
| # errors continue to exit 1 so the shepherd's transient-failure | ||
| # path is unchanged. | ||
| sys.exit(_review_feedback_exit_code(result.error)) | ||
| code = _review_feedback_exit_code(result.error) |
There was a problem hiding this comment.
P3 — the implementer still prints a deliberate no-op as fail, which is the log-level version of the bug this PR fixes.
The summary line two lines above (L2098) runs for every non-None result.error, so a refusal now prints:
fail mctl-web/test-slug: deliberate no-op: Findings 2 and 3 are out of scope by explicit operator decision...
CONTRIBUTING.md makes ok / fail / skip the greppable status prefixes for unattended cron/Argo logs, for the same reason round 5 treated a warn: on 100% of healthy ticks as a P2: the prefix is the operator-facing classification, and a correct outcome labelled with the failure prefix devalues it. A refusal is the one outcome in this PR's table that is explicitly not a failure — the shepherd charges nothing and the design note says the proposal is not at fault — yet the implementer's own log says it failed. An operator grepping fail across implementer runs (or reading the summary to decide whether a tick went wrong) gets the pre-#360 picture back.
One branch before the print, e.g.:
if result.error:
code = _review_feedback_exit_code(result.error)
prefix = "info" if code == EXIT_DELIBERATE_NO_OP else "fail"
print(f" {prefix} {result.ref.service}/{result.ref.slug}: {result.error}")(keeping the exit code exactly as it is — the classification that matters is unchanged). Non-blocking; the shepherd-side framing is already right.
| """ | ||
| try: | ||
| path.write_text( | ||
| json.dumps({"refused": True, "reason": reason}, ensure_ascii=False), |
There was a problem hiding this comment.
P3 — the writer and the reader of this file agree only by coincidence; nothing fails if the key drifts.
The "reason" key is the entire wire format between the two processes, and each side is pinned against its own hand-written literal:
test_main_writes_the_refusal_reason_and_exits_47asserts this writer produces{"refused": True, "reason": reason};- every shepherd-side test (
test_apply_followup_raises_refused_and_carries_the_reason,test_the_empty_marker_file_is_only_read_on_a_refusal, …) hand-writes{"refused": true, "reason": ...}in its ownfake_runand then asserts_read_refusal_reasonreturns it.
Neither side ever consumes the other's output. Rename the key here (or on the reader at run_shepherd.py:627, which also never checks refused — only reason) and both suites stay green while production silently degrades: _read_refusal_reason returns None, the note becomes implementer declined to act: no reason recorded by the implementer, and the one thing a human needs to settle the standoff is gone. Bounded — the exit code still carries the decision, so no attempt is charged — but it is invisible, and the evidence is the whole reason this file exists.
This is the same cross-module drift class the PR already decided to pin rather than document: test_the_duplicated_byte_cap_matches_the_implementers exists for exactly this, on a constant with a smaller consequence. One test feeding the writer's output straight to the reader closes it:
def test_the_reason_survives_the_real_round_trip(tmp_path) -> None:
"""The two processes agree on the wire format, not just on their own literals."""
path = tmp_path / "refusal.json"
run_implementer._write_refusal_out(path, "operator deferred this to a separate issue")
assert run_shepherd._read_refusal_reason(str(path)) == (
"operator deferred this to a separate issue"
)| # Counted per HEAD SHA (`ProposalRef.refusals_head`), not per PR lifetime: the | ||
| # premise is "the same bundle produces the same answer", which stops holding | ||
| # the moment the branch moves. | ||
| MAX_REFUSALS = 3 |
There was a problem hiding this comment.
P3 — README.md:279 now describes the cap in a way that is wrong for three of the five outcomes.
Review-attempt cap. The outer loop tracks
review_attempts:in.status.yaml. After five consecutiveaddress-reviewticks without resolving the findings, the next tick flips the proposal tostatus: review-stuck(terminal) instead of forking the implementer again.
After this PR there are three counters and three review-stuck doors, and an address-review tick can end without charging anything (transient), charging harness_failures (46), or charging refusals (47). So "five consecutive address-review ticks" is no longer the rule for how a proposal reaches review-stuck — and a reader trying to answer the exact question #360 is about ("did that tick cost the PR an attempt?") gets the pre-#366 answer from the only operator-facing doc that covers it.
Parity note, not a new omission: #366/#367 did not document harness_failures there either — grep -rn harness_failures outside run_shepherd.py and its tests finds nothing. That is what makes it worth a line here rather than there: this PR is the second counter added past that sentence, and it is the one that makes the sentence actively misleading about cost. Three sentences naming the three counters (5 / 3 / 3, and which outcomes clear which) would settle it. tools/diagram_facts.py:62 only extracts MAX_REVIEW_ATTEMPTS, so the generated diagram has the same gap.
Non-blocking, and squarely the author's call on whether README carries this level of detail at all — the in-code rationale is already thorough.
There was a problem hiding this comment.
No P1/P2 findings (3 P3). Good to merge.
Round 5 is closed in the right shape. The refusal-reason read is gated on proc.returncode in _refusal_codes(), so the empty mkstemp file no longer produces a warn: on every healthy tick — and test_a_non_refusal_run_prints_no_warning, parametrised over 0/42/43/44/46/1/137, pins the class rather than the instance, which is the right response to a gap that was about coverage rather than the handler. The try now opens above mkstemp with refusal_path None-guarded in the cleanup loop, so the finally owns both paths for their whole lifetime (test_bundle_is_cleaned_up_when_mkstemp_itself_fails). The git ls-files call is inside a broad handler like the rest of _read_refusal_marker, deliberately absorbing ImplementerOperationTimeout too — the honest statement is the narrower one (we could not establish whether the marker is tracked → 42, not "the implementer timed out" → 44) — and the comment no longer claims a missing git binary arrives as a return code, with both routes tested.
I re-traced the path end to end on this commit and found no way for it to reach the counter-less transient arm: every malformed, unreadable, oversized, or unverifiable marker returns None → 42 (charged); an unwritable --refusal-out still exits 47; a missing or unparseable reason file leaves the kind refused, since the exit code decides; git ls-files runs check=False so it cannot raise the one exception that reaches transient. Every interleaving on one head converges on exactly one cap — 47×3 → MAX_REFUSALS; 46, 47, … → MAX_REFUSALS with no "harness defect" claim; 47, 42, … and 47, success, … → MAX_REVIEW_ATTEMPTS; 46×3 → MAX_HARNESS_FAILURES — pinned by COUNTER_CONTRACT and the alternation regression, with the counters seeded on disk as well as in memory because "untouched" is only observable in the file. Also checked beyond the diff: nothing outside run_shepherd.py and its tests enumerates the counter fields, update_status_file treats None as delete-the-key so every clear is real on disk and mirrored in memory, and the agent-authored reason is whitespace-normalised and capped at the trust boundary before it reaches a YAML note (and never reaches a command line — the only gh pr comment body is a fixed literal).
P3s, all non-blocking and inline: the implementer still prints a refusal under the fail prefix CONTRIBUTING reserves for failures, which is the log-level version of the bug this PR fixes; the "reason" wire format between _write_refusal_out and _read_refusal_reason is pinned only by each side asserting its own literal, so a key rename keeps both suites green while production silently loses the evidence; and README.md:279 still describes the cap as "five consecutive address-review ticks" now that three counters and three review-stuck doors exist.
Not verified: pytest, ruff, and mypy needed command approval in this environment, so the reported 1107 passed / 8 skipped is the author's result, not one I reproduced. I did check line length (≤120) and the new noqa: BLE001 markers by hand.
Fixes #360.
The problem
exit 42conflated three outcomes:MAX_REVIEW_ATTEMPTSslot?On
portfolio#56the refusal was correct twice, burned 2 of 5 attempts, pushed a healthyPR into the cap, and forced a GitOps reset (mctl-gitops#1205).
The signal, and why a marker file
The implementer now reports "I deliberately did not commit, and here is why" distinctly
from "I ran and produced nothing". The agent writes
.implementer-refusal.jsonin theworktree root:
{"refused": true, "reason": "Findings 2 and 3 are out of scope by explicit operator decision recorded on the PR; not applying."}Why not scrape the final message. There is a final message — the prompt already asks
the agent to "STOP without committing and explain why" — but the prose is the agent's
free composition, unstable across model versions and prompt edits. Any regex over it has
two failure modes, and they are not symmetric: missing a real refusal costs one attempt,
while matching a crash narration ("I was unable to make any changes…") as a refusal
would exempt a genuinely broken proposal from the cap forever. A named JSON file with an
exact shape is an act the agent cannot perform by accident while doing something else, and
it either exists or it does not.
The marker is honoured only when all of these hold — the asymmetry above is the reason
each check errs toward "not a refusal":
MAX_REFUSAL_MARKER_BYTES, 64 KiB), checked withstat()before the read and refused outright rather than truncated — the file iswritten by an LLM with a Bash tool into a cloned worktree, and an implementer OOM is
the least diagnosable failure this platform has;
refusedexactlytrue(not1, not"true") and anon-empty string
reason;without it, committing
.implementer-refusal.jsoninto a target repo would make everyfuture follow-up on that repo read as a refusal — the repo would be permanently exempt
from the attempt cap, silently and for good.
_stage_implementer_agentregisters themarker in
.git/info/excludeso a broadgit add -Acannot commit it in the firstplace: the exclude entry is the defence, this check the backstop for anything that got
committed anyway (exclude does not affect already-tracked files);
A stale marker cannot survive between runs.
_clone_target()clones into atimestamped tmp directory (
rmtreefirst if the path somehow exists) andreview_feedback_oneuses only that clone, so the marker is always an untrackedworking-tree file created during the current run. The only way one could pre-exist a run
is by being committed — which is exactly what the untracked check rejects.
Anything else logs a warning and falls back to the ordinary "no follow-up commits"
failure, which still charges an attempt.
End to end
_build_prompt, review-feedback branch): step 5 now specifies the marker,its exact shape, and that it is only for a considered no-op — never next to a commit,
never as a progress note, never with a placeholder reason. The new-PR prompt is
unchanged (no attempt budget there).
EXIT_DELIBERATE_NO_OP = 47, mapped in_review_feedback_exit_code()in the existing string-prefix style (
deliberate no-op: …).--refusal-out <path>hands the reason to the caller as JSON; it is advisory — the exit code alone carries
the decision, so a child that dies before writing is still not charged.
_refusal_codes()classifies 47 as its own kind,"refused". It is keptas a separate function rather than widening fix(implementer): await the sub-agent the CLI launched in the background #367's
_followup_code_sets()tuple — arefusal is neither of those two things (not deterministic: the proposal is not at
fault; not a harness failure: nothing was lost), and it keeps the stacked diff off the
function fix(implementer): await the sub-agent the CLI launched in the background #367 is still editing. Same deferred
run_implementerimport, so the Temporalworker still does not pull in the SDK (refactor(temporal): check the Argo execution boundaries the worker reasons about #149).
process_one: noreview_attemptsincrement, no status change, reason recorded in.status.yamlnotes (bounded byMAX_NOTES_CHARS) and in the run summary line,decision
waitso a later tick can act on new information.The retry is bounded, on its own counter
A refusal moves no head SHA and re-triggers no review, so the next tick hands the agent
the identical bundle and pays for the identical answer. Following the same shape #367 just
introduced for harness failures: a separate
refusalscounter withMAX_REFUSALS = 3,counted per head SHA (
refusals_head) — the premise is "the same bundle produces thesame answer", which stops holding the moment the branch moves, so a refusal on a new head
starts the count over rather than inheriting a budget spent on different code.
At the bound the proposal flips to
review-stuckwith a note saying explicitly thatreview_attemptswas never charged and the proposal is not at fault — a standoff betweena reviewer finding and an operator decision is something only a human can settle.
review_attemptsis never touched on this path.The counter contract
review_attemptsharness_failuresrefusals/refusals_headTwo mirrored rules: an outcome proving the handoff worked clears
harness_failures; an outcome proving the agent engaged with the findings clearsrefusals. The mirror stops short in one place on purpose — a harness failure clearsneither, because the child never reached a terminal state, and because with 47 clearing
the harness counter a 46/47 alternation that also cleared
refusalswould trip neithercap and run as an unbounded paid loop. Pinned by one parametrised table
(
test_counter_contract_across_outcomes) plus the alternating-sequence regression,rather than one test per cell.
reconcile_one's review-stuck un-stick path clearsrefusalstoo, by exactly theargument in
b28c823: un-stuck on one axis only is un-stuck in name only. That commit'stest is extended to cover all three counters (and renamed to
test_reconcile_unstick_clears_every_counter) rather than gaining a near-duplicate — theinvariant is "every counter that can flip a proposal to
review-stuckis cleared here",and it is worth failing on as a whole. Verified it fails without the one-line change.
(mctl-agents#359 — inline comments re-anchored onto a new head inflating the finding count
— is the upstream cause of the findings that refusal was declining. Separate fix.)
Tests
New
tests/test_run_implementer_refusal.py(27) plus a block intests/test_run_shepherd.py(14, with two of main's existing tests widened rather thanduplicated):
refusedmissing / false /"true"/1, reason missing / blank / non-string, committed marker ignored,reason normalised and capped;
review_feedback_one: no commits + marker →deliberate no-op: …→ 47; no commitsalone → the old error → 42; a commit beats the marker;
main()writes the reason to--refusal-outon 47 and writes nothing on 42;refusedon its own, reason carried through the real--refusal-outplumbing, refusal without a reason file still not charged, temp filecleaned up, outer loop leaves counter + status untouched and records the reason, note
bounded, counter bounded →
review-stuck;.git/info/excludeentries present andidempotent, and the property asserted against real git — stage a marker plus a real
change,
git add -A, assert the change is staged and the marker is not;json.loadsso it cannot regress into anunparseable shape, and every
implementer.mdsub-agent definition carries the contract;_discover_refsround-trips every counter (the read that makes anycap bind at all), refusals accumulate on one head, reset when the branch moves, and the
un-stick path clears the head alongside the count;
test_42_43_44_still_charge_an_attempt_end_to_end— drives the real classificationin
apply_followupfor each deterministic sentinel and feeds the resulting exception toprocess_one, asserting the attempt is charged. shepherd charges an attempt when the implementer correctly refuses to act (exit 42 conflates refusal with failure) #360 must not weaken [wip] feat(shepherd): tier 3 PR shepherd — tasks 1 + 2 #12's fix.uv run pytest -q→ 1107 passed, 8 skipped.uv run ruff checkanduv run mypyclean.🤖 Generated with Claude Code
https://claude.ai/code/session_01VmJS1T2Lw4SN85pwou8Yj1