Summary
manifest_missing_tests (src/signals/focus-manifest.ts:690-701) fires when
manifest.testExpectations.length > 0 && testFileCount === 0 && passedValidationCount === 0. The caller
(src/queue/processors.ts, in the manifestPolicyGateMode block of maybePublishPrPublicSurface)
currently computes passedValidationCount from a pure PR-body regex match (hasValidationNote) with
zero connection to the PR's actual CI/check-run results.
Confirmed false positive
JSONbored/awesome-claude PR #4719 — a fully-automated github-actions[bot]-authored PR that only
regenerates README.md — got a FAILURE conclusion on the "Gittensory Orb Review Agent" check run
("Configured validation evidence missing") even though the real CI checks all passed, because the bot's
templated PR body naturally contains none of the trigger words, and that repo's manifestPolicy: block
config escalates the finding into a hard blocker. There is also no bot/author exemption anywhere in this
code path — review.auto_review.ignore_authors only gates whether the AI review runs, not this
deterministic manifest-policy check.
Fix
- Make
passedValidationCount's computation also consult the PR's real live CI state (reusing the
existing cachedLiveCiAggregate infra) — a fully-green CI rollup now counts as evidence, in addition
to (not instead of) the existing PR-body text match.
- Add a bot-author exemption to the
manifestPolicyGateMode block, mirroring the existing
review.auto_review.ignore_authors glob pattern, so a fully-automated bot PR is not held to
"did you demonstrate test evidence" scrutiny meant for human contributors.
The live symptom was already worked around by softening awesome-claude's manifestPolicy from block
to advisory in its private VPS config, but the underlying detection heuristic is still wrong and would
misfire the same way for any repo that sets manifestPolicy: block (including this repo, or metagraphed),
or for any human PR whose body just doesn't happen to use "tested"/"validated" wording even though CI
genuinely passed.
Summary
manifest_missing_tests(src/signals/focus-manifest.ts:690-701) fires whenmanifest.testExpectations.length > 0 && testFileCount === 0 && passedValidationCount === 0. The caller(src/queue/processors.ts, in the
manifestPolicyGateModeblock ofmaybePublishPrPublicSurface)currently computes
passedValidationCountfrom a pure PR-body regex match (hasValidationNote) withzero connection to the PR's actual CI/check-run results.
Confirmed false positive
JSONbored/awesome-claude PR #4719 — a fully-automated
github-actions[bot]-authored PR that onlyregenerates
README.md— got a FAILURE conclusion on the "Gittensory Orb Review Agent" check run("Configured validation evidence missing") even though the real CI checks all passed, because the bot's
templated PR body naturally contains none of the trigger words, and that repo's
manifestPolicy: blockconfig escalates the finding into a hard blocker. There is also no bot/author exemption anywhere in this
code path —
review.auto_review.ignore_authorsonly gates whether the AI review runs, not thisdeterministic manifest-policy check.
Fix
passedValidationCount's computation also consult the PR's real live CI state (reusing theexisting
cachedLiveCiAggregateinfra) — a fully-green CI rollup now counts as evidence, in additionto (not instead of) the existing PR-body text match.
manifestPolicyGateModeblock, mirroring the existingreview.auto_review.ignore_authorsglob pattern, so a fully-automated bot PR is not held to"did you demonstrate test evidence" scrutiny meant for human contributors.
The live symptom was already worked around by softening awesome-claude's
manifestPolicyfromblockto
advisoryin its private VPS config, but the underlying detection heuristic is still wrong and wouldmisfire the same way for any repo that sets
manifestPolicy: block(including this repo, or metagraphed),or for any human PR whose body just doesn't happen to use "tested"/"validated" wording even though CI
genuinely passed.