Skip to content

fix(review): manifest_missing_tests ignores real CI status and has no bot exemption #4401

Description

@JSONbored

Summary

manifest_missing_tests (src/signals/focus-manifest.ts:690-701) fires when
manifest.testExpectations.length > 0 && testFileCount === 0 && passedValidationCount === 0. The caller
(src/queue/processors.ts, in the manifestPolicyGateMode block of maybePublishPrPublicSurface)
currently computes passedValidationCount from a pure PR-body regex match (hasValidationNote) with
zero connection to the PR's actual CI/check-run results.

Confirmed false positive

JSONbored/awesome-claude PR #4719 — a fully-automated github-actions[bot]-authored PR that only
regenerates README.md — got a FAILURE conclusion on the "Gittensory Orb Review Agent" check run
("Configured validation evidence missing") even though the real CI checks all passed, because the bot's
templated PR body naturally contains none of the trigger words, and that repo's manifestPolicy: block
config escalates the finding into a hard blocker. There is also no bot/author exemption anywhere in this
code path — review.auto_review.ignore_authors only gates whether the AI review runs, not this
deterministic manifest-policy check.

Fix

  1. Make passedValidationCount's computation also consult the PR's real live CI state (reusing the
    existing cachedLiveCiAggregate infra) — a fully-green CI rollup now counts as evidence, in addition
    to (not instead of) the existing PR-body text match.
  2. Add a bot-author exemption to the manifestPolicyGateMode block, mirroring the existing
    review.auto_review.ignore_authors glob pattern, so a fully-automated bot PR is not held to
    "did you demonstrate test evidence" scrutiny meant for human contributors.

The live symptom was already worked around by softening awesome-claude's manifestPolicy from block
to advisory in its private VPS config, but the underlying detection heuristic is still wrong and would
misfire the same way for any repo that sets manifestPolicy: block (including this repo, or metagraphed),
or for any human PR whose body just doesn't happen to use "tested"/"validated" wording even though CI
genuinely passed.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.

Projects

Status
Done

Relationships

None yet

Development

No branches or pull requests

Issue actions