Skip to content

orb(safety): SELFHOST_DEPLOYMENT_MODE=dry-run does not reach the executor — a 'posts nothing' instance still cancels CI, globally blacklists contributors, and writes fabricated merge outcomes #9130

Description

@JSONbored

Problem

SELFHOST_DEPLOYMENT_MODE=dry-run — the posture .env.selfhost.example:71 ships as the recommended
first-boot default — does not stop the action executor. It is enforced one layer too low: inside the
HTTP client, after every decision that depends on it has already been made.

forcedSelfhostMode has exactly two references in src/, both in the same file:

src/github/client.ts:693:  export function forcedSelfhostMode(env: …): AgentActionMode | null {
src/github/client.ts:718:  const effectiveMode = forcedSelfhostMode(env) ?? mode;

Line 718 is inside makeInstallationOctokit. The executor computes its own mode from a completely
different input set (src/settings/agent-execution.ts:38-42):

export function resolveAgentActionMode(input: { globalPaused: boolean; agentPaused?; agentDryRun? }): AgentActionMode {
  if (input.globalPaused || input.agentPaused === true) return "paused";
  if (input.agentDryRun === true) return "dry_run";
  return "live";
}

SELFHOST_DEPLOYMENT_MODE appears nowhere in it. So under dry-run the executor computes live and
runs the entire mutation path. mergePullRequest returns the synthetic { merged: true, sha: null }
(client.ts:682) without throwing, so the executor takes its success branch and proceeds to do real,
non-GitHub work:

consequence site
writes a real pr_outcome row recording "merged" — and its if (existing) return probe makes that row absorbing, so the PR's true later outcome can never be recorded agent-action-executor.ts:1055outcomes-wire.ts:365-402
issues a real POST /repos/{r}/actions/runs/{id}/cancel, killing the contributor's CI (raw timeoutFetch, outside the octokit hook) agent-action-executor.ts:603-608src/github/app.ts:612
fires Discord/Slack notifications with outcome: "merged" agent-action-executor.ts:620-631
maybeEscalateModeration short-circuits only on args.mode !== "live" — mode is live — so it records a violation and can upsertGlobalContributorBlacklist: a permanent global ban written off a mutation that never happened agent-action-executor.ts:671, :753-769

What we promise

.env.example:30-34 states the dry-run instance "posts NOTHING … so it can shadow the live cloud App
safely." What actually happens on a shadow instance: contributor CI is cancelled in the shadowed repo,
contributors are globally blacklisted, operators are paged about merges that never occurred, and the
calibration ledger is permanently poisoned with fabricated ground truth — which then feeds
computeGateEvalisHoldOnly / isCloseHoldOnly and changes the real instance's behaviour.

Trigger

Any self-host started from the shipped .env.selfhost.example, on the first contributor PR that reaches
a merge or close disposition.

Impact

The instance-level kill switch does less than every document describing it claims, and its failure mode
writes durable, contributor-visible harm (cancelled CI, global bans) plus durable data corruption
(fabricated outcomes). A safety control that is trusted precisely when an operator is least sure of their
setup is the worst place for this gap.

Requirements

  • Fold forcedSelfhostMode(env) into resolveAgentActionMode — or introduce a single
    resolveEffectiveActionMode(env, settings) that every call site uses — so the instance switch is the
    first precedence term, not an HTTP-layer afterthought.
  • Once that lands, delete the ad-hoc mode !== "live" guards that exist to compensate; they are
    individually blind to this switch (src/review/visual/actions-fallback.ts:106 checks a mode that never
    sees SELFHOST_DEPLOYMENT_MODE — the orb(safety): visual-capture workflow_dispatch bypasses the write-suppression chokepoint and fires under dry_run #9067 fix has the same blindness).
  • Route the CI-cancel call (src/github/app.ts:612) through the suppression chokepoint; it is the last
    remaining installation-scoped write on raw timeoutFetch and a direct sibling of orb(safety): visual-capture workflow_dispatch bypasses the write-suppression chokepoint and fires under dry_run #9067.
  • Make the synthetic merge return not look like success to the executor, so a suppressed mutation can
    never drive downstream side effects.
  • Add an invariant test: with SELFHOST_DEPLOYMENT_MODE=dry-run, a full merge disposition produces zero
    GitHub writes, zero outcome rows, zero notifications, and zero moderation escalations.

Test Coverage Requirements

99%+ patch coverage, branch-counted. Both arms of the instance-mode precedence, plus the four downstream
side effects asserted absent under dry-run.

Links & Resources

Boundaries

Mode precedence and the executor's side-effect gating. No change to what dry_run means or to the
per-repo agentDryRun switch.

maintainer-only — kill-switch architecture.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions