[aviggiano]

Paid benchmark Actions could consume provider and Modal credits, while the automatic history publisher held a write-capable App credential. Remove the producer, recovery, and publisher workflows and their unused orchestration helpers. Keep ordinary credential-free CI and the manual benchmark launch, validation, collection, and termination tools, with updated instructions and a regression guard against reintroducing paid workflow wiring.

Fix two runtime CI failures without weakening their assertions: replace a live OpenRouter diagnostic probe with an offline transport in the isolated test dependency, and move expensive controller snapshot coverage to the Node lane while keeping generated-child credential isolation checks in Bun. The retry test verifies session resume and actual child-process cleanup.

The dependency advisory gate also found GHSA-7q85-xj36-vmfc in the pinned ZIP reader. Update adm-zip to 0.6.1 and accept its earlier duplicate-entry rejection in the existing archive test.

Validation: full workspace build; 673 Modal tests; 102 CI-script tests; 17 focused adapter/credential tests; the Node controller forwarding regression; 17 archive-security/benchmark-analysis tests; the dependency advisory gate (zero High/Critical advisories, zero exceptions); formatting, changed-code strict lint, TypeScript, docs checks, and Knip. Changes received independent review. No paid inference or cloud benchmark was launched.

Repository operations are already complete: all three paid workflows are disabled, eight repository Actions secrets and the publisher App client-ID variable have been deleted. Operational evidence and account-owner follow-up records the stale queued Actions entry and the remaining provider-side revocation, Modal inventory, and publisher App/bypass checks that require account access.

Fixes #1129

Confidence Score: 5/5

The PR appears safe to merge; the remaining documentation omissions are non-blocking but still leave parts of the manual benchmark runbook incomplete.

Findings

1.  Incomplete collection command ▶
2.  Missing cleanup commands ▶

Fix with agent prompt

### Issue 1
docs/how-to/run-evals-on-modal.md:258-260
The manual instructions describe collection as `collect --public-results --config <path>`, but the retained CLI also requires `--state`; without it, the command exits before collecting anything. `unpack-public` likewise requires both `--bundle` and `--output`. Since this is now the only documented path for obtaining publishable results after paid compute completes, please include complete collection and extraction commands using the pair's preserved state and output paths.

### Issue 2
docs/how-to/run-evals-on-modal.md:252-256
This section names the missing-state cleanup helpers but does not show their required arguments or explain how to generate the pair list before termination. A launcher failure is exactly when an operator may have only the preserved plan, so the sole remaining orphan-cleanup runbook is not directly executable during a costly incident. Please include concrete preparation and termination commands using the previously defined candidate, generation, control directory, and checkout.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

This PR removes the automated paid benchmark, recovery, and history-publication workflows while retaining manual benchmark operations and credential-free CI coverage. It also stabilizes runtime tests by replacing a live provider diagnostic with an offline transport and redistributing expensive coverage between Node and Bun.

• Deletes write-capable and paid-compute workflow wiring plus orchestration helpers used only by those workflows.
• Updates evaluation documentation and regression tests for the retained manual benchmark lifecycle.
• Keeps runtime retry, session-resume, child-process cleanup, and credential-isolation assertions without requiring live provider access.
• Updates adm-zip to 0.6.1 so duplicate ZIP entries are rejected by the dependency as well as the surrounding archive validation.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  Operator[Operator] --> Plan[Prepare benchmark plan]
  Plan --> Validate[Validate launch guardrails]
  Validate --> Launch[Manual paid Modal launch]
  Launch --> Status[Monitor retained state]
  Status --> Collect[Collect validated public bundles]
  Collect --> Review[Review results locally]
  Review --> PR[Publish history through normal PR]
  Launch --> Terminate[Explicit termination and cleanup]
  AutomaticCI[Credential-free CI] -. no paid launch .-> Validate

Loading

_{Reviews (2) · Last reviewed commit: "Update adm-zip to patched release and re..."}

[greptile-apps]

Incomplete collection command

The manual instructions describe collection as collect --public-results --config <path>, but the retained CLI also requires --state; without it, the command exits before collecting anything. unpack-public likewise requires both --bundle and --output. Since this is now the only documented path for obtaining publishable results after paid compute completes, please include complete collection and extraction commands using the pair's preserved state and output paths.

Knowledge Base Used: Modal worker integration

Prompt To Fix With AI

This is a comment left during a code review.
Path: docs/how-to/run-evals-on-modal.md
Line: 258-260

Comment:
**Incomplete collection command**

The manual instructions describe collection as `collect --public-results --config <path>`, but the retained CLI also requires `--state`; without it, the command exits before collecting anything. `unpack-public` likewise requires both `--bundle` and `--output`. Since this is now the only documented path for obtaining publishable results after paid compute completes, please include complete collection and extraction commands using the pair's preserved state and output paths.

**Knowledge Base Used:** [Modal worker integration](https://app.greptile.com/monad-foudnation/-/custom-context/knowledge-base/monad-developers/ultrafuzz/-/docs/modal-worker-integration.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

Missing cleanup commands

This section names the missing-state cleanup helpers but does not show their required arguments or explain how to generate the pair list before termination. A launcher failure is exactly when an operator may have only the preserved plan, so the sole remaining orphan-cleanup runbook is not directly executable during a costly incident. Please include concrete preparation and termination commands using the previously defined candidate, generation, control directory, and checkout.

Knowledge Base Used:

• Execution runtime
• Modal worker integration

Prompt To Fix With AI

This is a comment left during a code review.
Path: docs/how-to/run-evals-on-modal.md
Line: 252-256

Comment:
**Missing cleanup commands**

This section names the missing-state cleanup helpers but does not show their required arguments or explain how to generate the pair list before termination. A launcher failure is exactly when an operator may have only the preserved plan, so the sole remaining orphan-cleanup runbook is not directly executable during a costly incident. Please include concrete preparation and termination commands using the previously defined candidate, generation, control directory, and checkout.

**Knowledge Base Used:**
- [Execution runtime](https://app.greptile.com/monad-foudnation/-/custom-context/knowledge-base/monad-developers/ultrafuzz/-/docs/execution-runtime.md)
- [Modal worker integration](https://app.greptile.com/monad-foudnation/-/custom-context/knowledge-base/monad-developers/ultrafuzz/-/docs/modal-worker-integration.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	adm-zip@​0.6.0 ⏵ 0.6.1	+1	+18	+1	-1

View full report