Skip to content

v1.91.16.0 fix: safe installs and upgrades, skills start everywhere, truthful completion (parallel wave) - #3014

Merged
garrytan merged 29 commits into
mainfrom
garrytan/parallel-waves
Oct 3, 2026
Merged

garrytan merged 29 commits into
mainfrom
garrytan/parallel-waves

Conversation

@garrytan

@garrytan garrytan commented Oct 2, 2026 •

Copy link
Copy Markdown
Owner

One reliability release, planned with /autoplan (CEO, DX and Eng reviews, each with Claude and GPT-6 Astra voices; design skipped, no UI) and approved by Garry. It ships three promises, each backed by tests.

1. Installing or upgrading never quietly changes another copy

2. Skills start everywhere

3. Truthful completion

Also in this release

Co-authored-by credits for the Copilot port are on its commit (@andrey-esipov, @ridermw, @lolisaigao1234).

Eval and test reliability (found while shipping)

The full paid gate ran on every push because this PR regenerates every SKILL.md, which exposed rotating reds. Each was traced to its source and fixed, with no thresholds lowered and no trials added:

  • /qa protocol names defined so the clarity judge can act on them; plan-eng-review's scope gate always sends its menu (no headless "Scope pending" exit).
  • /ship documentation section has one saved source inserted by cat, never retyped.
  • Harness: async PTY judge (no frozen cleanup deadlines), PTY runs end promptly on an idle turn-end or API Error: panel, one position-bound coverage-read rule (consolidated two detectors; replayed on 243 captured sessions), plan-design focus-title rule, skillify state-root fixture, TPA shims out of the agent's cwd, coverage prompts with absolute paths.
  • Flakes fixed at the cause: terminal-agent concurrent startup, qa-evidence resumed pipe, design daemon log-open crash (a real product bug from the lazy log), macOS group-kill EPERM, Windows path/CRLF fixtures, telemetry test on Bun 1.3.
  • One timing change worth reviewing: three bws setup cases moved from a hardcoded 30 s to the standard JUDGE_MS tier (120 s) per the repo's timeout policy.
  • Census of all gate evals since 09-27 is in TODOS: judges' actionability sits exactly at the 4.0 gate (per-skill clarity work, not a lower bar), --case can't select loop-registered cases, and an API-error fail-fast policy call.

Validation

  • Full free suite on Ubicloud (Bun 1.4.0): 24,194 pass, 0 fail, no flaky retries (8e652b5); CI free suite, Windows, macOS named regressions and the full paid gate green on bc57ca5.
  • Focused per-workstream tests, each regression red before its fix; typecheck 0 errors; typecheck:test no new diagnostics; parity, module-size, state-root, spawnSync tripwire, skill:check, format:cso clean; credential scan 0 HIGH.
  • Live OpenAI check of the new design defaults (gpt-5.5 + gpt-image-2): image and vision calls PASS.

garrytan and others added 10 commits October 2, 2026 18:44
…ex CLI in the CI image, macOS named-regressions job (#1948, #1706, #1997)
…ns, truthful 'not assessed' reports, env.kv false positive, Windows browse spawn fixes (#2894, #2912, #2048, #2637, #2805)
…, bounded timeouts with KILL escalation, probe cache identity, guarded mktemp, skills hidden in nested Codex (#2776, #2914, #2787, #2881, #2847)
…er-scope migrations, host tiers and conformance kit, OpenCode/Hermes fixes, Copilot CLI (experimental), disabled_skills, slimmer CLAUDE.md (#1925, #2879, #2347, #2338, #2626, #2629, #2825, #2015, #393, #1206, #2096)

Co-authored-by: Andrey Esipov <61357035+andrey-esipov@users.noreply.github.com>
Co-authored-by: Matthew Williams <5863886+ridermw@users.noreply.github.com>
Co-authored-by: Rocky Wu <92764750+lolisaigao1234@users.noreply.github.com>
… — one literal skill-start line, context-recovery helper, no eval in skill blocks, $N token tripwire, honest plan-mode precedence (#2763, #2896, #2851)
…ig installs, space-safe alias roots, per-host /cso launcher path (#1882, #2906)
…orphaned browse daemons, symlinked-TMPDIR (macOS) fixes, gbrain-on-PATH isolation (#2895, #2816, #2829, #2841, #2982, #2911, #2909, #2908, #2910)
… docs, TODOS, touchfiles, duration seeds, Bun 1.3 telemetry test fix, replay fixtures pinned to their recorded bytes
@trunk-io

trunk-io Bot commented Oct 2, 2026 •

Copy link
Copy Markdown

😎 This pull request was merged.

@github-actions

github-actions Bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

E2E Evals: ✅ PASS

[test:paid] VERDICT GREEN — lane gate/pr, attempt 1
  rule 94/339 · behavior 0/28 · judge 24/24 · quarantined 0 (0 failing the lane)
  SKIPPED 274 · INFRA 0 · INCOMPLETE 0 · unattributed 22 · ACTION REQUIRED 0
  wall 10m05s · cost $86.82

104 executed, 14 reused rule/judge records | 0 manual accepted (unscored; no score-cache credit) | $86.82 rule/judge cost | reconcile exit: 0

Profile: pr / full-fallback; selected behaviors: 103, judges: 24. Deferred to scheduled/release coverage: 108 behaviors and 29 changed prompt files. Deferred checks did not run and receive no PR-pass credit.

Rule and judge shards
Shard Automated result Manual/unscored Executed Reused Status Cost
e2e-review-army/skill-e2e-review-army--review-army-perf-n-plus-one 1/1 0 1 0 ✅ $0.99
e2e/skill-e2e-shared-libs--shared-libs-review-revalidation 1/1 0 1 0 ✅ $4.32
e2e-session-intelligence/skill-e2e-session-intelligence 4/4 0 4 0 ✅ $0.92
e2e/skill-e2e-shared-libs-paths--shared-libs-review-index-flags 1/1 0 1 0 ✅ $2.15
e2e-triage/skill-e2e-triage 1/1 0 1 0 ✅ $0.53
e2e-review-army/skill-e2e-review-army--review-army-migration-safety 1/1 0 1 0 ✅ $1.18
e2e-retro/skill-e2e-retro 1/1 0 1 0 ✅ $1.24
e2e-ship-docsync/skill-e2e-ship-docsync--ship-docsync-timeout-unsettled 1/1 0 1 0 ✅ $0.53
e2e-skillify/skill-e2e-skillify 3/3 0 3 0 ✅ $2.37
e2e-ship-docsync/skill-e2e-ship-docsync--ship-docsync-launch-failure 1/1 0 1 0 ✅ $0.46
e2e-review-army/skill-e2e-review-army--review-army-quality-score 1/1 0 1 0 ✅ $0.39
e2e-qa-workflow/skill-e2e-qa-workflow 3/3 0 3 0 ✅ $3.41
e2e-ship-docsync/skill-e2e-ship-docsync--ship-docsync-current 1/1 0 1 0 ✅ $3.13
e2e-browse/skill-e2e-bws 6/6 0 6 0 ✅ $0.82
e2e/skill-e2e-shared-libs--shared-libs-review-lifecycle 1/1 0 1 0 ✅ $2.37
e2e-qa-callers/skill-e2e-qa-callers--ship-exploratory-unavailable 1/1 0 1 0 ✅ $1.66
e2e-ship-docsync/skill-e2e-ship-docsync--ship-docsync-stale-after 1/1 0 1 0 ✅ $0.94
e2e/skill-e2e-workflow 4/4 0 4 0 ✅ $2.89
e2e/skill-e2e-test-value 3/3 0 3 0 ✅ $2.35
llm-judge/skill-llm-eval 24/24 0 10 14 ✅ $0.6
e2e-qa-functional-fix/skill-e2e-qa-functional-fix 2/2 0 2 0 ✅ $2.93
e2e-qa-callers/skill-e2e-qa-callers--review-exploratory-small-cli 1/1 0 1 0 ✅ $1.81
e2e-qa-callers/skill-e2e-qa-callers--ship-exploratory-late-input 1/1 0 1 0 ✅ $2.23
e2e-cso/skill-e2e-cso 1/1 0 1 0 ✅ $1.5
e2e/skill-e2e-shared-libs--shared-libs-unsupported-git 1/1 0 1 0 ✅ $0.48
e2e-ship-docsync/skill-e2e-ship-docsync--ship-docsync-failure 1/1 0 1 0 ✅ $1.73
e2e-qa-functional/skill-e2e-qa-functional 2/2 0 2 0 ✅ $2.64
e2e-ship-docsync/skill-e2e-ship-docsync--ship-docsync-completion 1/1 0 1 0 ✅ $4.24
e2e-design/skill-e2e-design--design-review-detector-shim 1/1 0 1 0 ✅ $0.36
e2e-review-army/skill-e2e-review-army--review-army-delivery-audit 1/1 0 1 0 ✅ $0.33
e2e/skill-e2e-coverage-audit 2/2 0 2 0 ✅ $0.75
e2e-review-army/skill-e2e-review-army--review-army-json-findings 1/1 0 1 0 ✅ $0.22
e2e/skill-e2e-investigate-owned-termination 2/2 0 2 0 ✅ $0.51
e2e-design/skill-e2e-design--plan-design-review-no-ui-scope 1/1 0 1 0 ✅ $0.35
e2e-third-party-actions/skill-e2e-third-party-actions 5/5 0 5 0 ✅ $1.25
e2e-learnings/skill-e2e-learnings 1/1 0 1 0 ✅ $0.36
e2e-plan-tune/skill-e2e-plan-tune 1/1 0 1 0 ✅ $0.51
e2e-design/skill-e2e-design--design-review-detector-shim-dom 1/1 0 1 0 ✅ $0.65
e2e-review-attribution/skill-e2e-review-attribution 3/3 0 3 0 ✅ $0.94
e2e/skill-e2e-plan--plan-review-report 1/1 0 1 0 ✅ $1.27
e2e/skill-e2e-investigate-owned-completion 1/1 0 1 0 ✅ $0.25
e2e/skill-e2e-ship-hook-consent 2/2 0 2 0 ✅ $0.56
e2e/skill-e2e-ship-hook-refresh 1/1 0 1 0 ✅ $0.31
e2e/skill-e2e-ship-skip 1/1 0 1 0 ✅ $0.62
e2e-ship-docsync/skill-e2e-ship-docsync--ship-docsync-missing-marker 1/1 0 1 0 ✅ $0.48
e2e-docsync-spawned/skill-e2e-docsync-spawned 1/1 0 1 0 ✅ $1.12
e2e-ship-docsync/skill-e2e-ship-docsync--ship-docsync-stale-before 1/1 0 1 0 ✅ $0.94
e2e/skill-e2e-shared-libs--shared-libs-read-only 1/1 0 1 0 ✅ $1.12
e2e-ship-docsync/skill-e2e-ship-docsync--ship-docsync-late-result 1/1 0 1 0 ✅ $0.92
e2e-design/skill-e2e-design--design-review-plugin-handoff 1/1 0 1 0 ✅ $0.63
e2e-review/skill-e2e-review 2/2 0 2 0 ✅ $1.29
e2e/skill-e2e-shared-libs-paths--shared-libs-review-path-eligibility 1/1 0 1 0 ✅ $3.9
e2e-ship-docsync/skill-e2e-ship-docsync--ship-docsync-recovery 1/1 0 1 0 ✅ $0.8
e2e-hermetic/skill-e2e-hermetic-canary 2/2 0 2 0 ✅ $0.02
e2e/skill-e2e-shared-libs-paths--shared-libs-review-prior-coverage 1/1 0 1 0 ✅ $2.12
e2e-qa-callers/skill-e2e-qa-callers--ship-exploratory-plan-checks 1/1 0 1 0 ✅ $1.88
e2e-qa-callers/skill-e2e-qa-callers--ship-exploratory-small-cli 1/1 0 1 0 ✅ $1.8
e2e-ship-docsync/skill-e2e-ship-docsync--ship-docsync-missing-asset 1/1 0 1 0 ✅ $1.6
e2e-ship-docsync/skill-e2e-ship-docsync--ship-docsync-store 1/1 0 1 0 ✅ $3.46
e2e-deploy/skill-e2e-deploy 6/6 0 6 0 ✅ $4.69
Fail-closed reconciliation
atory-unavailable
  slice 8  passed            150s  test/skill-e2e-ship-docsync.test.ts#ship-docsync-stale-after
  slice 8  passed            158s  test/skill-e2e-shared-libs.test.ts#shared-libs-review-lifecycle
  slice 8  passed            265s  test/skill-e2e-plan-design-with-ui.test.ts
  slice 9  passed            123s  test/skill-e2e-ship-docsync.test.ts#ship-docsync-late-result
  slice 9  passed            162s  test/skill-e2e-shared-libs-paths.test.ts#shared-libs-review-path-eligibility
  slice 9  passed            141s  test/skill-e2e-ship-docsync.test.ts#ship-docsync-stale-before
  slice 9  passed            187s  test/skill-e2e-shared-libs.test.ts#shared-libs-read-only
  slice 9  passed            155s  test/skill-e2e-review.test.ts
  slice 9  passed             84s  test/skill-e2e-design.test.ts#design-review-plugin-handoff
  slice 10  passed            148s  test/skill-e2e-skillify.test.ts
  slice 10  passed            145s  test/skill-e2e-shared-libs-paths.test.ts#shared-libs-review-index-flags
  slice 10  passed             79s  test/skill-e2e-ship-docsync.test.ts#ship-docsync-timeout-unsettled
  slice 10  passed            180s  test/skill-e2e-review-army.test.ts#review-army-migration-safety
  slice 10  passed            158s  test/skill-e2e-retro.test.ts
  slice 10  passed             49s  test/skill-e2e-ship-docsync.test.ts#ship-docsync-launch-failure
  slice 10  passed             66s  test/skill-e2e-session-intelligence.test.ts
  slice 10  passed             40s  test/skill-e2e-triage.test.ts
  slice 11  passed            111s  test/skill-e2e-docsync-spawned.test.ts
  slice 11  passed             56s  test/skill-e2e-ship-docsync.test.ts#ship-docsync-missing-marker
  slice 11  passed             65s  test/skill-e2e-office-hours-auto-mode.test.ts
  slice 11  passed             89s  test/skill-e2e-third-party-actions.test.ts
  slice 11  passed             76s  test/skill-e2e-design.test.ts#design-review-detector-shim-dom
  slice 11  passed             88s  test/skill-e2e-ship-skip.test.ts
  slice 11  passed            103s  test/skill-e2e-review-attribution.test.ts
  slice 11  passed            263s  test/skill-e2e-plan.test.ts#plan-review-report
  slice 11  passed             41s  test/skill-e2e-ship-hook-refresh.test.ts
  slice 11  passed             49s  test/skill-e2e-ship-hook-consent.test.ts
  slice 11  passed             68s  test/skill-e2e-plan-tune.test.ts
  slice 11  passed             57s  test/skill-e2e-ask-user-question-format-compliance.test.ts
  slice 11  passed             45s  test/skill-e2e-investigate-owned-completion.test.ts
  slice 11  passed             25s  test/skill-e2e-learnings.test.ts
  slice 12  passed             59s  test/skill-e2e-coverage-audit.test.ts
  slice 12  passed             31s  test/skill-e2e-review-army.test.ts#review-army-delivery-audit
  slice 12  passed             42s  test/skill-e2e-investigate-owned-termination.test.ts
  slice 12  passed             37s  test/skill-e2e-design.test.ts#design-review-detector-shim
  slice 12  passed             35s  test/skill-e2e-plan-ceo-plan-mode.test.ts
  slice 12  passed             55s  test/skill-e2e-design.test.ts#plan-design-review-no-ui-scope
  slice 12  passed             29s  test/skill-e2e-review-army.test.ts#review-army-json-findings
[test:paid] collector final outcomes: 104 executed, 14 reused; 118 passed, 0 failed, 0 manual accepted (unscored; no score-cache credit) (118 attempt records from 61 collectors; every record counts)
[test:paid] report: ⚠ 1 shard(s) passed with EVERY test skipped — they verified nothing:
  ⚠ test/skill-routing-e2e.test.ts (11 skipped — external service missing or tier mismatch)
[test:paid] VERDICT GREEN — lane gate/pr, attempt 1
  rule 94/339 · behavior 0/28 · judge 24/24 · quarantined 0 (0 failing the lane)
  SKIPPED 274 · INFRA 0 · INCOMPLETE 0 · unattributed 22 · ACTION REQUIRED 0
  wall 10m05s · cost $86.82
[test:paid] report: every planned shard accounted and passed

Sliced lane: planner → duration-packed executors → fail-closed report. Behavior cases run a pre-registered 3-trial panel (PASS at 2/3 with no contract violation); a PASS 2/3 is shown with its failed trial, never as a clean pass. Reused results retain their original provenance and expiry.

…, truthful home-guard attribution with --attribute-home, gstack-paths shim in ship-hook fixtures, Copilot in team-init probes, test-of-test detector, cookie review fixture pinned to the approved excerpt
…ath.relative fixture containment, ship push test spawn timeouts
…s so a loaded macOS runner can't time out the single-batch call
…so overlapping sessions keep their cleanup deadline, bws setup cases use the standard JUDGE_MS tier
…is a shebang script, like the PTY harness it covers)
…d, and skips the Step 7 artifact nothing consumes (removes the mechanism behind 3 of 4 historical timeouts)
…r record before spawning, like the product) and qa-evidence resumed pipe (barrier on a full high-water mark instead of a sleep)
…g the daemon; shutdown test awaits the stubbed exit
…TACK_HOME bypass), negative cases check every root; tpa shims live outside the agent's cwd and answer consistently
…ls and [+] groups; coverage prompts name absolute skill paths; TODOs from the gate eval census
…Scope pending' exit); PTY harness ends idle turn-ends and API-error panels promptly with a precise diagnostic
…&& chains (labels, context ranges, git displays in either order) and still rejects forged output
… (never retyped); spawned-marker evidence is the child output; refreshed codex/factory ship goldens
…-command observer refuses backslash paths by design)
…ess 'dimensions or focus' wording (replayed on the captured CI run, negative controls kept)
@capy-ai
capy-ai Bot marked this pull request as ready for review October 3, 2026 03:31
This was referenced Oct 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant