Skip to content

Weekly periodic evals: red lane needs triage #3038

Closed
Closed
Weekly periodic evals: red lane needs triage#3038

Description

@github-actions
github-actions
bot
opened on Oct 5, 2026

Automated weekly report — run: https://github.com/garrytan/gstack/actions/runs/37272185151

  • periodic reconciliation exit: 1 (slices job: failure, Codex slices job: success)
  • gate census reconciliation exit: 0 (census job: success)
  • pass-rate history gate exit: 0

Periodic lane

[test:paid] VERDICT RED — lane periodic/full census, attempt 1
  rule 97/98 · behavior 23/23 · judge 25/25 · quarantined 0 (0 failing the lane)
  SKIPPED 0 · INFRA 0 · INCOMPLETE 0 · unattributed 0 · ACTION REQUIRED 1
  wall 8m42s · cost $93.28

Failures and split verdicts

✗ auto-decide-preserved  rule  FAIL  assertion — AUTO_DECIDE preserve test inconclusive: outcome=timeout  [slice 27, attempt 1]  rerun: bun run scripts/test-paid-shards.ts --tier periodic --case auto-decide-preserved

Skipped, deselected and unattributed testcases

deselected testcases, zero credit (292):
   275  sibling case of a case shard (its own shard runs it)
     8  Bun hook placeholder of a skipped describe block (not a test)
     7  gate-tier case (this lane runs periodic)
     2  isolated case (its trial shards run it)

ACTION REQUIRED (1)

test/skill-e2e-auto-decide-preserved.test.ts: failed

Gate census

[test:paid] VERDICT GREEN — lane gate/full census, attempt 1
  rule 106/106 · behavior 0/0 · judge 0/0 · quarantined 0 (0 failing the lane)
  SKIPPED 0 · INFRA 0 · INCOMPLETE 0 · unattributed 0 · ACTION REQUIRED 0
  wall 10m21s · cost $88.74

Skipped, deselected and unattributed testcases

deselected testcases, zero credit (307):
   293  sibling case of a case shard (its own shard runs it)
     8  periodic-tier case (this lane runs gate)
     6  Bun hook placeholder of a skipped describe block (not a test)

Pass-rate history (ACTION REQUIRED)

(no pass-rate alarms)
Full reconciliation output 
osons-neutral-neg~t2
  slice 24  trial passed      110s  test/skill-e2e-plan.test.ts#plan-ceo-review-expansion-energy~t2
  slice 24  passed             41s  test/skill-e2e-setup-gbrain-bad-token.test.ts
  slice 25  passed            187s  test/skill-e2e-outside-voice.test.ts
  slice 25  passed             19s  test/codex-e2e.test.ts
  slice 25  passed            122s  test/skill-e2e-workflow.test.ts
  slice 25  passed            138s  test/skill-e2e-safety-codex-boundary.test.ts
  slice 25  passed             67s  test/skill-e2e-outside-plan-disabled.test.ts
  slice 25  passed             63s  test/codex-e2e-shared-libs.test.ts
  slice 25  passed             48s  test/codex-e2e-sol-scope.test.ts
  slice 25  passed             20s  test/codex-e2e-recommendation-substance.test.ts
  slice 26  trial passed       45s  test/skill-e2e-plan-prosons.test.ts#plan-review-prosons-hardstop-neg~t3
  slice 26  trial passed       53s  test/skill-e2e-plan-prosons.test.ts#plan-review-prosons-neutral-neg~t3
  slice 26  passed            202s  test/skill-e2e-design.test.ts#design-consultation-core
  slice 26  trial passed      130s  test/skill-e2e-design.test.ts#design-html-slop-gate~t1
  slice 26  passed             97s  test/skill-e2e-plan-decision-classification.test.ts
  slice 26  passed            133s  test/carve-section-loading.test.ts#carve-section-loading-browse
  slice 27  trial passed      115s  test/skill-e2e-design.test.ts#design-html-slop-gate~t2
  slice 27  passed            130s  test/carve-section-loading.test.ts#carve-section-loading-land-and-deploy
  slice 27  passed            170s  test/skill-e2e-plan-design-finding-floor.test.ts
  slice 27  failed            214s  test/skill-e2e-auto-decide-preserved.test.ts
  slice 27  passed             95s  test/skill-e2e-plan-design-plan-mode.test.ts
  slice 27  passed             70s  test/carve-section-loading.test.ts#carve-section-loading-document-release
  slice 28  trial passed      120s  test/skill-e2e-design.test.ts#design-html-slop-gate~t3
  slice 28  passed            124s  test/carve-section-loading.test.ts#carve-section-loading-qa
  slice 28  trial passed       84s  test/skill-e2e-office-hours.test.ts#office-hours-builder-wildness~t3
  slice 28  trial passed       45s  test/skill-e2e-office-hours.test.ts#office-hours-forcing-energy~t1
  slice 28  passed            121s  test/skill-e2e-plan.test.ts#plan-eng-review
  slice 28  passed            107s  test/skill-e2e-design.test.ts#design-consultation-research
  slice 29  trial passed       55s  test/skill-e2e-office-hours.test.ts#office-hours-forcing-energy~t2
  slice 29  trial passed      123s  test/skill-e2e-review-army.test.ts#review-army-simplification~t1
  slice 29  passed            134s  test/skill-e2e-plan.test.ts#plan-ceo-review
  slice 29  trial passed       78s  test/skill-e2e-review-army.test.ts#review-army-red-team~t2
  slice 29  passed             76s  test/skill-e2e-plan-devex-peer-comparison-classification.test.ts
  slice 29  passed             69s  test/skill-e2e-auq-consistency.test.ts
  slice 29  passed             72s  test/skill-routing-e2e.test.ts
  slice 29  passed             52s  test/skill-e2e-sync-gbrain-readiness.test.ts
  slice 29  trial passed       44s  test/skill-e2e-office-hours-phase4.test.ts#office-hours-phase4-fork~t3
  slice 30  trial passed       58s  test/skill-e2e-office-hours.test.ts#office-hours-forcing-energy~t3
  slice 30  trial passed      137s  test/skill-e2e-review-army.test.ts#review-army-simplification~t2
  slice 30  trial passed      102s  test/skill-e2e-review-army.test.ts#review-army-red-team~t3
  slice 30  passed             68s  test/skill-e2e-auq-verbose-vs-carved-ab.test.ts
  slice 30  trial passed       64s  test/skill-e2e-skillify.test.ts#scrape-match-path~t1
  slice 30  trial passed       50s  test/skill-e2e-skillify.test.ts#scrape-prototype-path~t1
  slice 30  passed             47s  test/skill-e2e-diagram.test.ts
  slice 31  trial passed      149s  test/skill-e2e-review-army.test.ts#review-army-simplification~t3
  slice 31  trial passed       51s  test/skill-e2e-skillify.test.ts#scrape-match-path~t2
  slice 31  trial passed       37s  test/skill-e2e-skillify.test.ts#scrape-prototype-path~t2
  slice 32  trial passed       63s  test/skill-e2e-skillify.test.ts#scrape-match-path~t3
  slice 32  trial passed       40s  test/skill-e2e-skillify.test.ts#scrape-prototype-path~t3
  slice 33  passed            250s  test/skill-e2e-overlay-harness-opus-4-7-literal-interpretation.test.ts
  slice 33  passed             97s  test/skill-e2e-overlay-harness-claude-dedicated-tools-vs-bash-sonnet.test.ts
  slice 33  passed             55s  test/skill-e2e-overlay-harness-claude-dedicated-tools-vs-bash.test.ts
  slice 33  passed             66s  test/skill-e2e-overlay-harness-claude-dedicated-tools-vs-bash-opus-5-5.test.ts
  slice 33  passed             37s  test/skill-e2e-overlay-harness-opus-4-7-effort-match-trivial.test.ts
[test:paid] collector final outcomes: 193 executed, 0 reused; 193 passed, 0 failed, 0 manual accepted (unscored; no score-cache credit) (193 attempt records from 39 collectors; every record counts)
[test:paid] deselected testcases, zero credit (292):
[test:paid]    275  sibling case of a case shard (its own shard runs it)
[test:paid]      8  Bun hook placeholder of a skipped describe block (not a test)
[test:paid]      7  gate-tier case (this lane runs periodic)
[test:paid]      2  isolated case (its trial shards run it)
[test:paid] VERDICT RED — lane periodic/full census, attempt 1
  rule 97/98 · behavior 23/23 · judge 25/25 · quarantined 0 (0 failing the lane)
  SKIPPED 0 · INFRA 0 · INCOMPLETE 0 · unattributed 0 · ACTION REQUIRED 1
  wall 8m42s · cost $93.28
[test:paid] failures and split verdicts:
  ✗ auto-decide-preserved  rule  FAIL  assertion — AUTO_DECIDE preserve test inconclusive: outcome=timeout  [slice 27, attempt 1]  rerun: bun run scripts/test-paid-shards.ts --tier periodic --case auto-decide-preserved

            46s  test/skill-e2e-qa-workflow.test.ts#qa-bootstrap
  slice 6  passed            188s  test/skill-e2e-qa-callers.test.ts#ship-exploratory-plan-checks
  slice 6  passed            194s  test/skill-e2e-qa-callers.test.ts#ship-exploratory-small-cli
  slice 6  passed            122s  test/skill-e2e-safety-codex-consult.test.ts
  slice 6  passed             80s  test/skill-e2e-third-party-actions.test.ts
  slice 6  passed             48s  test/skill-e2e-design.test.ts#design-review-detector-shim
  slice 6  passed             49s  test/skill-e2e-safety-pair-agent.test.ts
  slice 7  passed            200s  test/skill-e2e-qa-callers.test.ts#ship-exploratory-unavailable
  slice 7  passed            134s  test/skill-e2e-ship-docsync.test.ts#ship-docsync-stale-after
  slice 7  passed            157s  test/skill-e2e-review-army.test.ts#review-army-migration-safety
  slice 7  passed            110s  test/skill-e2e-plan-devex-plan-mode.test.ts
  slice 8  passed            174s  test/skill-e2e-shared-libs.test.ts#shared-libs-review-lifecycle
  slice 8  passed            123s  test/skill-e2e-review-army.test.ts#review-army-perf-n-plus-one
  slice 8  passed            125s  test/skill-e2e-deploy.test.ts#land-and-deploy-review-gate
  slice 8  passed             64s  test/skill-e2e-ship-docsync.test.ts#ship-docsync-timeout-unsettled
  slice 9  passed            246s  test/skill-e2e-plan-design-with-ui.test.ts
  slice 9  passed            207s  test/skill-e2e-qa-functional-fix.test.ts
  slice 9  passed            102s  test/skill-e2e-shared-libs.test.ts#shared-libs-unsupported-git
  slice 9  passed            239s  test/skill-e2e-qa-workflow.test.ts#qa-quick
  slice 10  passed            406s  test/skill-e2e-cso.test.ts
  slice 10  passed            125s  test/skill-e2e-ship-docsync.test.ts#ship-docsync-recovery
  slice 10  passed            147s  test/skill-e2e-skillify.test.ts
  slice 10  passed            175s  test/skill-e2e-shared-libs-paths.test.ts#shared-libs-review-index-flags
  slice 11  passed            217s  test/skill-e2e-plan-mode-no-op.test.ts
  slice 11  passed            146s  test/skill-e2e-shared-libs-paths.test.ts#shared-libs-review-prior-coverage
  slice 11  passed            199s  test/skill-e2e-qa-workflow.test.ts#qa-only-no-fix
  slice 11  passed            153s  test/skill-e2e-deploy.test.ts#land-and-deploy-workflow
  slice 12  passed            142s  test/skill-e2e-ship-docsync.test.ts#ship-docsync-late-result
  slice 12  passed            171s  test/skill-e2e-plan-devex-finding-floor.test.ts
  slice 12  passed            169s  test/skill-e2e-shared-libs-paths.test.ts#shared-libs-review-path-eligibility
  slice 12  passed            183s  test/skill-e2e-ship-docsync.test.ts#ship-docsync-stale-before
  slice 13  passed            144s  test/skill-e2e-shared-libs.test.ts#shared-libs-read-only
  slice 13  passed            342s  test/skill-e2e-workflow.test.ts
  slice 13  passed            228s  test/skill-e2e-ship-docsync.test.ts#ship-docsync-failure
  slice 13  passed            118s  test/skill-e2e-ship-docsync.test.ts#ship-docsync-missing-asset
  slice 13  passed            105s  test/skill-e2e-review-attribution.test.ts
  slice 13  passed             55s  test/skill-e2e-ship-hook-refresh.test.ts
  slice 14  passed            201s  test/skill-e2e-retro.test.ts
  slice 14  passed             60s  test/skill-e2e-ship-docsync.test.ts#ship-docsync-launch-failure
  slice 14  passed            100s  test/skill-e2e-docsync-spawned.test.ts
  slice 14  passed            118s  test/skill-e2e-review.test.ts
  slice 14  passed             65s  test/skill-e2e-design.test.ts#design-review-plugin-handoff
  slice 14  passed            110s  test/skill-e2e-deploy.test.ts#land-and-deploy-first-run
  slice 15  passed            103s  test/skill-e2e-design.test.ts#design-review-detector-shim-dom
  slice 15  passed             84s  test/skill-e2e-ship-skip.test.ts
  slice 15  passed            204s  test/skill-e2e-plan.test.ts#plan-review-report
  slice 15  passed             52s  test/skill-e2e-ship-hook-consent.test.ts
  slice 15  passed             62s  test/skill-e2e-plan-tune.test.ts
  slice 15  passed             50s  test/skill-e2e-ask-user-question-format-compliance.test.ts
  slice 15  passed             75s  test/skill-e2e-session-intelligence.test.ts
  slice 15  passed             43s  test/skill-e2e-triage.test.ts
  slice 15  passed             52s  test/skill-e2e-coverage-audit.test.ts
  slice 15  passed             43s  test/skill-e2e-deploy.test.ts#canary-workflow
  slice 15  passed             43s  test/skill-e2e-review-army.test.ts#review-army-delivery-audit
  slice 15  passed             40s  test/skill-e2e-investigate-owned-termination.test.ts
  slice 15  passed             43s  test/skill-e2e-investigate-owned-completion.test.ts
  slice 15  passed            183s  test/skill-e2e-plan-ceo-plan-mode.test.ts
  slice 15  passed              8s  test/skill-e2e-hermetic-canary.test.ts
  slice 16  passed             29s  test/skill-e2e-diagram.test.ts
  slice 16  passed             35s  test/skill-e2e-learnings.test.ts
  slice 16  passed             48s  test/skill-e2e-design.test.ts#plan-design-review-no-ui-scope
[test:paid] collector final outcomes: 97 executed, 0 reused; 97 passed, 0 failed, 0 manual accepted (unscored; no score-cache credit) (97 attempt records from 69 collectors; every record counts)
[test:paid] deselected testcases, zero credit (307):
[test:paid]    293  sibling case of a case shard (its own shard runs it)
[test:paid]      8  periodic-tier case (this lane runs gate)
[test:paid]      6  Bun hook placeholder of a skipped describe block (not a test)
[test:paid] VERDICT GREEN — lane gate/full census, attempt 1
  rule 106/106 · behavior 0/0 · judge 0/0 · quarantined 0 (0 failing the lane)
  SKIPPED 0 · INFRA 0 · INCOMPLETE 0 · unattributed 0 · ACTION REQUIRED 0
  wall 10m21s · cost $88.74
[test:paid] report: every planned shard accounted and passed

Policy: EVAL_POLICY and CASE_QUARANTINE in test/helpers/periodic-exclude-data.ts; history: bun run eval:pass-rates.

Activity

  1. garrytan
    added 2 commits that reference this issue on Oct 6, 2026

    fix(plan-ceo-review): the mode handoff helper reminds the model to le…

    76ccc69

    fix(plan-ceo-review): show the mode handoff line through a PostToolUs…

    2591231
  2. capy-ai
    added a commit that references this issue on Oct 6, 2026

    v1.91.32.0 fix: Codex works from zsh, free text never enters shell co…

    28f1385

  3. capy-ai commented on Oct 6, 2026

    @capy-ai
    capy-aibot
    on Oct 6, 2026 – with Capy AI
    Contributor

    Fixed in v1.91.32.0 (#3057). auto-decide-preserved failed because the model paraphrased the Step 0E handoff line. /plan-ceo-review now shows that line through a PostToolUse hook, and diagnostic measurement went from 1/10 (a reminder alone) to 10/10, with the line visible in every trial.

    Upgrade: Claude Code /gstack-upgrade; Codex and other hosts cd <source from ./setup --status> && git pull && ./setup --host <host>. If it still reproduces, reopen with your ./setup --status output.

  4. capy-ai
    closed this as completedon Oct 6, 2026
  5. levi85
    added a commit that references this issue on Oct 14, 2026

    v1.91.32.0 fix: Codex works from zsh, free text never enters shell co…

    11b9737
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions