Skip to content

v1.88.1.0 chore: integrate upstream gstack v1.88.1.0 #23

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
nezovskii merged 5 commits into from
Sep 23, 2026
Merged

v1.88.1.0 chore: integrate upstream gstack v1.88.1.0 #23

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
636175d
v1.87.6.0 fix: make checks reliable and everyday validation faster (#…
garrytan Sep 22, 2026
b9706f3
v1.88.1.0 fix: harden credential boundaries and owned state (#2942)
garrytan Sep 23, 2026
363f425
chore: merge upstream gstack v1.88.1.0
nezovskii Sep 23, 2026
64123b7
fix: preserve session evidence across abort races
nezovskii Sep 23, 2026
3e7bae8
fix: preserve host skill generation filter API
nezovskii Sep 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
84 changes: 67 additions & 17 deletions .github/workflows/evals-periodic.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ name: Periodic Evals
# tests can't rot invisibly — the class where the autoplan-dual-voice E2E was
# silently broken for months until a lucky local diff selected it. Engine:
# scripts/test-paid-shards.ts (the same runner local eval:bg:periodic uses):
# one planner manifest, 6 ordinary slices plus dedicated Autoplan slice 7, and a FAIL-CLOSED report — a slice
# one planner manifest, 6 ordinary slices plus overlay and Autoplan slices, and a FAIL-CLOSED report — a slice
# whose artifact never landed is a failure, not an absence. The gate-census
# job is the weekly EVALS_ALL backstop for the gate tier (PR lanes are
# diff-billed, so without it the full gate census might never execute
Expand All @@ -21,6 +21,9 @@ concurrency:

env:
IMAGE: ghcr.io/actvox/gstack/ci
EVALS_PROFILE: full
EVALS_FRESH: "1"
EVALS_CACHE_PURPOSE: periodic

jobs:
build-image:
Expand Down Expand Up @@ -95,22 +98,32 @@ jobs:
- name: Emit run manifest (ALL periodic tests minus reasoned excludes)
env:
EVALS_ALL: "1"
run: EVALS_TIER=periodic bun --no-install run scripts/test-paid-shards.ts --tier periodic --emit-plan /tmp/paid-plan/manifest.json --slices 7 --autoplan-slice
run: EVALS_TIER=periodic bun --no-install run scripts/test-paid-shards.ts --tier periodic --emit-plan /tmp/paid-plan/manifest.json --slices 8 --autoplan-slice

- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: paid-plan
path: /tmp/paid-plan/manifest.json
retention-days: 30

- name: Emit gate census manifest (ALL gate tests)
env:
EVALS_ALL: "1"
run: EVALS_TIER=gate bun run scripts/test-paid-shards.ts --tier gate --emit-plan /tmp/gate-census-plan/manifest.json --slices 6

- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: gate-census-plan
path: /tmp/gate-census-plan/manifest.json
retention-days: 30

eval-slices:
if: vars.ENABLE_PAID_EVALS == 'true'
runs-on: ubuntu-latest
needs: [build-image, plan-slices]
# Six ordinary slices retain their existing walls and concurrency. Slice 7
# runs only Autoplan: its specified 172min two-attempt wall leaves 28min
# for setup/upload. This is not a measured latency bound for ordinary work.
timeout-minutes: 200
# Eight slices retain every registered case and retry. The complete
# census needs at most 318m40 per slice, plus 20 minutes setup/upload.
timeout-minutes: 355
permissions:
contents: read
packages: read
Expand All @@ -123,7 +136,7 @@ jobs:
strategy:
fail-fast: false
matrix:
slice: [1, 2, 3, 4, 5, 6, 7]
slice: [1, 2, 3, 4, 5, 6, 7, 8]
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
Expand Down Expand Up @@ -161,7 +174,7 @@ jobs:
name: paid-plan
path: /tmp/paid-plan

- name: Run slice ${{ matrix.slice }}/7
- name: Run slice ${{ matrix.slice }}/8
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
Expand Down Expand Up @@ -202,8 +215,9 @@ jobs:
gate-census:
if: vars.ENABLE_PAID_EVALS == 'true'
runs-on: ubuntu-latest
needs: build-image
timeout-minutes: 300
needs: [build-image, plan-slices]
# Six slices need at most 330m each, plus 20 minutes setup/upload.
timeout-minutes: 350
permissions:
contents: read
packages: read
Expand All @@ -213,6 +227,12 @@ jobs:
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
options: --user runner
strategy:
# Four file workers total, each retaining two in-file case workers.
fail-fast: false
max-parallel: 4
matrix:
slice: [1, 2, 3, 4, 5, 6]
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
Expand All @@ -232,23 +252,27 @@ jobs:

- run: bun run build

- name: Run full gate census
- uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
name: gate-census-plan
path: /tmp/gate-census-plan

- name: Run gate census slice ${{ matrix.slice }}/6
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
GEMINI_API_KEY: ${{ secrets.GEMINI_API_KEY }}
PLAYWRIGHT_BROWSERS_PATH: /opt/playwright-browsers
EVALS_ALL: "1"
EVALS_JOBS: "4"
EVALS_JOBS: "1"
EVALS_CONCURRENCY: "2"
GSTACK_EVAL_DIR: /tmp/gate-census-results
run: EVALS_TIER=gate bun run scripts/test-paid-shards.ts --tier gate
run: EVALS_TIER=gate bun run scripts/test-paid-shards.ts --tier gate --plan /tmp/gate-census-plan/manifest.json --slice ${{ matrix.slice }}

- name: Upload census results
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: gate-census
name: gate-census-${{ matrix.slice }}
path: /tmp/gate-census-results
retention-days: 90

Expand Down Expand Up @@ -284,8 +308,20 @@ jobs:
path: /tmp/paid-report
merge-multiple: true

- uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
name: gate-census-plan
path: /tmp/gate-census-report

- uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
pattern: gate-census-[0-9]*
path: /tmp/gate-census-report
merge-multiple: true

- name: Reconcile slices against the manifest (fail-closed)
id: reconcile
if: always()
run: |
set +e
EVALS_TIER=periodic bun --no-install run scripts/test-paid-shards.ts --tier periodic --report /tmp/paid-report | tee /tmp/report.txt
Expand All @@ -295,11 +331,19 @@ jobs:
# (caught by the ship review army; the wiring test now pins this).
echo "exit=${PIPESTATUS[0]}" >> "$GITHUB_OUTPUT"

- name: Reconcile gate census against the manifest (fail-closed)
id: gate-reconcile
if: always()
run: |
set +e
EVALS_TIER=gate bun run scripts/test-paid-shards.ts --tier gate --report /tmp/gate-census-report | tee /tmp/gate-report.txt
echo "exit=${PIPESTATUS[0]}" >> "$GITHUB_OUTPUT"

# A red weekly lane nobody must action is waste — upsert ONE tracking
# issue (never a new issue per week) with the reconciliation output, so
# failures have an owner-visible artifact with history in one place.
- name: Upsert tracking issue on failure
if: steps.reconcile.outputs.exit != '0' || needs.gate-census.result == 'failure'
if: always() && (steps.reconcile.outputs.exit != '0' || steps.gate-reconcile.outputs.exit != '0' || needs.eval-slices.result != 'success' || needs.gate-census.result != 'success')
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
Expand All @@ -310,12 +354,18 @@ jobs:
echo "Automated weekly report — run: ${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}"
echo
echo "- periodic reconciliation exit: ${{ steps.reconcile.outputs.exit }}"
echo "- periodic slices job: ${{ needs.eval-slices.result }}"
echo "- gate census reconciliation exit: ${{ steps.gate-reconcile.outputs.exit }}"
echo "- gate census job: ${{ needs.gate-census.result }}"
echo
echo '```'
tail -c 6000 /tmp/report.txt 2>/dev/null || echo "(no reconciliation output)"
echo '```'
echo
echo '```'
tail -c 6000 /tmp/gate-report.txt 2>/dev/null || echo "(no gate census reconciliation output)"
echo '```'
echo
echo "Exclusion policy: test/helpers/periodic-exclude-data.ts (every entry needs reason + tracking; removal re-activates the file next week)."
} > "$BODY_FILE"
EXISTING=$(gh issue list --repo "$GITHUB_REPOSITORY" --state open --search "in:title \"$TITLE\"" --json number --jq '.[0].number // empty')
Expand All @@ -327,5 +377,5 @@ jobs:
fi

- name: Fail the workflow when reconciliation failed
if: steps.reconcile.outputs.exit != '0'
if: always() && (steps.reconcile.outputs.exit != '0' || steps.gate-reconcile.outputs.exit != '0' || needs.eval-slices.result != 'success' || needs.gate-census.result != 'success')
run: exit 1
87 changes: 75 additions & 12 deletions .github/workflows/evals.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,9 @@ concurrency:

env:
IMAGE: ghcr.io/actvox/gstack/ci
# PRs run changed fast probes; manual runs retain the complete gate census.
EVALS_PROFILE: ${{ github.event_name == 'pull_request' && 'pr' || 'full' }}
EVALS_FRESH: ${{ github.event_name == 'workflow_dispatch' && '1' || '' }}

jobs:
# Build Docker image with pre-baked toolchain (cached — only rebuilds on Dockerfile/lockfile change)
Expand All @@ -32,6 +35,7 @@ jobs:
packages: write
outputs:
image-tag: ${{ steps.meta.outputs.tag }}
runtime-id: ${{ steps.runtime.outputs.id }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7

Expand Down Expand Up @@ -84,6 +88,15 @@ jobs:
${{ steps.meta.outputs.tag }}
${{ env.IMAGE }}:latest

- name: Identify the installed eval runtime
id: runtime
if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository
env:
EVAL_IMAGE: ${{ steps.meta.outputs.tag }}
run: |
docker manifest inspect "$EVAL_IMAGE" > /tmp/eval-runtime-manifest.json
echo "id=$(sha256sum /tmp/eval-runtime-manifest.json | cut -d ' ' -f1)" >> "$GITHUB_OUTPUT"

# ── Sliced lane (the ONLY paid lane; legacy 17-row matrix deleted) ──────────
# One PLANNER computes diff selection + the slice plan ONCE (killing
# per-slice selector divergence); K executors consume the manifest; the
Expand Down Expand Up @@ -139,7 +152,9 @@ jobs:
# 40-way per row queued claude session STARTUP behind 39 siblings and ate
# per-test budgets — the documented timeout-flake family). Tune with
# parity data before raising.
timeout-minutes: 35
# The complete gate census needs at most 197 minutes per slice; keep
# 20 minutes for setup/upload without preempting configured retries.
timeout-minutes: 220
permissions:
contents: read
packages: read
Expand Down Expand Up @@ -191,6 +206,16 @@ jobs:
name: paid-plan
path: /tmp/paid-plan

# Only this PR's receipts are eligible. No base-branch or cross-PR restore
# prefix; every receipt also verifies exact inputs and its original age.
- name: Restore this PR's verified judge results
if: github.event_name == 'pull_request'
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6
with:
path: /tmp/gstack-eval-input-cache
key: eval-input-v1-${{ github.repository_id }}-pr-${{ github.event.pull_request.number }}-${{ github.run_id }}-${{ github.run_attempt }}-${{ matrix.slice }}
restore-keys: eval-input-v1-${{ github.repository_id }}-pr-${{ github.event.pull_request.number }}-

- name: Run slice ${{ matrix.slice }}/6
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
Expand All @@ -200,8 +225,35 @@ jobs:
EVALS_JOBS: "2"
EVALS_CONCURRENCY: "2"
GSTACK_EVAL_DIR: /tmp/paid-slice-results
EVALS_CACHE_DIR: /tmp/gstack-eval-input-cache
EVALS_CACHE_REPOSITORY: ${{ github.repository }}
EVALS_CACHE_PR: ${{ github.event.pull_request.number }}
EVALS_CACHE_RUNTIME_ID: ${{ needs.build-image.outputs.runtime-id }}
run: EVALS_TIER=gate bun run scripts/test-paid-shards.ts --tier gate --plan /tmp/paid-plan/manifest.json --slice ${{ matrix.slice }}

- name: Find finalized passing receipts
id: receipts
if: ${{ !cancelled() && github.event_name == 'pull_request' }}
run: |
# Only a producer publishes. A later reuse-only slice must not become
# the newest prefix match and hide another slice's newly earned pass.
for receipt in /tmp/gstack-eval-input-cache/*.json; do
[ -f "$receipt" ] || continue
if jq -e --arg run "$GITHUB_RUN_ID/$GITHUB_RUN_ATTEMPT" '.proof.source.runId == $run' "$receipt" >/dev/null 2>&1; then
echo 'present=true' >> "$GITHUB_OUTPUT"
break
fi
done

# An unrelated failing case does not discard already verified passes.
# Failed/retried/partial attempts never become receipts in the first place.
- name: Save verified judge results for this PR
if: ${{ !cancelled() && steps.receipts.outputs.present == 'true' }}
uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6
with:
path: /tmp/gstack-eval-input-cache
key: eval-input-v1-${{ github.repository_id }}-pr-${{ github.event.pull_request.number }}-${{ github.run_id }}-${{ github.run_attempt }}-${{ matrix.slice }}

- name: Upload slice results
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
Expand Down Expand Up @@ -296,7 +348,7 @@ jobs:
slices-comment:
runs-on: ubuntu-latest
needs: slices-report
if: vars.ENABLE_PAID_EVALS == 'true' && always() && github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && needs.slices-report.result != 'skipped'
if: always() && github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && needs.slices-report.result != 'skipped'
timeout-minutes: 5
permissions:
pull-requests: write
Expand All @@ -305,6 +357,11 @@ jobs:
# `issues` permission, not `pull-requests` (#1802 CI fix).
issues: write
steps:
- uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
name: paid-plan
path: /tmp/paid-report

- uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
pattern: paid-slice-[0-9]*
Expand All @@ -327,7 +384,7 @@ jobs:
run: |
# shellcheck disable=SC2086,SC2059
RESULTS=$(find /tmp/paid-report -name '*.json' ! -name 'manifest.json' ! -name 'slice-*.json' ! -name '_partial*' 2>/dev/null | sort)
TOTAL=0; PASSED=0; FAILED=0; FLAKY=0; COST="0"
TOTAL=0; PASSED=0; FAILED=0; FLAKY=0; EXECUTED=0; REUSED=0; COST="0"
SUITE_LINES=""
for f in $RESULTS; do
if ! jq -e '.total_tests' "$f" >/dev/null 2>&1; then
Expand All @@ -337,15 +394,15 @@ jobs:
# FINAL-attempt accounting: eval-store keeps EVERY retry attempt
# as its own record (that's the flake telemetry), so counting raw
# records marks a pass-on-retry as a failure and inflates totals.
# Group by test name, judge the LAST record — flaky passes render
# as the ⚠ line, never as ❌ (WS1 policy: recorded, not blocking).
# Group by test name and judge the LAST record. Retry metadata
# includes both passing and failing final outcomes; show it separately.
# Guarded: a file with total_tests but a null/non-array `tests`
# passes the -e probe, the group_by then fails, and an empty $T
# would abort the whole step under bash -e ([ "" -eq 0 ] is an
# error) — killing the comment on exactly the corrupted-artifact
# runs where the red evidence matters (claude adversarial).
STATS=$(jq -r '[.tests | group_by(.name)[] | last] as $final | "\($final | length) \([$final[] | select(.passed)] | length) \([$final[] | select(.passed | not)] | length) \(.flaky_retries // [] | length)"' "$f" 2>/dev/null) || { echo "Skipping malformed tests[] in: $f"; continue; }
read -r T P F FL <<< "$STATS"
STATS=$(jq -r '[.tests | group_by(.name)[] | last] as $final | "\($final | length) \([$final[] | select(.passed)] | length) \([$final[] | select(.passed | not)] | length) \(.flaky_retries // [] | length) \([$final[] | select(.execution != "reused")] | length) \([$final[] | select(.execution == "reused")] | length)"' "$f" 2>/dev/null) || { echo "Skipping malformed tests[] in: $f"; continue; }
read -r T P F FL EX RE <<< "$STATS"
[ -z "$T" ] && { echo "Skipping malformed tests[] in: $f"; continue; }
C=$(jq -r '.total_cost_usd // 0' "$f")
TIER=$(jq -r '.tier // "unknown"' "$f")
Expand All @@ -355,22 +412,28 @@ jobs:
PASSED=$((PASSED + P))
FAILED=$((FAILED + F))
FLAKY=$((FLAKY + FL))
EXECUTED=$((EXECUTED + EX))
REUSED=$((REUSED + RE))
COST=$(echo "$COST + $C" | bc)
STATUS_ICON="✅"
[ "$F" -gt 0 ] && STATUS_ICON="❌"
[ "$F" -eq 0 ] && [ "$FL" -gt 0 ] && STATUS_ICON="✅⚠"
SUITE_LINES="${SUITE_LINES}| ${TIER}/${SHARD} | ${P}/${T} | ${STATUS_ICON} | \$${C} |\n"
SUITE_LINES="${SUITE_LINES}| ${TIER}/${SHARD} | ${P}/${T} | ${EX} | ${RE} | ${STATUS_ICON} | \$${C} |\n"
done

COVERAGE=$(jq -r '"Profile: \(.profile // "full") / \(.prCoverage.mode // "broad"); selected behaviors: \(.selection.e2e | if . == null then "all" else length end), judges: \(.selection.judges | if . == null then "all" else length end). Deferred to scheduled/release coverage: \(.prCoverage.deferred // [] | length) behaviors and \(.prCoverage.deferredPromptFiles // [] | length) changed prompt files. Deferred checks did not run and receive no PR-pass credit."' /tmp/paid-report/manifest.json) || COVERAGE='Coverage manifest unavailable; no coverage claim.'

STATUS="✅ PASS"
if [ "${RECONCILE_EXIT:-1}" != "0" ] || [ "$FAILED" -gt 0 ]; then STATUS="❌ FAIL"; fi

BODY="## E2E Evals: ${STATUS}

**${PASSED}/${TOTAL}** tests passed | **\$${COST}** total cost | reconcile exit: ${RECONCILE_EXIT:-missing}$([ "$FLAKY" -gt 0 ] && printf ' | ⚠ %s flaky pass(es) — recorded, not blocking' "$FLAKY")
**${PASSED}/${TOTAL}** recorded final results passed | **${EXECUTED} executed, ${REUSED} reused** | **\$${COST}** total cost | reconcile exit: ${RECONCILE_EXIT:-missing}$([ "$FLAKY" -gt 0 ] && printf ' | ⚠ %s cases with multiple attempts' "$FLAKY")

${COVERAGE}

| Shard | Result | Status | Cost |
|-------|--------|--------|------|
| Shard | Result | Executed | Reused | Status | Cost |
|-------|--------|----------|--------|--------|------|
$(echo -e "$SUITE_LINES")

<details><summary>Fail-closed reconciliation</summary>
Expand All @@ -381,7 +444,7 @@ jobs:
</details>

---
*Sliced lane: diff-selected gate census via scripts/test-paid-shards.ts (planner → 6 executors → fail-closed report)*"
*Sliced lane: declared PR profile or broad fallback via scripts/test-paid-shards.ts (planner → 6 executors → fail-closed report). Reused scores retain their original provenance and expiry.*"

if [ "$FAILED" -gt 0 ]; then
FAILURES=""
Expand Down
Loading
Loading