Repository navigation
v1.91.52.0 fix(qa): annotations limits shape; qa-quick does less work in the same 300 s budget - #3081
Merged
Merged
Conversation
/qa runs inside a repository: its clean-tree check and the evidence recorder's revision come from git. The qa-quick working directory was a bare temp dir, so all 15 CI transcripts read (runs 37374409928 to 37744583272) hit 'revision is required when git rev-parse HEAD is unavailable' on the first materialize and spent a retry (6-46 s) plus a learnings-log of the pitfall. qa-only-no-fix already commits its fixture. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
basic.html's /page1 and /page2 links 404 on the test server. With the links in scope, the 15 CI transcripts spent 55-75 s of the 30 s caller probe window and its wrap-up navigating, screenshotting and checkpointing those 404s, then triaging and reporting them as issues. The case proves /qa Quick completes end to end (fallback browser, deadline guard, checkpoint, evidence materialize, report); link-following and defect detection are covered by the qa-b6/b7/b8 and qa-fix-loop cases. The links are now listed as untested coverage. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
exploratory.md §4 gave the annotations shape as {evidence: [...], limits}
with 'checkpoints in limits', but never said what type limits is. The
materialize validator (lib/qa-evidence.ts) wants a non-empty array of
strings. In 11 of 15 qa-quick CI transcripts the agent's first
annotations.json wrote limits as an object, failed with 'need limits
(non-empty string array)', and spent a retry. The section now shows the
shape with a one-line example and says it is never an object. Shared by
/qa and /qa-only through the QA_EXPLORATORY resolver.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
… in the same 300 s budget Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
|
😎 This pull request was merged. |
Contributor
E2E Evals: ✅ PASS18 executed, 0 reused rule/judge records | 0 manual accepted (unscored; no score-cache credit) | $18.55 rule/judge cost | reconcile exit: 0 Profile: pr / mode Rule and judge shards
Fail-closed reconciliationSliced lane: planner → duration-packed executors → fail-closed report. Behavior cases run a pre-registered 3-trial panel (PASS at 2/3 with no contract violation); a PASS 2/3 is shown with its failed trial, never as a clean pass. Reused results retain their original provenance and expiry. |
levi85
pushed a commit
to levi85/gstack
that referenced
this pull request
Oct 14, 2026
… in the same 300 s budget (garrytan#3081) * test(qa-quick): run the fixture in a committed git repository /qa runs inside a repository: its clean-tree check and the evidence recorder's revision come from git. The qa-quick working directory was a bare temp dir, so all 15 CI transcripts read (runs 37374409928 to 37744583272) hit 'revision is required when git rev-parse HEAD is unavailable' on the first materialize and spent a retry (6-46 s) plus a learnings-log of the pitfall. qa-only-no-fix already commits its fixture. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test(qa-quick): scope Quick to the fixture page, not its 404 links basic.html's /page1 and /page2 links 404 on the test server. With the links in scope, the 15 CI transcripts spent 55-75 s of the 30 s caller probe window and its wrap-up navigating, screenshotting and checkpointing those 404s, then triaging and reporting them as issues. The case proves /qa Quick completes end to end (fallback browser, deadline guard, checkpoint, evidence materialize, report); link-following and defect detection are covered by the qa-b6/b7/b8 and qa-fix-loop cases. The links are now listed as untested coverage. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(qa): say that annotations.json limits is a non-empty string array exploratory.md §4 gave the annotations shape as {evidence: [...], limits} with 'checkpoints in limits', but never said what type limits is. The materialize validator (lib/qa-evidence.ts) wants a non-empty array of strings. In 11 of 15 qa-quick CI transcripts the agent's first annotations.json wrote limits as an object, failed with 'need limits (non-empty string array)', and spent a retry. The section now shows the shape with a one-line example and says it is never an object. Shared by /qa and /qa-only through the QA_EXPLORATORY resolver. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * v1.91.52.0 fix(qa): annotations limits shape; qa-quick does less work in the same 300 s budget Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
qa-quick(gate, rule) runs on every PR that touchesbrowse/src. In 15 recent CI runs it passed 13 times in 206–280 s; the other 2 hit the 300 s session timeout. It is also the one red blocking #3079. CI transcripts show three avoidable retries or detours. The session budget, the thresholds and the case's kind are unchanged./qaprose (affects users):exploratory.md§4 describedannotations.jsonas{evidence: [...], limits}but never said what typelimitsis. The validator accepts only a non-empty string array. In 11 of 15 runs the first attempt wrote an object andgstack-qa-evidence materializefailed. The section now showslimits: ["checkpoint 001"]and says it is never an object./qa-onlyshares the same section.materializefirst failed with "revision is required when git rev-parse HEAD is unavailable". The fixture is now committed, asqa-only-no-fixalready does.basic.html's/page1and/page2links return 404 on the test server, and runs spent 55–75 s investigating and reporting them. Quick scope is now the page itself, and the links are listed as untested coverage. qa-b6/b7/b8 and qa-fix-loop still cover link following.Measurement (diagnostic; never changes a recorded verdict)
CI image
ghcr.io/garrytan/gstack/ci@sha256:6132a8501ba73a236f352cd4a2a7013f790aae1f089d58b8827206532d1df89a, Claude Code 2.1.292, Bun 1.4.2,--jobs 2,CI=true.CI=true, so sandboxed Chromium crashed; invalid)CI=truelaunch correctionlimitsshape in exploratory §4Round-3 per trial:
The pass median went from 252 s (CI before) to 187 s, and the mean cost from $1.75 to $1.46. Failed
materializecalls went from 15 of 15 runs to none in the round-3 captures. The slowest pass (256 s, 85.3%) sits atHEADROOM_FAIL(255 s), so the weekly headroom alarm may still flag this case.Validation
qa/SKILL.md workflowandqa-only/SKILL.md workflow: PASS.typecheckandtypecheck:testclean; harness-version check: no drift; credential scan: 0 high.bun run test(Ubicloud): 17/17 shards passed, 26,959 tests in 1,126 files.