Skip to content

chore: make agent mistakes fail early - #26167

Open
t3dotgg wants to merge 5 commits into
mainfrom
t3/audit-agent-codebase-effectiveness
Open

t3dotgg wants to merge 5 commits into
mainfrom
t3/audit-agent-codebase-effectiveness

Conversation

@t3dotgg

@t3dotgg t3dotgg commented Oct 27, 2026 •

Copy link
Copy Markdown
Member

Agents keep making mistakes in this repo that show up only in CI or in review. We audited recent PR reviews and failing CI runs, and fixed each mistake as high up as possible: in code or tooling first, prose last. This PR has the fixes that fit in one small change.

What changes

  1. The pre-commit hook lints. Lint was the most common PR CI failure (13 of 40 sampled), but the hook only formatted. It now runs vp lint after vp fmt on staged files. A small commit takes about 1s more.
  2. check:changed runs the checks it missed. It ran only the tests beside changed files, so these failed only in CI:
    • AgentContextBudget.test.ts now runs when the server is affected. Its local and CI failure messages say to run the evals for the text before you cut it to fit. Cuts made for size with no eval behind them caused the inline chart regression after #24850, fixed in #25766 and #25776.
    • React Compiler coverage runs only the cases for the changed web and mobile files, in one step. Mobile's checks that read more than one file always run.
    • Vitest steps run one after another. When the lockfile changes, vitest clears its cache on the next run, and two runs that start together race (ENOTEMPTY). The old script failed 2 of 3 runs after a lockfile change. The new one passed 4 of 4.
  3. One copy of the pending-input answer rules. Web and mobile each had their own code to turn question drafts into answers, and the two copies had drifted. The rules are now in @t3tools/client-runtime/state/pending-user-input, taken from mobile's stricter copy. Real input gives the same answers as before: the schema trims option labels, and both UIs store exact option values.

Since the first version, main made two items unnecessary, so I removed them:

  • #26161 replaced the React effect budget with a reason on every effect, so the item that made the budget exact is gone.
  • #26166 kept the test rules in both AGENTS.md and testing.md on purpose and added to them, so the item that moved them out of AGENTS.md is gone.

Proof: a staged file with text-[13px] is now blocked at commit. With three probe edits (a web try/finally compiler bailout, a deleted mobile file still in KNOWN_BAILOUTS, and a longer MCP tool description), the old check:changed caught none and the new one caught all three. The longer description fails with the eval pointer. On the rebased branch, vp run check:changed passes, and the compiler step runs 12 cases for 8 changed files.

Council review

Rounds: 3 (plan review, then build)

Delegate Round 1 Round 2 Round 3
Claude Opus 5.5 7/10 8.5/10 9.2/10
Muse Wonder Full 7/10 9/10 no score
Codex GPT-6.1 Sol 8.5/10 9.2/10 no score

Round 3 went only to the one reviewer still below 9.

Reviewed with sol-loop: 1 round with GPT-6.1 Sol. Sol verdict: approved

Created with Claude Opus 5.5 in Claude Code (inside T3 Code).

🤖 Generated with Claude Code

@github-actions github-actions Bot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:XXL 1,000+ changed lines (additions + deletions). labels Oct 27, 2026
Comment thread scripts/check-changed.ts Outdated
@macroscopeapp

macroscopeapp Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Approved at fccafa9

Macroscope's review found this PR approvable — This change strengthens local checks and consolidates pending-input answer rules across web and mobile. Its effects are bounded to developer validation and existing answer handling, with no product-default changes or diagnostic suppression.

You can add or adjust custom eligibility rules. Learn more.

macroscopeapp[bot]
macroscopeapp Bot previously approved these changes Oct 27, 2026
@coderabbitai

coderabbitai Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review.

Recent review info
Run configuration
Commits

Reviewing files that changed from the base of the PR and between 3c2e477 and fccafa9.

Files selected for processing: 10
  • AGENTS.md
  • apps/mobile/src/features/threads/PendingUserInputCard.tsx
  • apps/mobile/src/lib/threadActivity.test.ts
  • apps/mobile/src/lib/threadActivity.ts
  • apps/web/src/components/chat/ChatComposer.tsx
  • apps/web/src/pendingUserInput.test.ts
  • apps/web/src/pendingUserInput.ts
  • docs/operations/development.md
  • scripts/check-changed.ts
  • vite.config.ts
Files with no reviewable changes: 2
  • apps/mobile/src/lib/threadActivity.test.ts
  • apps/mobile/src/lib/threadActivity.ts
Files skipped from review as they are similar to previous changes: 2
  • docs/operations/development.md
  • AGENTS.md

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 8 remain after this review.

Walkthrough

Pending user-input answer logic is centralized in client-runtime and adopted by web and mobile callers. The changed-file check script adds targeted checks and sequential test execution. Staged-file linting and agent-context-budget guidance are also updated.

Changes: Pending user-input answer handling

Layer / File(s) Summary
Shared answer helpers
packages/client-runtime/src/state/pendingUserInput.ts, packages/client-runtime/src/state/pendingUserInput.test.ts, packages/client-runtime/package.json
Client-runtime adds shared types and helpers for normalizing drafts, resolving answers, updating selections, and building answer maps. Tests cover answer resolution and selection cases. The package exports the new module.
Web and mobile adoption
apps/web/src/pendingUserInput.ts, apps/web/src/pendingUserInput.test.ts, apps/web/src/components/chat/*, apps/mobile/src/lib/threadActivity.ts, apps/mobile/src/lib/threadActivity.test.ts, apps/mobile/src/features/threads/*, apps/mobile/src/state/use-selected-thread-requests.ts
Web progress derivation normalizes selections against the active question and uses shared answer types. Web and mobile callers import shared helpers. The local answer-processing implementations and their related tests are removed.

Changed-file checks and verification guidance

Layer / File(s) Summary
Changed-file check selection and execution
scripts/check-changed.ts, docs/operations/development.md, AGENTS.md
check:changed selects Agent Context Budget coverage when t3 is affected and filtered React Compiler coverage for changed web and mobile source files. Test steps run sequentially. Documentation describes the added checks.
Staged lint and budget guidance
vite.config.ts, .github/workflows/agent-context-budget.yml, apps/server/src/provider/AgentContextBudget.test.ts
The staged-file task runs formatting followed by linting. Agent-context-budget failure messages direct contributors to update the budget and run the referenced evals.

Priority: Low

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Other

Suggested reviewers: juliusmarminge

Merge Risk: Minimal · up to fccaf

No merge-blocking issue was identified; normal checks remain appropriate.

Pre-merge checks: Passed 3 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Description check Warning The description clearly explains the problem, changes, and verification results. It does not provide the required Scope and approval information, including explicit maintainer approval or a clear expl… Add a Scope and approval section. Link the triaged issue or approval discussion with the maintainer approval comment, or explain why this focused configuration change qualifies for an exemption.
Passed Passed checks: 3 passed
Check name Status Explanation
Title check Passed The title clearly summarizes the primary change: catching common agent mistakes earlier through local tooling and checks.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Description check

Explanation

The description clearly explains the problem, changes, and verification results. It does not provide the required Scope and approval information, including explicit maintainer approval or a clear explanation for why the focused change is exempt.

Resolution

Add a Scope and approval section. Link the triaged issue or approval discussion with the maintainer approval comment, or explain why this focused configuration change qualifies for an exemption.

Finishing Touches
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

Provider Metric Main baseline This PR Impact PR ceiling
Codex Total thread wire 5.0 KiB 5.0 KiB 0 B (0.0%) 6.8 KiB ✅
Codex Thread snapshot wire 3.8 KiB 3.8 KiB 0 B (0.0%) 4.9 KiB ✅
Codex Live turn WebSocket wire 1.3 KiB 1.3 KiB 0 B (0.0%) 2.0 KiB ✅
Codex Live turn WebSocket decoded 21.6 KiB 21.6 KiB 0 B (0.0%) 24.4 KiB ✅
Codex Live turn messages 2 2 0 (0.0%) 8 ✅
Claude Total thread wire 5.0 KiB 5.0 KiB 0 B (0.0%) 6.8 KiB ✅
Claude Thread snapshot wire 3.8 KiB 3.8 KiB 0 B (0.0%) 4.9 KiB ✅
Claude Live turn WebSocket wire 1.3 KiB 1.3 KiB 0 B (0.0%) 2.0 KiB ✅
Claude Live turn WebSocket decoded 21.9 KiB 21.9 KiB 0 B (0.0%) 24.4 KiB ✅
Claude Live turn messages 2 2 0 (0.0%) 8 ✅

Baseline: d8f4980 · PR result: fccafa9 · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

  • Codex decoded thread snapshot: 108.5 KiB
  • Claude decoded thread snapshot: 108.8 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

@t3dotgg
t3dotgg force-pushed the t3/audit-agent-codebase-effectiveness branch from 8ed06f3 to 3c2e477 Compare October 27, 2026 23:09
@macroscopeapp
macroscopeapp Bot dismissed their stale review October 27, 2026 23:09

Dismissing prior approval to re-evaluate 3c2e477

@github-actions github-actions Bot added size:XL 500-999 changed lines (additions + deletions). and removed size:XXL 1,000+ changed lines (additions + deletions). labels Oct 27, 2026
macroscopeapp[bot]
macroscopeapp Bot previously approved these changes Oct 27, 2026
t3dotgg and others added 5 commits October 28, 2026 18:24
Lint failures were the most common PR CI failure while the hook only
formatted. The hook now runs lint after format on staged files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Web and mobile each had their own rules for turning question drafts into
answers, and the two had drifted. The rules now live in client-runtime,
taken from mobile's stricter copy, and both clients import them directly.
Web's progress helpers use the shared normalizer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The size budget fails when model-facing text grows, and its messages said
to cut text. Cuts with no eval behind them caused the inline chart
regression after #24850. Now that check:changed runs the budget locally,
both messages say to run the evals that cover the text before cutting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@t3dotgg
t3dotgg force-pushed the t3/audit-agent-codebase-effectiveness branch from 3c2e477 to fccafa9 Compare October 29, 2026 01:24
@macroscopeapp
macroscopeapp Bot dismissed their stale review October 29, 2026 01:26

Dismissing prior approval to re-evaluate fccafa9

@juliusmarminge juliusmarminge left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for theo's agent:

Comment thread vite.config.ts
staged: {
"*": "vp fmt --no-error-on-unmatched-pattern",
// In order, so lint reads the formatted file. CI lint failures cost a full round trip.
"*": ["vp fmt --no-error-on-unmatched-pattern", "vp lint --no-error-on-unmatched-pattern"],

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add knip too

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XL 500-999 changed lines (additions + deletions). vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants