Skip to content

docs(agents): default to no new tests and tighten the anti-pattern list - #37993

Merged
nwparker merged 1 commit into
mainfrom
nwparker/low-value-tests
Nov 4, 2026
Merged

nwparker merged 1 commit into
mainfrom
nwparker/low-value-tests

Conversation

@nwparker

@nwparker nwparker commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor
Files Added Deleted Net
Test 0 0 0 0
Prod 3 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​26 $\color{#cf222e}{\Huge{\mathbf{−}}}$​20 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​6

ELI5

Agents keep adding tests that look busy but catch nothing. AGENTS.md now tells them not to add tests by default, says when a test is worth adding, and tells them to delete bad tests they touch. The anti-pattern doc is shorter and names the patterns we keep seeing.

What Changed

  • AGENTS.md: the one-line link becomes a single, stronger rule: Do not add tests by default. Add a test only if it fails when the change is reverted, hardcodes its expected values, and runs at the single layer that owns the behavior. It also lists what never to test, and says to delete such tests rather than update them.
  • docs/agents-md-refs/test-antipatterns.md: 585 → ~350 words. It now names the recurring patterns: production code as its own oracle, assertions that pass either way, one contract re-tested per layer/platform/locale, Cartesian it.each, i18n copy, pinned constants, third-party behavior, work counts and wall-clock timing, private state, and hand-copied mocks.
  • PR template: a checkbox that every added test fails without the change and avoids the anti-patterns.

Why

An agent audit of 259 PRs merged in the last 3 days found 9 that added clearly low-value tests (#37148, #37151, #37189, #37213, #37246, #37262, #37390, #37657, #37730) and about 85 with minor issues. Cleanup PRs keep deleting hundreds of tests a day, so the old one-line link wasn't preventing them.

Linked Issue

N/A (maintainer docs change)

Visual Proof

N/A: docs and PR template only.

Testing

I ran the rule against a small sandbox: a TypeScript repo with 6 tasks (i18n, constant bump, bug fix, perf, CSS, cross-layer field), run by both Claude Sonnet and Codex. To match real pressure, the repo already contained bad tests to copy, and every prompt said "Add appropriate tests." Counts are new test cases on the 5 tasks that don't need a test:

Rule New test cases Bad seeded tests deleted Wall-clock tests Constant re-pinned
main (link only) 20 1 1 2/2
First draft 7 2 1 1/2
This PR 5 8 0 0/2

On the bug-fix task, both agents still added the one hardcoded regression test under every version. Each version got one trial per agent and task, so treat the numbers as directional.

  • I manually tested these changes locally
  • Automated tests added/updated, or explained why not below
  • Every added test fails without this change and avoids the test anti-patterns

No tests, since this is docs only. pnpm check:changed passed.

Risk

Level: Low. Only agent and contributor guidance changes. To back it out, revert this PR.

AI Disclosure

Review

Agent skill upstream boundary

  • Not applicable, or this change copies or mechanically translates no upstream skill-installer source, tests, fixtures, registry entries, path tables, comments, or documentation.

Notes

Ensure no issues in: Security, Cross-platform support (Linux, Windows, Mac), Remote SSH, Mobile, general backwards compatibility, performance

Checklist

  • This PR is small and focused
  • I explained what changed and why (ELI5, the user-facing before/after, the mechanism, and why over the alternatives)
  • I stated the risk: what could break, for whom, and how to back it out
  • Before/after screenshots or videos attached for UI changes, or N/A with reason
  • Self-reviewed for correctness, security, and performance
  • Cross-platform, SSH/remote, and path/shortcut impact considered (or N/A)
  • pnpm lint, pnpm typecheck, pnpm test, and pnpm build pass (or CI will cover; local preferred)

@coderabbitai

coderabbitai Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Walkthrough

The repository test guidance now sets criteria for test expectations, scope, and assertion targets. The anti-pattern guide adds a pre-commit checklist and examples of patterns to avoid. The pull request template adds a checkbox requiring tests to fail without the change and avoid the linked anti-patterns.

Merge Risk: Low · up to 73135

The guidance should cover updated tests and provide valid assertion examples before merging.

Pre-merge checks: Passed 4
Check name Status Explanation
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Title check Passed The title clearly and concisely describes the documentation changes: defaulting to no new tests and tightening the anti-pattern guidance.
Description check Passed The description covers the required sections, including the change summary, rationale, linked-issue exception, visual proof, testing, risk, scope, and checklist. It explains that no tests were added b…
Full details: Description check

Explanation

The description covers the required sections, including the change summary, rationale, linked-issue exception, visual proof, testing, risk, scope, and checklist. It explains that no tests were added because the change is documentation-only.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

Review info
Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Repository UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: cf669b5b-0cfa-4b23-926d-79b0e99479b4
Commits

Reviewing files that changed from the base of the PR and between c332002 and 7313585.

Files selected for processing: 3
  • .github/pull_request_template.md
  • AGENTS.md
  • docs/agents-md-refs/test-antipatterns.md

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 7 remain after this review.


- [ ] I manually tested these changes locally
- [ ] Automated tests added/updated, or explained why not below
- [ ] Every added test fails without this change and avoids the [test anti-patterns](https://github.com/stablyai/orca/blob/main/docs/agents-md-refs/test-antipatterns.md)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor · Maintainability & Code Quality · Quick win

Apply the test gate to updated tests too.

AGENTS.md applies this gate when contributors write or edit tests, and the anti-pattern guide says to check each test before committing. This checkbox covers only added tests, so an updated existing test can bypass the gate. Change Every added test to Every added or updated test. (github.com)

- Copy inventories, manifests, exports, imports, or strings instead of proving behavior.
- Re-test one contract at every layer (shared, IPC, renderer, mobile, CLI) or in both desktop and mobile copies.
- Use Cartesian `it.each` matrices, or branch with `if`/ternaries inside a parameterized body, when one case per distinct behavior would do.
- Copy inventories, manifests, exports, imports, or strings instead of proving behavior. This includes hand-typed command or registry lists, and counts such as `registry.size).toBe(50)`.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor · Maintainability & Code Quality · Quick win

Fix the assertion examples.

registry.size).toBe(50) and editorFormatOnSave).toBe(false) omit expect(. They are invalid assertion syntax and can lead contributors to copy a syntax error. Use expect(registry.size).toBe(50) and expect(editorFormatOnSave).toBe(false). (raw.githubusercontent.com)

Also applies to: 25-25

AGENTS.md now says not to add tests by default, names the gate inline, and
tells agents to delete low-value tests they touch. The anti-pattern doc names
the recurring patterns from an audit of 259 recent PRs in about 40% fewer words.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@nwparker
nwparker force-pushed the nwparker/low-value-tests branch from 7313585 to 72665c5 Compare November 4, 2026 02:01
@nwparker nwparker changed the title docs(agents): inline the low-value test gate and name recurring anti-patterns docs(agents): default to no new tests and tighten the anti-pattern list Nov 4, 2026
@nwparker
nwparker enabled auto-merge (squash) November 4, 2026 02:03
@nwparker
nwparker merged commit 542a2ec into main Nov 4, 2026
22 of 23 checks passed
@nwparker
nwparker deleted the nwparker/low-value-tests branch November 4, 2026 02:05
brennanb2025 added a commit that referenced this pull request Nov 6, 2026
Main's test rule (#37993) bans tests of work counts and expected values
computed by production code. Removes the join and snapshot-visit counting
tests and their helper; the output preview test now checks a saved running
row against literal head, length and digest values.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant